* USB fuzzing with syzbot @ 2019-04-17 16:33 Andrey Konovalov 2019-04-19 8:35 ` Greg Kroah-Hartman 2019-08-13 13:06 ` Andrey Konovalov 0 siblings, 2 replies; 9+ messages in thread From: Andrey Konovalov @ 2019-04-17 16:33 UTC (permalink / raw) To: Alan Stern, Greg Kroah-Hartman Cc: Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko Hi, As you might have noticed, syzbot has started reporting bugs in the USB subsystem that can be triggered externally by a malicious USB device. Right now the fuzzing is done via a GadgetFS-like interface to emulate USB devices through the USB Gadget subsystem and the Dummy HCD/UDC module to "loopback" connect those devices to the running kernel. There are more details in my OffensiveCon talk [1], [2]. A few questions/comments: 1. Which tree should we use for testing? Right now we're targeting the upstream tree, but we can switch to some USB development tree, where the fixes are likely to end up earlier. 2. Is there an easy way to figure out which config options enable drivers reachable over USB? Right now our kernel config is based on one of the Debian kernel configs, that supposedly enables enough relevant USB drivers. At the same time it enables a lot of other unnecessary stuff, which makes the kernel big and long to compile. Ideally, we would to have a way to auto-generate a kernel config that enables all the relevant (enabled by at least one of the distros) USB drivers. I've looked at whether it's possible to figure out which particular options in some kernel config are related to USB, but it seems that neither the option names, nor the way they are grouped in the config file, are representative enough. 3. Regarding that GadgetFS-like interface. Initially I was using GadgetFS (together with the Dummy HCD/UDC module) to perform emulation of USB devices for fuzzing, but later switched to a custom written interface. This interface is essentially implemented in the following patch [3]. An example that emulates a USB keyboard through this interface can be found here [4]. And the syzkaller parts responsible for USB fuzzing are here [5], [6]. The incentive to implement a different interface was to provide a somewhat raw and direct access to the USB Gadget layer for the userspace, where every USB request is passed to the userspace to get a response. The main differences between this interface (referred to as usb-fuzzer for now) and GadgetFS are: 1) GadgetFS does some sanity checks on the provided USB descriptors, which is something we don't want for fuzzing. We want the descriptors to be as corrupted as they can. 2) GadgetFS handles some of the USB requests internally based on the provided device descriptor, which is also something we don't want. For example we may want to be able to provide differently corrupted responses to the same request. 3) usb-fuzzer has ioctl-based interface instead of a filesystem-based one. I wouldn't say it's that big of a deal, but it makes it somewhat easier to incorporate into a fuzzer. 4) Somewhat related to the previous point: usb-fuzzer uses predictable endpoint names across different UDCs. Right now each UDC driver defines endpoint names via EP_INFO() as it pleases. And GadgetFS uses those names to create file entries for each of the endpoints. As a result, endpoint names for different UDCs will be different and it requires more work to write a portable userspace gadget. The usb-fuzzer interface auto selects and assigns an endpoint based on the required features like the transfer type. 5) GadgetFS binds to the first available UDC, usb-fuzzer provides a way to select a UDC to bind to. Since the fuzzing happens in multiple processes each of which has its own Dummy UDC assigned, we want to have control over which UDC we bind to. This part is a bit confusing, but what I found out is that a UDC is selected based on two different identifying names. I call the first one "udc_device_name" and the second one "udc_driver_name". "udc_device_name" has to be assigned to usb_gadget_driver->udc_name when usb_gadget_probe_driver() is called, and "udc_driver_name" is what we have to compare usb_gadget->name with inside of the usb_gadget_driver->bind() callback. For example, Dummy UDC has "dummy_udc" as its "udc_driver_name" and "dummy_udc.N" as its "udc_device_name". At the same time the dwc2 driver that is used on Raspberry Pi Zero, has "20980000.usb" as both "udc_driver_name" and "udc_device_name". Overall, the usb-fuzzer interface implementation has a similar structure to that of GadgetFS, but looks way simpler (although that might be because I've missed to implement some functionality :). We'd like to get this upstreamed, but I'm not sure whether this should be a separate interface (which we can rebrand as a raw usb gadget interface or something like that) or we should try to make it a special mode of GadgetFS. I like the former approach more, as GadgetFS looks quite complicated from my point of view and making fundamental changes to it doesn't seem like an easy task. This is where we'd like to get your input. If you have any other questions or comments, please let us know. Alan, thanks a lot for answering to lots of my USB related questions while I was working on this, that helped tremendously! Thanks! [1] https://docs.google.com/presentation/d/1z-giB9kom17Lk21YEjmceiNUVYeI6yIaG5_gZ3vKC-M/edit#slide=id.g1925acbbf3_0_0 [2] https://www.youtube.com/watch?v=1MD5JV6LfxA [3] https://github.com/google/kasan/commit/9a33b36996cb6138a806eab931ba43c63a836708 [4] https://github.com/google/syzkaller/blob/usb-fuzzer/tools/usb/keyboard.c [5] https://github.com/google/syzkaller/blob/master/sys/linux/vusb.txt [6] https://github.com/google/syzkaller/blob/master/executor/common_usb.h ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-17 16:33 USB fuzzing with syzbot Andrey Konovalov @ 2019-04-19 8:35 ` Greg Kroah-Hartman 2019-04-19 14:39 ` Alan Stern 2019-04-24 16:05 ` Andrey Konovalov 2019-08-13 13:06 ` Andrey Konovalov 1 sibling, 2 replies; 9+ messages in thread From: Greg Kroah-Hartman @ 2019-04-19 8:35 UTC (permalink / raw) To: Andrey Konovalov Cc: Alan Stern, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Wed, Apr 17, 2019 at 06:33:39PM +0200, Andrey Konovalov wrote: > Hi, > > As you might have noticed, syzbot has started reporting bugs in the > USB subsystem that can be triggered externally by a malicious USB > device. Right now the fuzzing is done via a GadgetFS-like interface to > emulate USB devices through the USB Gadget subsystem and the Dummy > HCD/UDC module to "loopback" connect those devices to the running > kernel. There are more details in my OffensiveCon talk [1], [2]. > > A few questions/comments: > > 1. Which tree should we use for testing? > > Right now we're targeting the upstream tree, but we can switch to some > USB development tree, where the fixes are likely to end up earlier. USB fixes land in the usb.git tree on git.kernel.org in the usb-testing branch (for the next release), and in the usb-linus branch (for this release). So you can just merge those two branches together if you like to be sure you have all of the latest fixes. > 2. Is there an easy way to figure out which config options enable > drivers reachable over USB? Looking for all options that depend on USB is a good start. > Right now our kernel config is based on one of the Debian kernel > configs, that supposedly enables enough relevant USB drivers. At the > same time it enables a lot of other unnecessary stuff, which makes the > kernel big and long to compile. Ideally, we would to have a way to > auto-generate a kernel config that enables all the relevant (enabled > by at least one of the distros) USB drivers. I've looked at whether > it's possible to figure out which particular options in some kernel > config are related to USB, but it seems that neither the option names, > nor the way they are grouped in the config file, are representative > enough. Yeah, it's hard to just carve out this type of configuration, but here's what I have done in the past to try to be sure I enabled all USB drivers in my kernel configuration. First, start with a "minimally working configuration" by running: make localmodconfig on a working system, with the needed modules for booting and operating properly already loaded. That gives you a .config file that should take only minutes to build, compared to much longer for the normal distro configuration (also be sure to disable some debugging options so you don't spend extra time building and stripping symbols). Boot and make sure that configuration works. Then, take that .config and do: - disable USB from the configuration by deleting the: CONFIG_USB_SUPPORT=y option from your .config - run 'make oldconfig' to disable all USB drivers - turn USB back on by setting CONFIG_USB_SUPPORT=y back on in your .config - run 'make oldconfig' and answer 'y' or 'm' to all of the driver options you are presented with. That usually catches almost all of them. Sometimes you need to make sure you have some other subsystem enabled (like SCSI), but odds are, if you start with a "normally stripped down" configuration that works, you should be fine. > 3. Regarding that GadgetFS-like interface. > > Initially I was using GadgetFS (together with the Dummy HCD/UDC > module) to perform emulation of USB devices for fuzzing, but later > switched to a custom written interface. This interface is essentially > implemented in the following patch [3]. An example that emulates a USB > keyboard through this interface can be found here [4]. And the > syzkaller parts responsible for USB fuzzing are here [5], [6]. The > incentive to implement a different interface was to provide a somewhat > raw and direct access to the USB Gadget layer for the userspace, where > every USB request is passed to the userspace to get a response. > > The main differences between this interface (referred to as usb-fuzzer > for now) and GadgetFS are: > > 1) GadgetFS does some sanity checks on the provided USB descriptors, > which is something we don't want for fuzzing. We want the descriptors > to be as corrupted as they can. > > 2) GadgetFS handles some of the USB requests internally based on the > provided device descriptor, which is also something we don't want. For > example we may want to be able to provide differently corrupted > responses to the same request. > > 3) usb-fuzzer has ioctl-based interface instead of a filesystem-based > one. I wouldn't say it's that big of a deal, but it makes it somewhat > easier to incorporate into a fuzzer. > > 4) Somewhat related to the previous point: usb-fuzzer uses predictable > endpoint names across different UDCs. > > Right now each UDC driver defines endpoint names via EP_INFO() as it > pleases. And GadgetFS uses those names to create file entries for each > of the endpoints. As a result, endpoint names for different UDCs will > be different and it requires more work to write a portable userspace > gadget. The usb-fuzzer interface auto selects and assigns an endpoint > based on the required features like the transfer type. > > 5) GadgetFS binds to the first available UDC, usb-fuzzer provides a > way to select a UDC to bind to. > > Since the fuzzing happens in multiple processes each of which has its > own Dummy UDC assigned, we want to have control over which UDC we bind > to. This part is a bit confusing, but what I found out is that a UDC > is selected based on two different identifying names. I call the first > one "udc_device_name" and the second one "udc_driver_name". > "udc_device_name" has to be assigned to usb_gadget_driver->udc_name > when usb_gadget_probe_driver() is called, and "udc_driver_name" is > what we have to compare usb_gadget->name with inside of the > usb_gadget_driver->bind() callback. For example, Dummy UDC has > "dummy_udc" as its "udc_driver_name" and "dummy_udc.N" as its > "udc_device_name". At the same time the dwc2 driver that is used on > Raspberry Pi Zero, has "20980000.usb" as both "udc_driver_name" and > "udc_device_name". > > Overall, the usb-fuzzer interface implementation has a similar > structure to that of GadgetFS, but looks way simpler (although that > might be because I've missed to implement some functionality :). > > We'd like to get this upstreamed, but I'm not sure whether this should > be a separate interface (which we can rebrand as a raw usb gadget > interface or something like that) or we should try to make it a > special mode of GadgetFS. I like the former approach more, as GadgetFS > looks quite complicated from my point of view and making fundamental > changes to it doesn't seem like an easy task. This is where we'd like > to get your input. I'll leave the gadgetfs questions to others that know that api better than I do :) thanks, greg k-h ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-19 8:35 ` Greg Kroah-Hartman @ 2019-04-19 14:39 ` Alan Stern 2019-04-24 16:09 ` Andrey Konovalov 2019-04-24 16:05 ` Andrey Konovalov 1 sibling, 1 reply; 9+ messages in thread From: Alan Stern @ 2019-04-19 14:39 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Andrey Konovalov, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Fri, 19 Apr 2019, Greg Kroah-Hartman wrote: > > 3. Regarding that GadgetFS-like interface. > > > > Initially I was using GadgetFS (together with the Dummy HCD/UDC > > module) to perform emulation of USB devices for fuzzing, but later > > switched to a custom written interface. This interface is essentially > > implemented in the following patch [3]. An example that emulates a USB > > keyboard through this interface can be found here [4]. And the > > syzkaller parts responsible for USB fuzzing are here [5], [6]. The > > incentive to implement a different interface was to provide a somewhat > > raw and direct access to the USB Gadget layer for the userspace, where > > every USB request is passed to the userspace to get a response. > > > > The main differences between this interface (referred to as usb-fuzzer > > for now) and GadgetFS are: > > > > 1) GadgetFS does some sanity checks on the provided USB descriptors, > > which is something we don't want for fuzzing. We want the descriptors > > to be as corrupted as they can. > > > > 2) GadgetFS handles some of the USB requests internally based on the > > provided device descriptor, which is also something we don't want. For > > example we may want to be able to provide differently corrupted > > responses to the same request. > > > > 3) usb-fuzzer has ioctl-based interface instead of a filesystem-based > > one. I wouldn't say it's that big of a deal, but it makes it somewhat > > easier to incorporate into a fuzzer. > > > > 4) Somewhat related to the previous point: usb-fuzzer uses predictable > > endpoint names across different UDCs. > > > > Right now each UDC driver defines endpoint names via EP_INFO() as it > > pleases. And GadgetFS uses those names to create file entries for each > > of the endpoints. As a result, endpoint names for different UDCs will > > be different and it requires more work to write a portable userspace > > gadget. The usb-fuzzer interface auto selects and assigns an endpoint > > based on the required features like the transfer type. > > > > 5) GadgetFS binds to the first available UDC, usb-fuzzer provides a > > way to select a UDC to bind to. > > > > Since the fuzzing happens in multiple processes each of which has its > > own Dummy UDC assigned, we want to have control over which UDC we bind > > to. This part is a bit confusing, but what I found out is that a UDC > > is selected based on two different identifying names. I call the first > > one "udc_device_name" and the second one "udc_driver_name". > > "udc_device_name" has to be assigned to usb_gadget_driver->udc_name > > when usb_gadget_probe_driver() is called, and "udc_driver_name" is > > what we have to compare usb_gadget->name with inside of the > > usb_gadget_driver->bind() callback. For example, Dummy UDC has > > "dummy_udc" as its "udc_driver_name" and "dummy_udc.N" as its > > "udc_device_name". At the same time the dwc2 driver that is used on > > Raspberry Pi Zero, has "20980000.usb" as both "udc_driver_name" and > > "udc_device_name". > > > > Overall, the usb-fuzzer interface implementation has a similar > > structure to that of GadgetFS, but looks way simpler (although that > > might be because I've missed to implement some functionality :). > > > > We'd like to get this upstreamed, but I'm not sure whether this should > > be a separate interface (which we can rebrand as a raw usb gadget > > interface or something like that) or we should try to make it a > > special mode of GadgetFS. I like the former approach more, as GadgetFS > > looks quite complicated from my point of view and making fundamental > > changes to it doesn't seem like an easy task. This is where we'd like > > to get your input. > > I'll leave the gadgetfs questions to others that know that api better > than I do :) Assuming the usb-fuzzer implementation gets merged, I think it definitely should be kept separate from gadgetfs. The differences in functionality are too great; it doesn't make sense to combine them in a single driver. Also, while it often is reasonable for people to build gadgetfs for production use, usb-fuzzer is very much more of a developer/debugger type of thing. Alan Stern ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-19 14:39 ` Alan Stern @ 2019-04-24 16:09 ` Andrey Konovalov 0 siblings, 0 replies; 9+ messages in thread From: Andrey Konovalov @ 2019-04-24 16:09 UTC (permalink / raw) To: Alan Stern Cc: Greg Kroah-Hartman, Andrey Konovalov, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Fri, Apr 19, 2019 at 4:39 PM Alan Stern <stern@rowland.harvard.edu> wrote: > > On Fri, 19 Apr 2019, Greg Kroah-Hartman wrote: > > > > 3. Regarding that GadgetFS-like interface. > > > > > > Initially I was using GadgetFS (together with the Dummy HCD/UDC > > > module) to perform emulation of USB devices for fuzzing, but later > > > switched to a custom written interface. This interface is essentially > > > implemented in the following patch [3]. An example that emulates a USB > > > keyboard through this interface can be found here [4]. And the > > > syzkaller parts responsible for USB fuzzing are here [5], [6]. The > > > incentive to implement a different interface was to provide a somewhat > > > raw and direct access to the USB Gadget layer for the userspace, where > > > every USB request is passed to the userspace to get a response. > > > > > > The main differences between this interface (referred to as usb-fuzzer > > > for now) and GadgetFS are: > > > > > > 1) GadgetFS does some sanity checks on the provided USB descriptors, > > > which is something we don't want for fuzzing. We want the descriptors > > > to be as corrupted as they can. > > > > > > 2) GadgetFS handles some of the USB requests internally based on the > > > provided device descriptor, which is also something we don't want. For > > > example we may want to be able to provide differently corrupted > > > responses to the same request. > > > > > > 3) usb-fuzzer has ioctl-based interface instead of a filesystem-based > > > one. I wouldn't say it's that big of a deal, but it makes it somewhat > > > easier to incorporate into a fuzzer. > > > > > > 4) Somewhat related to the previous point: usb-fuzzer uses predictable > > > endpoint names across different UDCs. > > > > > > Right now each UDC driver defines endpoint names via EP_INFO() as it > > > pleases. And GadgetFS uses those names to create file entries for each > > > of the endpoints. As a result, endpoint names for different UDCs will > > > be different and it requires more work to write a portable userspace > > > gadget. The usb-fuzzer interface auto selects and assigns an endpoint > > > based on the required features like the transfer type. > > > > > > 5) GadgetFS binds to the first available UDC, usb-fuzzer provides a > > > way to select a UDC to bind to. > > > > > > Since the fuzzing happens in multiple processes each of which has its > > > own Dummy UDC assigned, we want to have control over which UDC we bind > > > to. This part is a bit confusing, but what I found out is that a UDC > > > is selected based on two different identifying names. I call the first > > > one "udc_device_name" and the second one "udc_driver_name". > > > "udc_device_name" has to be assigned to usb_gadget_driver->udc_name > > > when usb_gadget_probe_driver() is called, and "udc_driver_name" is > > > what we have to compare usb_gadget->name with inside of the > > > usb_gadget_driver->bind() callback. For example, Dummy UDC has > > > "dummy_udc" as its "udc_driver_name" and "dummy_udc.N" as its > > > "udc_device_name". At the same time the dwc2 driver that is used on > > > Raspberry Pi Zero, has "20980000.usb" as both "udc_driver_name" and > > > "udc_device_name". > > > > > > Overall, the usb-fuzzer interface implementation has a similar > > > structure to that of GadgetFS, but looks way simpler (although that > > > might be because I've missed to implement some functionality :). > > > > > > We'd like to get this upstreamed, but I'm not sure whether this should > > > be a separate interface (which we can rebrand as a raw usb gadget > > > interface or something like that) or we should try to make it a > > > special mode of GadgetFS. I like the former approach more, as GadgetFS > > > looks quite complicated from my point of view and making fundamental > > > changes to it doesn't seem like an easy task. This is where we'd like > > > to get your input. > > > > I'll leave the gadgetfs questions to others that know that api better > > than I do :) > > Assuming the usb-fuzzer implementation gets merged, I think it > definitely should be kept separate from gadgetfs. The differences in > functionality are too great; it doesn't make sense to combine them in > a single driver. > > Also, while it often is reasonable for people to build gadgetfs for > production use, usb-fuzzer is very much more of a developer/debugger > type of thing. OK, I'll clean up the patches and send them as an RFC. Thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-19 8:35 ` Greg Kroah-Hartman 2019-04-19 14:39 ` Alan Stern @ 2019-04-24 16:05 ` Andrey Konovalov 2019-04-25 12:44 ` Andrey Konovalov 1 sibling, 1 reply; 9+ messages in thread From: Andrey Konovalov @ 2019-04-24 16:05 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Alan Stern, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Fri, Apr 19, 2019 at 10:35 AM Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > On Wed, Apr 17, 2019 at 06:33:39PM +0200, Andrey Konovalov wrote: > > Hi, > > > > As you might have noticed, syzbot has started reporting bugs in the > > USB subsystem that can be triggered externally by a malicious USB > > device. Right now the fuzzing is done via a GadgetFS-like interface to > > emulate USB devices through the USB Gadget subsystem and the Dummy > > HCD/UDC module to "loopback" connect those devices to the running > > kernel. There are more details in my OffensiveCon talk [1], [2]. > > > > A few questions/comments: > > > > 1. Which tree should we use for testing? > > > > Right now we're targeting the upstream tree, but we can switch to some > > USB development tree, where the fixes are likely to end up earlier. > > USB fixes land in the usb.git tree on git.kernel.org in the usb-testing > branch (for the next release), and in the usb-linus branch (for this > release). OK, I've switched to the usb-linus branch of the usb.git tree, since I see that all Alan's fixes went in that one. > > So you can just merge those two branches together if you like to be sure > you have all of the latest fixes. > > > 2. Is there an easy way to figure out which config options enable > > drivers reachable over USB? > > Looking for all options that depend on USB is a good start. > > > Right now our kernel config is based on one of the Debian kernel > > configs, that supposedly enables enough relevant USB drivers. At the > > same time it enables a lot of other unnecessary stuff, which makes the > > kernel big and long to compile. Ideally, we would to have a way to > > auto-generate a kernel config that enables all the relevant (enabled > > by at least one of the distros) USB drivers. I've looked at whether > > it's possible to figure out which particular options in some kernel > > config are related to USB, but it seems that neither the option names, > > nor the way they are grouped in the config file, are representative > > enough. > > Yeah, it's hard to just carve out this type of configuration, but here's > what I have done in the past to try to be sure I enabled all USB drivers > in my kernel configuration. > > First, start with a "minimally working configuration" by running: > make localmodconfig > on a working system, with the needed modules for booting and operating > properly already loaded. > > That gives you a .config file that should take only minutes to build, > compared to much longer for the normal distro configuration (also be > sure to disable some debugging options so you don't spend extra time > building and stripping symbols). > > Boot and make sure that configuration works. > > Then, take that .config and do: > - disable USB from the configuration by deleting the: > CONFIG_USB_SUPPORT=y > option from your .config > - run 'make oldconfig' to disable all USB drivers > - turn USB back on by setting CONFIG_USB_SUPPORT=y back on in > your .config > - run 'make oldconfig' and answer 'y' or 'm' to all of the > driver options you are presented with. > > That usually catches almost all of them. Sometimes you need to make > sure you have some other subsystem enabled (like SCSI), but odds are, if > you start with a "normally stripped down" configuration that works, you > should be fine. I suspect that make localmodconfig (+ switching CONFIG_USB_SUPPORT off and on) would likely include a lot of stuff that we don't need (there are many options that are =y, but not related to USB at all), but it definitely sounds better than what I have right now (converting almost all =m into =y). I'll give it a shot, thanks! > > > 3. Regarding that GadgetFS-like interface. > > > > Initially I was using GadgetFS (together with the Dummy HCD/UDC > > module) to perform emulation of USB devices for fuzzing, but later > > switched to a custom written interface. This interface is essentially > > implemented in the following patch [3]. An example that emulates a USB > > keyboard through this interface can be found here [4]. And the > > syzkaller parts responsible for USB fuzzing are here [5], [6]. The > > incentive to implement a different interface was to provide a somewhat > > raw and direct access to the USB Gadget layer for the userspace, where > > every USB request is passed to the userspace to get a response. > > > > The main differences between this interface (referred to as usb-fuzzer > > for now) and GadgetFS are: > > > > 1) GadgetFS does some sanity checks on the provided USB descriptors, > > which is something we don't want for fuzzing. We want the descriptors > > to be as corrupted as they can. > > > > 2) GadgetFS handles some of the USB requests internally based on the > > provided device descriptor, which is also something we don't want. For > > example we may want to be able to provide differently corrupted > > responses to the same request. > > > > 3) usb-fuzzer has ioctl-based interface instead of a filesystem-based > > one. I wouldn't say it's that big of a deal, but it makes it somewhat > > easier to incorporate into a fuzzer. > > > > 4) Somewhat related to the previous point: usb-fuzzer uses predictable > > endpoint names across different UDCs. > > > > Right now each UDC driver defines endpoint names via EP_INFO() as it > > pleases. And GadgetFS uses those names to create file entries for each > > of the endpoints. As a result, endpoint names for different UDCs will > > be different and it requires more work to write a portable userspace > > gadget. The usb-fuzzer interface auto selects and assigns an endpoint > > based on the required features like the transfer type. > > > > 5) GadgetFS binds to the first available UDC, usb-fuzzer provides a > > way to select a UDC to bind to. > > > > Since the fuzzing happens in multiple processes each of which has its > > own Dummy UDC assigned, we want to have control over which UDC we bind > > to. This part is a bit confusing, but what I found out is that a UDC > > is selected based on two different identifying names. I call the first > > one "udc_device_name" and the second one "udc_driver_name". > > "udc_device_name" has to be assigned to usb_gadget_driver->udc_name > > when usb_gadget_probe_driver() is called, and "udc_driver_name" is > > what we have to compare usb_gadget->name with inside of the > > usb_gadget_driver->bind() callback. For example, Dummy UDC has > > "dummy_udc" as its "udc_driver_name" and "dummy_udc.N" as its > > "udc_device_name". At the same time the dwc2 driver that is used on > > Raspberry Pi Zero, has "20980000.usb" as both "udc_driver_name" and > > "udc_device_name". > > > > Overall, the usb-fuzzer interface implementation has a similar > > structure to that of GadgetFS, but looks way simpler (although that > > might be because I've missed to implement some functionality :). > > > > We'd like to get this upstreamed, but I'm not sure whether this should > > be a separate interface (which we can rebrand as a raw usb gadget > > interface or something like that) or we should try to make it a > > special mode of GadgetFS. I like the former approach more, as GadgetFS > > looks quite complicated from my point of view and making fundamental > > changes to it doesn't seem like an easy task. This is where we'd like > > to get your input. > > I'll leave the gadgetfs questions to others that know that api better > than I do :) > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-24 16:05 ` Andrey Konovalov @ 2019-04-25 12:44 ` Andrey Konovalov 2019-04-25 14:25 ` Greg Kroah-Hartman 0 siblings, 1 reply; 9+ messages in thread From: Andrey Konovalov @ 2019-04-25 12:44 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Alan Stern, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Wed, Apr 24, 2019 at 6:05 PM Andrey Konovalov <andreyknvl@google.com> wrote: > > On Fri, Apr 19, 2019 at 10:35 AM Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: > > > > > 2. Is there an easy way to figure out which config options enable > > > drivers reachable over USB? > > > > Looking for all options that depend on USB is a good start. > > > > > Right now our kernel config is based on one of the Debian kernel > > > configs, that supposedly enables enough relevant USB drivers. At the > > > same time it enables a lot of other unnecessary stuff, which makes the > > > kernel big and long to compile. Ideally, we would to have a way to > > > auto-generate a kernel config that enables all the relevant (enabled > > > by at least one of the distros) USB drivers. I've looked at whether > > > it's possible to figure out which particular options in some kernel > > > config are related to USB, but it seems that neither the option names, > > > nor the way they are grouped in the config file, are representative > > > enough. > > > > Yeah, it's hard to just carve out this type of configuration, but here's > > what I have done in the past to try to be sure I enabled all USB drivers > > in my kernel configuration. > > > > First, start with a "minimally working configuration" by running: > > make localmodconfig > > on a working system, with the needed modules for booting and operating > > properly already loaded. > > > > That gives you a .config file that should take only minutes to build, > > compared to much longer for the normal distro configuration (also be > > sure to disable some debugging options so you don't spend extra time > > building and stripping symbols). > > > > Boot and make sure that configuration works. > > > > Then, take that .config and do: > > - disable USB from the configuration by deleting the: > > CONFIG_USB_SUPPORT=y > > option from your .config > > - run 'make oldconfig' to disable all USB drivers > > - turn USB back on by setting CONFIG_USB_SUPPORT=y back on in > > your .config > > - run 'make oldconfig' and answer 'y' or 'm' to all of the > > driver options you are presented with. > > > > That usually catches almost all of them. Sometimes you need to make > > sure you have some other subsystem enabled (like SCSI), but odds are, if > > you start with a "normally stripped down" configuration that works, you > > should be fine. > > I suspect that make localmodconfig (+ switching CONFIG_USB_SUPPORT off > and on) would likely include a lot of stuff that we don't need (there > are many options that are =y, but not related to USB at all), but it > definitely sounds better than what I have right now (converting almost > all =m into =y). I'll give it a shot, thanks! I've tried this and unfortunately it doesn't work as desired. The reason is that localmodconfig will only enable options for the modules that are currently loaded, and if a module that some USB driver depends on is not loaded, then this driver won't be enabled after yes | make oldconfig. For example my machine didn't have the cfg80211 module loaded, and thus e.g. CONFIG_AT76C50X_USB didn't get enabled after oldconfig. However when I plug in a wireless USB adapter, cfg80211 gets loaded together with the USB driver for that adapter. I guess the same applies to other kinds of dependency modules (e.g. bluetooth). So this would only work if all the dependency modules are already loaded. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-25 12:44 ` Andrey Konovalov @ 2019-04-25 14:25 ` Greg Kroah-Hartman 2019-05-14 12:43 ` Andrey Konovalov 0 siblings, 1 reply; 9+ messages in thread From: Greg Kroah-Hartman @ 2019-04-25 14:25 UTC (permalink / raw) To: Andrey Konovalov Cc: Alan Stern, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Thu, Apr 25, 2019 at 02:44:11PM +0200, Andrey Konovalov wrote: > On Wed, Apr 24, 2019 at 6:05 PM Andrey Konovalov <andreyknvl@google.com> wrote: > > > > On Fri, Apr 19, 2019 at 10:35 AM Greg Kroah-Hartman > > <gregkh@linuxfoundation.org> wrote: > > > > > > > 2. Is there an easy way to figure out which config options enable > > > > drivers reachable over USB? > > > > > > Looking for all options that depend on USB is a good start. > > > > > > > Right now our kernel config is based on one of the Debian kernel > > > > configs, that supposedly enables enough relevant USB drivers. At the > > > > same time it enables a lot of other unnecessary stuff, which makes the > > > > kernel big and long to compile. Ideally, we would to have a way to > > > > auto-generate a kernel config that enables all the relevant (enabled > > > > by at least one of the distros) USB drivers. I've looked at whether > > > > it's possible to figure out which particular options in some kernel > > > > config are related to USB, but it seems that neither the option names, > > > > nor the way they are grouped in the config file, are representative > > > > enough. > > > > > > Yeah, it's hard to just carve out this type of configuration, but here's > > > what I have done in the past to try to be sure I enabled all USB drivers > > > in my kernel configuration. > > > > > > First, start with a "minimally working configuration" by running: > > > make localmodconfig > > > on a working system, with the needed modules for booting and operating > > > properly already loaded. > > > > > > That gives you a .config file that should take only minutes to build, > > > compared to much longer for the normal distro configuration (also be > > > sure to disable some debugging options so you don't spend extra time > > > building and stripping symbols). > > > > > > Boot and make sure that configuration works. > > > > > > Then, take that .config and do: > > > - disable USB from the configuration by deleting the: > > > CONFIG_USB_SUPPORT=y > > > option from your .config > > > - run 'make oldconfig' to disable all USB drivers > > > - turn USB back on by setting CONFIG_USB_SUPPORT=y back on in > > > your .config > > > - run 'make oldconfig' and answer 'y' or 'm' to all of the > > > driver options you are presented with. > > > > > > That usually catches almost all of them. Sometimes you need to make > > > sure you have some other subsystem enabled (like SCSI), but odds are, if > > > you start with a "normally stripped down" configuration that works, you > > > should be fine. > > > > I suspect that make localmodconfig (+ switching CONFIG_USB_SUPPORT off > > and on) would likely include a lot of stuff that we don't need (there > > are many options that are =y, but not related to USB at all), but it > > definitely sounds better than what I have right now (converting almost > > all =m into =y). I'll give it a shot, thanks! > > I've tried this and unfortunately it doesn't work as desired. The > reason is that localmodconfig will only enable options for the modules > that are currently loaded, and if a module that some USB driver > depends on is not loaded, then this driver won't be enabled after yes > | make oldconfig. For example my machine didn't have the cfg80211 > module loaded, and thus e.g. CONFIG_AT76C50X_USB didn't get enabled > after oldconfig. However when I plug in a wireless USB adapter, > cfg80211 gets loaded together with the USB driver for that adapter. I > guess the same applies to other kinds of dependency modules (e.g. > bluetooth). So this would only work if all the dependency modules are > already loaded. Yes, sorry, I thought I said that with: > > > First, start with a "minimally working configuration" by running: > > > make localmodconfig > > > on a working system, with the needed modules for booting and operating > > > properly already loaded. I guess "working system" implied everything that you _knew_ you wanted to have loaded :) Sorry about the dependancy mess, hopefully you have sorted this out better now. thanks, greg k-h ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-25 14:25 ` Greg Kroah-Hartman @ 2019-05-14 12:43 ` Andrey Konovalov 0 siblings, 0 replies; 9+ messages in thread From: Andrey Konovalov @ 2019-05-14 12:43 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Alan Stern, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko From: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Date: Thu, Apr 25, 2019 at 4:25 PM To: Andrey Konovalov Cc: Alan Stern, Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko > On Thu, Apr 25, 2019 at 02:44:11PM +0200, Andrey Konovalov wrote: > > On Wed, Apr 24, 2019 at 6:05 PM Andrey Konovalov <andreyknvl@google.com> wrote: > > > > > > On Fri, Apr 19, 2019 at 10:35 AM Greg Kroah-Hartman > > > <gregkh@linuxfoundation.org> wrote: > > > > > > > > > 2. Is there an easy way to figure out which config options enable > > > > > drivers reachable over USB? > > > > > > > > Looking for all options that depend on USB is a good start. > > > > > > > > > Right now our kernel config is based on one of the Debian kernel > > > > > configs, that supposedly enables enough relevant USB drivers. At the > > > > > same time it enables a lot of other unnecessary stuff, which makes the > > > > > kernel big and long to compile. Ideally, we would to have a way to > > > > > auto-generate a kernel config that enables all the relevant (enabled > > > > > by at least one of the distros) USB drivers. I've looked at whether > > > > > it's possible to figure out which particular options in some kernel > > > > > config are related to USB, but it seems that neither the option names, > > > > > nor the way they are grouped in the config file, are representative > > > > > enough. > > > > > > > > Yeah, it's hard to just carve out this type of configuration, but here's > > > > what I have done in the past to try to be sure I enabled all USB drivers > > > > in my kernel configuration. > > > > > > > > First, start with a "minimally working configuration" by running: > > > > make localmodconfig > > > > on a working system, with the needed modules for booting and operating > > > > properly already loaded. > > > > > > > > That gives you a .config file that should take only minutes to build, > > > > compared to much longer for the normal distro configuration (also be > > > > sure to disable some debugging options so you don't spend extra time > > > > building and stripping symbols). > > > > > > > > Boot and make sure that configuration works. > > > > > > > > Then, take that .config and do: > > > > - disable USB from the configuration by deleting the: > > > > CONFIG_USB_SUPPORT=y > > > > option from your .config > > > > - run 'make oldconfig' to disable all USB drivers > > > > - turn USB back on by setting CONFIG_USB_SUPPORT=y back on in > > > > your .config > > > > - run 'make oldconfig' and answer 'y' or 'm' to all of the > > > > driver options you are presented with. > > > > > > > > That usually catches almost all of them. Sometimes you need to make > > > > sure you have some other subsystem enabled (like SCSI), but odds are, if > > > > you start with a "normally stripped down" configuration that works, you > > > > should be fine. > > > > > > I suspect that make localmodconfig (+ switching CONFIG_USB_SUPPORT off > > > and on) would likely include a lot of stuff that we don't need (there > > > are many options that are =y, but not related to USB at all), but it > > > definitely sounds better than what I have right now (converting almost > > > all =m into =y). I'll give it a shot, thanks! > > > > I've tried this and unfortunately it doesn't work as desired. The > > reason is that localmodconfig will only enable options for the modules > > that are currently loaded, and if a module that some USB driver > > depends on is not loaded, then this driver won't be enabled after yes > > | make oldconfig. For example my machine didn't have the cfg80211 > > module loaded, and thus e.g. CONFIG_AT76C50X_USB didn't get enabled > > after oldconfig. However when I plug in a wireless USB adapter, > > cfg80211 gets loaded together with the USB driver for that adapter. I > > guess the same applies to other kinds of dependency modules (e.g. > > bluetooth). So this would only work if all the dependency modules are > > already loaded. > > Yes, sorry, I thought I said that with: > > > > > First, start with a "minimally working configuration" by running: > > > > make localmodconfig > > > > on a working system, with the needed modules for booting and operating > > > > properly already loaded. > > I guess "working system" implied everything that you _knew_ you wanted > to have loaded :) > > Sorry about the dependancy mess, hopefully you have sorted this out > better now. I've written a script [1], [2] on top of Kconfiglib [3] that merges in all USB configs and their dependencies from a provided (distro) config. The dependency extraction a somewhat best effort, but seems to be working. Maybe you'll find some use for it as well. Thanks! [1] https://github.com/google/syzkaller/blob/master/dashboard/config/kconfiglib-merge-usb-configs.py [2] https://github.com/google/syzkaller/blob/master/dashboard/config/generate-config-usb.sh [3] https://github.com/ulfalizer/Kconfiglib ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: USB fuzzing with syzbot 2019-04-17 16:33 USB fuzzing with syzbot Andrey Konovalov 2019-04-19 8:35 ` Greg Kroah-Hartman @ 2019-08-13 13:06 ` Andrey Konovalov 1 sibling, 0 replies; 9+ messages in thread From: Andrey Konovalov @ 2019-08-13 13:06 UTC (permalink / raw) To: Alan Stern, Greg Kroah-Hartman, Oliver Neukum, Hillf Danton Cc: Gustavo A. R. Silva, USB list, Dmitry Vyukov, Kostya Serebryany, Alexander Potapenko On Wed, Apr 17, 2019 at 6:33 PM Andrey Konovalov <andreyknvl@google.com> wrote: > > Hi, > > As you might have noticed, syzbot has started reporting bugs in the > USB subsystem that can be triggered externally by a malicious USB > device. Right now the fuzzing is done via a GadgetFS-like interface to > emulate USB devices through the USB Gadget subsystem and the Dummy > HCD/UDC module to "loopback" connect those devices to the running > kernel. There are more details in my OffensiveCon talk [1], [2]. > > A few questions/comments: > > 1. Which tree should we use for testing? > > Right now we're targeting the upstream tree, but we can switch to some > USB development tree, where the fixes are likely to end up earlier. > > 2. Is there an easy way to figure out which config options enable > drivers reachable over USB? > > Right now our kernel config is based on one of the Debian kernel > configs, that supposedly enables enough relevant USB drivers. At the > same time it enables a lot of other unnecessary stuff, which makes the > kernel big and long to compile. Ideally, we would to have a way to > auto-generate a kernel config that enables all the relevant (enabled > by at least one of the distros) USB drivers. I've looked at whether > it's possible to figure out which particular options in some kernel > config are related to USB, but it seems that neither the option names, > nor the way they are grouped in the config file, are representative > enough. > > 3. Regarding that GadgetFS-like interface. > > Initially I was using GadgetFS (together with the Dummy HCD/UDC > module) to perform emulation of USB devices for fuzzing, but later > switched to a custom written interface. This interface is essentially > implemented in the following patch [3]. An example that emulates a USB > keyboard through this interface can be found here [4]. And the > syzkaller parts responsible for USB fuzzing are here [5], [6]. The > incentive to implement a different interface was to provide a somewhat > raw and direct access to the USB Gadget layer for the userspace, where > every USB request is passed to the userspace to get a response. > > The main differences between this interface (referred to as usb-fuzzer > for now) and GadgetFS are: > > 1) GadgetFS does some sanity checks on the provided USB descriptors, > which is something we don't want for fuzzing. We want the descriptors > to be as corrupted as they can. > > 2) GadgetFS handles some of the USB requests internally based on the > provided device descriptor, which is also something we don't want. For > example we may want to be able to provide differently corrupted > responses to the same request. > > 3) usb-fuzzer has ioctl-based interface instead of a filesystem-based > one. I wouldn't say it's that big of a deal, but it makes it somewhat > easier to incorporate into a fuzzer. > > 4) Somewhat related to the previous point: usb-fuzzer uses predictable > endpoint names across different UDCs. > > Right now each UDC driver defines endpoint names via EP_INFO() as it > pleases. And GadgetFS uses those names to create file entries for each > of the endpoints. As a result, endpoint names for different UDCs will > be different and it requires more work to write a portable userspace > gadget. The usb-fuzzer interface auto selects and assigns an endpoint > based on the required features like the transfer type. > > 5) GadgetFS binds to the first available UDC, usb-fuzzer provides a > way to select a UDC to bind to. > > Since the fuzzing happens in multiple processes each of which has its > own Dummy UDC assigned, we want to have control over which UDC we bind > to. This part is a bit confusing, but what I found out is that a UDC > is selected based on two different identifying names. I call the first > one "udc_device_name" and the second one "udc_driver_name". > "udc_device_name" has to be assigned to usb_gadget_driver->udc_name > when usb_gadget_probe_driver() is called, and "udc_driver_name" is > what we have to compare usb_gadget->name with inside of the > usb_gadget_driver->bind() callback. For example, Dummy UDC has > "dummy_udc" as its "udc_driver_name" and "dummy_udc.N" as its > "udc_device_name". At the same time the dwc2 driver that is used on > Raspberry Pi Zero, has "20980000.usb" as both "udc_driver_name" and > "udc_device_name". > > Overall, the usb-fuzzer interface implementation has a similar > structure to that of GadgetFS, but looks way simpler (although that > might be because I've missed to implement some functionality :). > > We'd like to get this upstreamed, but I'm not sure whether this should > be a separate interface (which we can rebrand as a raw usb gadget > interface or something like that) or we should try to make it a > special mode of GadgetFS. I like the former approach more, as GadgetFS > looks quite complicated from my point of view and making fundamental > changes to it doesn't seem like an easy task. This is where we'd like > to get your input. > > If you have any other questions or comments, please let us know. > > Alan, thanks a lot for answering to lots of my USB related questions > while I was working on this, that helped tremendously! > > Thanks! > > [1] https://docs.google.com/presentation/d/1z-giB9kom17Lk21YEjmceiNUVYeI6yIaG5_gZ3vKC-M/edit#slide=id.g1925acbbf3_0_0 > > [2] https://www.youtube.com/watch?v=1MD5JV6LfxA > > [3] https://github.com/google/kasan/commit/9a33b36996cb6138a806eab931ba43c63a836708 > > [4] https://github.com/google/syzkaller/blob/usb-fuzzer/tools/usb/keyboard.c > > [5] https://github.com/google/syzkaller/blob/master/sys/linux/vusb.txt > > [6] https://github.com/google/syzkaller/blob/master/executor/common_usb.h A small update: all bugs found on the USB fuzzing syzbot instance can now be viewed here: https://syzkaller.appspot.com/upstream?manager=ci2-upstream-usb ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-08-13 13:06 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-17 16:33 USB fuzzing with syzbot Andrey Konovalov 2019-04-19 8:35 ` Greg Kroah-Hartman 2019-04-19 14:39 ` Alan Stern 2019-04-24 16:09 ` Andrey Konovalov 2019-04-24 16:05 ` Andrey Konovalov 2019-04-25 12:44 ` Andrey Konovalov 2019-04-25 14:25 ` Greg Kroah-Hartman 2019-05-14 12:43 ` Andrey Konovalov 2019-08-13 13:06 ` Andrey Konovalov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).