REGRESSION: dwc2: gadget: Add scatter-gather mode

REGRESSION: dwc2: gadget: Add scatter-gather mode

[John Stultz]

Hey Andrzej,
  I just wanted to let you know, trying to boot linus/master on the
hikey board today, I've been seeing lots of errors on boot (which
sometimes crash the board, and sometimes doesn't). See the example
below.

I've bisected the issue down to 10209abe87f5 ("usb: dwc2: gadget: Add
scatter-gather mode"), and if I revert that change the issue goes
away.

Looking at the patch, I can't see anything obviously sticking out, but
let me know if you have anything suggestions you'd like me to test.

thanks
-john


[   13.088934] functionfs read size 512 > requested size 24, splitting
request into multiple reads.
[   13.089029] ------------[ cut here ]------------
[   13.102665] Trying to vfree() bad address (00000000ecf3404f)
[   13.108436] WARNING: CPU: 0 PID: 2014 at mm/vmalloc.c:1516 __vunmap+0xe0/0xe8
[   13.115587] CPU: 0 PID: 2014 Comm: adbd Not tainted
5.0.0-08291-ga032141-dirty #897
[   13.123244] Hardware name: HiKey Development Board (DT)
[   13.128469] pstate: 80400005 (Nzcv daif +PAN -UAO)
[   13.133266] pc : __vunmap+0xe0/0xe8
[   13.136757] lr : __vunmap+0xe0/0xe8
[   13.140243] sp : ffffff80114d3b40
[   13.143555] x29: ffffff80114d3b40 x28: ffffffc07333d280
[   13.148868] x27: ffffffc071b67a38 x26: 00000000000001e8
[   13.154180] x25: ffffff80114d3ca8 x24: ffffffc07190e400
[   13.159492] x23: ffffffc0748bca00 x22: ffffff8010ffd000
[   13.164803] x21: 0000000000000001 x20: ffffff80114d3c98
[   13.170113] x19: ffffff8010414e74 x18: ffffff8010ffda48
[   13.175423] x17: 0000000000000000 x16: 0000000000000000
[   13.180734] x15: ffffff80914d3867 x14: 0000000000000006
[   13.186044] x13: ffffff80114d3875 x12: ffffff801101c898
[   13.191371] x11: ffffff801101c000 x10: 0000000005f5e0ff
[   13.191416] type=1400 audit(16.743:79): avc: denied { read } for
comm="drmserver" name="enabled" dev="sysfs" ino=5492
scontext=u:r:drmserver:s0 tcont0
[   13.196698] x9 : ffffff80114d3800 x8 : 2966343034336663
[   13.196707] x7 : 6530303030303030 x6 : 0000000000000424
[   13.196713] x5 : 0000000000000000 x4 : 0000000000000000
[   13.196718] x3 : 0000000000000001 x2 : 0000000000000001
[   13.196724] x1 : b357113b3815f700 x0 : 0000000000000000
[   13.214445] type=1400 audit(17.075:84): avc: denied { read } for
comm="audioserver" name="enabled" dev="sysfs" ino=5492
scontext=u:r:audioserver:s0 t0
[   13.219604] Call trace:
[   13.219616]  __vunmap+0xe0/0xe8
[   13.219623]  __vfree+0x24/0x70
[   13.219635]  vfree+0x20/0x38
[   13.270258]  ffs_epfile_io.isra.12+0x190/0x6b0
[   13.274707]  ffs_epfile_read_iter+0xa0/0x168
[   13.278982]  __vfs_read+0x10c/0x168
[   13.282471]  vfs_read+0x8c/0x148
[   13.285700]  ksys_read+0x5c/0xc8
[   13.288930]  __arm64_sys_read+0x14/0x20
[   13.292773]  el0_svc_common.constprop.0+0xb0/0x110
[   13.297461] type=1400 audit(17.075:84): avc: denied { read } for
comm="audioserver" name="enabled" dev="sysfs" ino=5492
scontext=u:r:audioserver:s0 t0
[   13.297570]  el0_svc_handler+0x28/0x78
[   13.315592] type=1400 audit(17.179:85): avc: denied { read } for
comm="bootanimation" name="enabled" dev="sysfs" ino=5492
scontext=u:r:bootanim:s0 tc0
[   13.319239]  el0_svc+0x8/0xc
[   13.319245] ---[ end trace 729ab9edfbdb0764 ]---

Re: REGRESSION: dwc2: gadget: Add scatter-gather mode

[Minas Harutyunyan]

Hi Andrzej,

USB CV MSC tests failed starting from Test Case 6 with BNA interrupt on 
ep1in. It's first BULK IN transaction after GET MAXLUN.

[319523.955339] dwc2 dwc2.1.auto: dwc2_hsotg_irq: 04088028 00080000 
(d88c3cc4) retry 8
[319523.955357] dwc2 dwc2.1.auto: dwc2_hsotg_irq: daint=00020000
[319523.955366] dwc2 dwc2.1.auto: dwc2_hsotg_epint: ep1(out) 
DxEPINT=0x00000001
[319523.955372] dwc2 dwc2.1.auto: dwc2_hsotg_epint: XferCompl: 
DxEPCTL=0x00098200, DXEPTSIZ=15c7d486
[319523.955377] dwc2 dwc2.1.auto: complete: ep 00000000b58e703a ep1out, 
req 00000000bde78704, 0 => 000000002620ceb8
[319523.955396] dwc2 dwc2.1.auto: ep1in: req 00000000acdadb34: 
36@00000000450496f9, noi=0, zero=0, snok=0
[319523.955403] dwc2 dwc2.1.auto: dwc2_hsotg_start_req: 
DxEPCTL=0x004a8200, ep 1, dir in
[319523.955406] dwc2 dwc2.1.auto: ureq->length:36 ureq->actual:0
[319523.955408] dwc2 dwc2.1.auto: dwc2_hsotg_start_req: 1@36/36, 
0x20080024 => 0x00000930
[319523.955410] dwc2 dwc2.1.auto: dwc2_hsotg_start_req: 36cbc000 pad => 
0x00000934
[319523.955412] dwc2 dwc2.1.auto: ep0 state:0
[319523.955413] dwc2 dwc2.1.auto: dwc2_hsotg_start_req: DxEPCTL=0x844a8200
[319523.955420] dwc2 dwc2.1.auto: dwc2_hsotg_start_req: DXEPCTL=0x80488200
[319523.955425] dwc2 dwc2.1.auto: ep1in: req 00000000f74873bc: 
13@000000007a860fd2, noi=0, zero=0, snok=0
[319523.955446] dwc2 dwc2.1.auto: dwc2_hsotg_irq: 04048028 00040000 
(d88c3cc4) retry 8
[319523.955451] dwc2 dwc2.1.auto: dwc2_hsotg_irq: daint=00000002
[319523.955460] dwc2 dwc2.1.auto: dwc2_hsotg_epint: ep1(in) 
DxEPINT=0x00000200
[319523.955462] dwc2 dwc2.1.auto: dwc2_hsotg_epint: BNA interrupt
[319553.987219] dwc2 dwc2.1.auto: dwc2_hsotg_irq: 04008428 00000400 
(d88c3cc4) retry 8

Meanwhile it's passing smoke test with mass storage function.

Thanks,
Minas


On 3/9/2019 12:53 AM, John Stultz wrote:
> Hey Andrzej,
>    I just wanted to let you know, trying to boot linus/master on the
> hikey board today, I've been seeing lots of errors on boot (which
> sometimes crash the board, and sometimes doesn't). See the example
> below.
> 
> I've bisected the issue down to 10209abe87f5 ("usb: dwc2: gadget: Add
> scatter-gather mode"), and if I revert that change the issue goes
> away.
> 
> Looking at the patch, I can't see anything obviously sticking out, but
> let me know if you have anything suggestions you'd like me to test.
> 
> thanks
> -john
> 
> 
> [   13.088934] functionfs read size 512 > requested size 24, splitting
> request into multiple reads.
> [   13.089029] ------------[ cut here ]------------
> [   13.102665] Trying to vfree() bad address (00000000ecf3404f)
> [   13.108436] WARNING: CPU: 0 PID: 2014 at mm/vmalloc.c:1516 __vunmap+0xe0/0xe8
> [   13.115587] CPU: 0 PID: 2014 Comm: adbd Not tainted
> 5.0.0-08291-ga032141-dirty #897
> [   13.123244] Hardware name: HiKey Development Board (DT)
> [   13.128469] pstate: 80400005 (Nzcv daif +PAN -UAO)
> [   13.133266] pc : __vunmap+0xe0/0xe8
> [   13.136757] lr : __vunmap+0xe0/0xe8
> [   13.140243] sp : ffffff80114d3b40
> [   13.143555] x29: ffffff80114d3b40 x28: ffffffc07333d280
> [   13.148868] x27: ffffffc071b67a38 x26: 00000000000001e8
> [   13.154180] x25: ffffff80114d3ca8 x24: ffffffc07190e400
> [   13.159492] x23: ffffffc0748bca00 x22: ffffff8010ffd000
> [   13.164803] x21: 0000000000000001 x20: ffffff80114d3c98
> [   13.170113] x19: ffffff8010414e74 x18: ffffff8010ffda48
> [   13.175423] x17: 0000000000000000 x16: 0000000000000000
> [   13.180734] x15: ffffff80914d3867 x14: 0000000000000006
> [   13.186044] x13: ffffff80114d3875 x12: ffffff801101c898
> [   13.191371] x11: ffffff801101c000 x10: 0000000005f5e0ff
> [   13.191416] type=1400 audit(16.743:79): avc: denied { read } for
> comm="drmserver" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:drmserver:s0 tcont0
> [   13.196698] x9 : ffffff80114d3800 x8 : 2966343034336663
> [   13.196707] x7 : 6530303030303030 x6 : 0000000000000424
> [   13.196713] x5 : 0000000000000000 x4 : 0000000000000000
> [   13.196718] x3 : 0000000000000001 x2 : 0000000000000001
> [   13.196724] x1 : b357113b3815f700 x0 : 0000000000000000
> [   13.214445] type=1400 audit(17.075:84): avc: denied { read } for
> comm="audioserver" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:audioserver:s0 t0
> [   13.219604] Call trace:
> [   13.219616]  __vunmap+0xe0/0xe8
> [   13.219623]  __vfree+0x24/0x70
> [   13.219635]  vfree+0x20/0x38
> [   13.270258]  ffs_epfile_io.isra.12+0x190/0x6b0
> [   13.274707]  ffs_epfile_read_iter+0xa0/0x168
> [   13.278982]  __vfs_read+0x10c/0x168
> [   13.282471]  vfs_read+0x8c/0x148
> [   13.285700]  ksys_read+0x5c/0xc8
> [   13.288930]  __arm64_sys_read+0x14/0x20
> [   13.292773]  el0_svc_common.constprop.0+0xb0/0x110
> [   13.297461] type=1400 audit(17.075:84): avc: denied { read } for
> comm="audioserver" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:audioserver:s0 t0
> [   13.297570]  el0_svc_handler+0x28/0x78
> [   13.315592] type=1400 audit(17.179:85): avc: denied { read } for
> comm="bootanimation" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:bootanim:s0 tc0
> [   13.319239]  el0_svc+0x8/0xc
> [   13.319245] ---[ end trace 729ab9edfbdb0764 ]---
> 

Re: REGRESSION: dwc2: gadget: Add scatter-gather mode

[Marek Szyprowski]

Hi John


On 2019-03-08 21:53, John Stultz wrote:
> Hey Andrzej,
>   I just wanted to let you know, trying to boot linus/master on the
> hikey board today, I've been seeing lots of errors on boot (which
> sometimes crash the board, and sometimes doesn't). See the example
> below.
>
> I've bisected the issue down to 10209abe87f5 ("usb: dwc2: gadget: Add
> scatter-gather mode"), and if I revert that change the issue goes
> away.
>
> Looking at the patch, I can't see anything obviously sticking out, but
> let me know if you have anything suggestions you'd like me to test.

I've CCed new Andrzej's email.

Could you tell a bit more how to reproduce this issue? Which USB
function are you running on the target system? I've noticed that you are
using functionfs. Is is possible to get access to the usb function
daemon to reproduce the problem here? Which version of HiKey board are
you using?


> thanks
> -john
>
>
> [   13.088934] functionfs read size 512 > requested size 24, splitting
> request into multiple reads.
> [   13.089029] ------------[ cut here ]------------
> [   13.102665] Trying to vfree() bad address (00000000ecf3404f)
> [   13.108436] WARNING: CPU: 0 PID: 2014 at mm/vmalloc.c:1516 __vunmap+0xe0/0xe8
> [   13.115587] CPU: 0 PID: 2014 Comm: adbd Not tainted
> 5.0.0-08291-ga032141-dirty #897
> [   13.123244] Hardware name: HiKey Development Board (DT)
> [   13.128469] pstate: 80400005 (Nzcv daif +PAN -UAO)
> [   13.133266] pc : __vunmap+0xe0/0xe8
> [   13.136757] lr : __vunmap+0xe0/0xe8
> [   13.140243] sp : ffffff80114d3b40
> [   13.143555] x29: ffffff80114d3b40 x28: ffffffc07333d280
> [   13.148868] x27: ffffffc071b67a38 x26: 00000000000001e8
> [   13.154180] x25: ffffff80114d3ca8 x24: ffffffc07190e400
> [   13.159492] x23: ffffffc0748bca00 x22: ffffff8010ffd000
> [   13.164803] x21: 0000000000000001 x20: ffffff80114d3c98
> [   13.170113] x19: ffffff8010414e74 x18: ffffff8010ffda48
> [   13.175423] x17: 0000000000000000 x16: 0000000000000000
> [   13.180734] x15: ffffff80914d3867 x14: 0000000000000006
> [   13.186044] x13: ffffff80114d3875 x12: ffffff801101c898
> [   13.191371] x11: ffffff801101c000 x10: 0000000005f5e0ff
> [   13.191416] type=1400 audit(16.743:79): avc: denied { read } for
> comm="drmserver" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:drmserver:s0 tcont0
> [   13.196698] x9 : ffffff80114d3800 x8 : 2966343034336663
> [   13.196707] x7 : 6530303030303030 x6 : 0000000000000424
> [   13.196713] x5 : 0000000000000000 x4 : 0000000000000000
> [   13.196718] x3 : 0000000000000001 x2 : 0000000000000001
> [   13.196724] x1 : b357113b3815f700 x0 : 0000000000000000
> [   13.214445] type=1400 audit(17.075:84): avc: denied { read } for
> comm="audioserver" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:audioserver:s0 t0
> [   13.219604] Call trace:
> [   13.219616]  __vunmap+0xe0/0xe8
> [   13.219623]  __vfree+0x24/0x70
> [   13.219635]  vfree+0x20/0x38
> [   13.270258]  ffs_epfile_io.isra.12+0x190/0x6b0
> [   13.274707]  ffs_epfile_read_iter+0xa0/0x168
> [   13.278982]  __vfs_read+0x10c/0x168
> [   13.282471]  vfs_read+0x8c/0x148
> [   13.285700]  ksys_read+0x5c/0xc8
> [   13.288930]  __arm64_sys_read+0x14/0x20
> [   13.292773]  el0_svc_common.constprop.0+0xb0/0x110
> [   13.297461] type=1400 audit(17.075:84): avc: denied { read } for
> comm="audioserver" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:audioserver:s0 t0
> [   13.297570]  el0_svc_handler+0x28/0x78
> [   13.315592] type=1400 audit(17.179:85): avc: denied { read } for
> comm="bootanimation" name="enabled" dev="sysfs" ino=5492
> scontext=u:r:bootanim:s0 tc0
> [   13.319239]  el0_svc+0x8/0xc
> [   13.319245] ---[ end trace 729ab9edfbdb0764 ]---
>
>
Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: REGRESSION: dwc2: gadget: Add scatter-gather mode

[John Stultz]

On Mon, Mar 11, 2019 at 3:15 AM Marek Szyprowski
<m.szyprowski@samsung.com> wrote:
>
> Hi John
>
>
> On 2019-03-08 21:53, John Stultz wrote:
> > Hey Andrzej,
> >   I just wanted to let you know, trying to boot linus/master on the
> > hikey board today, I've been seeing lots of errors on boot (which
> > sometimes crash the board, and sometimes doesn't). See the example
> > below.
> >
> > I've bisected the issue down to 10209abe87f5 ("usb: dwc2: gadget: Add
> > scatter-gather mode"), and if I revert that change the issue goes
> > away.
> >
> > Looking at the patch, I can't see anything obviously sticking out, but
> > let me know if you have anything suggestions you'd like me to test.
>
> I've CCed new Andrzej's email.
>
> Could you tell a bit more how to reproduce this issue? Which USB
> function are you running on the target system? I've noticed that you are
> using functionfs. Is is possible to get access to the usb function
> daemon to reproduce the problem here? Which version of HiKey board are
> you using?

So this is on a LeMaker HiKey (2gb ram 8gb storage), running AOSP (so
its adbd that is the ffs daemon)

I have a mainline tracking tree which follows mainline here to provide
MALI gpu and other android requirements:
  https://git.linaro.org/people/john.stultz/android-dev.git/log/?h=dev/hikey-mainline-WIP

I'm traveling this week, but if you really want to reproduce this
yourself you can grab the system/vendor/boot images here:
https://snapshots.linaro.org/96boards/hikey/linaro/aosp-master/latest/

Then build the kernel w/ the hikey_defconfig and replace the kernel
image in boot.img w/ the abootimg tool.

Let me know if you have any trouble and I can try to help walk you through it.

thanks
-john

Re: REGRESSION: dwc2: gadget: Add scatter-gather mode

[John Stultz]

On Mon, Mar 11, 2019 at 9:50 AM John Stultz <john.stultz@linaro.org> wrote:
> On Mon, Mar 11, 2019 at 3:15 AM Marek Szyprowski
> <m.szyprowski@samsung.com> wrote:
> > On 2019-03-08 21:53, John Stultz wrote:
> > > Hey Andrzej,
> > >   I just wanted to let you know, trying to boot linus/master on the
> > > hikey board today, I've been seeing lots of errors on boot (which
> > > sometimes crash the board, and sometimes doesn't). See the example
> > > below.
> > >
> > > I've bisected the issue down to 10209abe87f5 ("usb: dwc2: gadget: Add
> > > scatter-gather mode"), and if I revert that change the issue goes
> > > away.
> > >
> > > Looking at the patch, I can't see anything obviously sticking out, but
> > > let me know if you have anything suggestions you'd like me to test.
> >
> > I've CCed new Andrzej's email.
> >
> > Could you tell a bit more how to reproduce this issue? Which USB
> > function are you running on the target system? I've noticed that you are
> > using functionfs. Is is possible to get access to the usb function
> > daemon to reproduce the problem here? Which version of HiKey board are
> > you using?
>
> So this is on a LeMaker HiKey (2gb ram 8gb storage), running AOSP (so
> its adbd that is the ffs daemon)
>
> I have a mainline tracking tree which follows mainline here to provide
> MALI gpu and other android requirements:
>   https://git.linaro.org/people/john.stultz/android-dev.git/log/?h=dev/hikey-mainline-WIP
>
> I'm traveling this week, but if you really want to reproduce this
> yourself you can grab the system/vendor/boot images here:
> https://snapshots.linaro.org/96boards/hikey/linaro/aosp-master/latest/
>
> Then build the kernel w/ the hikey_defconfig and replace the kernel
> image in boot.img w/ the abootimg tool.
>
> Let me know if you have any trouble and I can try to help walk you through it.

Just wanted to follow up to see if there was anything further you
needed from me to help debug this?

thanks
-john

Re: REGRESSION: dwc2: gadget: Add scatter-gather mode

[Andrzej Pietrasiewicz]

Hi John,


W dniu 20.03.2019 o 21:17, John Stultz pisze:
> On Mon, Mar 11, 2019 at 9:50 AM John Stultz <john.stultz@linaro.org> wrote:
>> On Mon, Mar 11, 2019 at 3:15 AM Marek Szyprowski
>> <m.szyprowski@samsung.com> wrote:
>>> On 2019-03-08 21:53, John Stultz wrote:

<snip>

>>> I've CCed new Andrzej's email.
>>>
>>> Could you tell a bit more how to reproduce this issue? Which USB
>>> function are you running on the target system? I've noticed that you are
>>> using functionfs. Is is possible to get access to the usb function
>>> daemon to reproduce the problem here? Which version of HiKey board are
>>> you using?
>>

Can you please apply this patch:

https://www.spinics.net/lists/linux-usb/msg178195.html

and see if this helps?

Andrzej

Re: REGRESSION: dwc2: gadget: Add scatter-gather mode

[John Stultz]

On Thu, Mar 21, 2019 at 12:19 AM Andrzej Pietrasiewicz
<andrzejtp2010@gmail.com> wrote:
> Can you please apply this patch:
>
> https://www.spinics.net/lists/linux-usb/msg178195.html
>
> and see if this helps?

Thanks so much for the suggestion!

So I added that, but unfortunately I'm still seeing things crash on boot with:

[   13.655003] init: starting service 'bootanim'...
[   13.699548] functionfs read size 512 > requested size 24, splitting
request into multiple reads.
[   13.699631] ------------[ cut here ]------------
[   13.713062] kernel BUG at mm/slub.c:3950!
[   13.717096] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[   13.722600] Modules linked in:
[   13.725685] CPU: 5 PID: 2123 Comm: adbd Not tainted
5.1.0-rc1-00032-g6cf2064-dirty #1015
[   13.733795] Hardware name: HiKey Development Board (DT)
[   13.739031] pstate: 40400005 (nZcv daif +PAN -UAO)
[   13.743834] pc : kfree+0x210/0x258
[   13.747246] lr : ffs_epfile_io.isra.12+0xfc/0x6a0
[   13.751950] sp : ffffff801199bb50
[   13.755278] x29: ffffff801199bb50 x28: ffffffc074e5c600
[   13.760596] x27: ffffffc06f200e38 x26: 00000000000001e8
[   13.765912] x25: ffffff801199bc98 x24: ffffffc06f0cfa00
[   13.771230] x23: ffffffc07497f000 x22: ffffff801146d000
[   13.776547] x21: ffffff80108a6294 x20: ffffff8012063000
[   13.781865] x19: ffffffbf004818c0 x18: 0000000000000000
[   13.787202] x17: 0000000000000000 x16: ffffffc074e5c600
[   13.792540] x15: 0000000000000000 x14: 0000000000000000
[   13.797877] x13: 0000000000000000 x12: 0000000000000000
[   13.803213] x11: 0000000000000000 x10: 0000000000000000
[   13.808547] x9 : 0000000000000000 x8 : 0000000000000000
[   13.813865] x7 : 0000000000000000 x6 : 000000761e605113
[   13.819182] x5 : 000000761e605113 x4 : 0000000000000000
[   13.824516] x3 : ffffffc06f200e00 x2 : 0000000000000000
[   13.829853] x1 : ffffffbf004818c8 x0 : ffffffbf004818c8
[   13.835199] Process adbd (pid: 2123, stack limit = 0x00000000c418bad6)
[   13.841744] Call trace:
[   13.844208]  kfree+0x210/0x258
[   13.847288]  ffs_epfile_io.isra.12+0xfc/0x6a0
[   13.851673]  ffs_epfile_read_iter+0xb4/0x188
[   13.855973]  new_sync_read+0xe4/0x158
[   13.859643]  __vfs_read+0x2c/0x40
[   13.862977]  vfs_read+0x8c/0x148
[   13.866230]  ksys_read+0x5c/0xc8
[   13.869486]  __arm64_sys_read+0x14/0x20
[   13.873352]  el0_svc_common+0xac/0x108
[   13.877132]  el0_svc_handler+0x2c/0x80
[   13.880910]  el0_svc+0x8/0xc
[   13.883810] Code: f9400260 378000a0 f9400660 37000060 (d4210000)
[   13.889928] ---[ end trace 967312c15ae7c93d ]---


But I like the direction your thinking with that patch.

I realize on some other dwc3 based hardware I was seeing different
issues with larger transfers that cropped up with 772a7a724f69 ("usb:
gadget: f_fs: Allow scatter-gather buffers"), and my hack workaround
there has been to comment out the assignment of
dwc->gadget.sg_supported to true. A similar hack w/ dwc2 seems to
avoid the issue too, so maybe the dwc2 sg_support patch is tripping
over the same thing in the ffs driver?

thanks
-john