lib/test_overflow.c causes WARNING and tainted kernel

lib/test_overflow.c causes WARNING and tainted kernel

[Randy Dunlap]

This is v5.0-11053-gebc551f2b8f9, MAR-12 around 4:00pm PT.

In the first test_kmalloc() in test_overflow_allocation():

[54375.073895] test_overflow: ok: (s64)(0 << 63) == 0
[54375.074228] WARNING: CPU: 2 PID: 5462 at ../mm/page_alloc.c:4584 __alloc_pages_nodemask+0x33f/0x540
[54375.074279] Modules linked in: test_overflow(+) chash cmac bluetooth drbg ansi_cprng ecdh_generic serdev of_mdio fixed_phy libphy ulpi nd_btt libnvdimm ufshcd_pltfrm tc_dwc_g210 ufshcd_dwc ufshcd_core dns_resolver fcrypt pcbc rxrpc quota_tree ocfs2_dlm ocfs2_stackglue dlm ocfs2_nodemanager ubi mtd 9pnet auth_rpcgss nls_koi8_u nls_cp932 nfsv4 nfs lockd grace sunrpc crc_itu_t reed_solomon libceph fscache fuse ctr ccm af_packet xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter coretemp hwmon xor intel_rapl zstd_compress uvcvideo x86_pkg_temp_thermal hid_generic raid6_pq intel_powerclamp kvm_intel usbmouse videobuf2_vmalloc usbkbd mei_hdcp
[54375.074409]  libcrc32c videobuf2_memops usbhid videobuf2_v4l2 kvm videobuf2_common iTCO_wdt zstd_decompress videodev hid iTCO_vendor_support arc4 iwldvm irqbypass media crct10dif_pclmul mac80211 crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic aesni_intel ledtrig_audio aes_x86_64 iwlwifi snd_hda_intel crypto_simd cryptd snd_hda_codec glue_helper snd_hda_core snd_hwdep uio_pdrv_genirq intel_cstate snd_pcm sdhci_pci intel_uncore snd_timer cfg80211 cqhci uio sdhci snd intel_rapl_perf toshiba_acpi joydev sparse_keymap sr_mod mmc_core wmi soundcore mousedev input_leds cdrom pcspkr mei_me serio_raw industrialio led_class mei e1000e rfkill toshiba_haps pcc_cpufreq lpc_ich thermal rtc_cmos evdev mac_hid battery ac sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua autofs4 [last unloaded: crc7]
[54375.075275] CPU: 2 PID: 5462 Comm: modprobe Not tainted 5.0.0mod #1
[54375.075314] Hardware name: TOSHIBA PORTEGE R835/Portable PC, BIOS Version 4.10   01/08/2013
[54375.075369] RIP: 0010:__alloc_pages_nodemask+0x33f/0x540
[54375.075406] Code: ff e8 65 e8 fe ff e9 30 fe ff ff e8 2b e8 fe ff e9 17 fe ff ff 48 89 cf e8 ce 0a 05 00 e9 9d fe ff ff 81 e7 00 20 00 00 75 b7 <0f> 0b 45 31 ff e9 0e ff ff ff 31 c0 e9 cc fd ff ff 65 4c 8b 3c 25
[54375.075501] RSP: 0018:ffff888085afefa8 EFLAGS: 00010246
[54375.075537] RAX: ffffed1010b5fdfa RBX: ffffffff97c95360 RCX: 0000000000000000
[54375.075581] RDX: 0000000000000000 RSI: 0000000000000034 RDI: 0000000000000000
[54375.075624] RBP: ffff888085aff078 R08: dffffc0000000000 R09: fffffbfff2c94ddd
[54375.075666] R10: 0000000000000001 R11: fffffbfff2c94ddc R12: 0000000000040cc0
[54375.075709] R13: 1ffff11010b5fdfa R14: 0000000000000000 R15: ffff8880a14f0040
[54375.075753] FS:  00007fe18de48b80(0000) GS:ffff88811f200000(0000) knlGS:0000000000000000
[54375.075800] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[54375.075837] CR2: 0000555ea601f034 CR3: 0000000082f02004 CR4: 00000000000606e0
[54375.075879] Call Trace:
[54375.075913]  ? __alloc_pages_slowpath+0x2180/0x2180
[54375.075968]  alloc_pages_current+0xd4/0x1b0
[54375.076008]  kmalloc_order+0x1f/0x70
[54375.076041]  __kmalloc+0x1a3/0x220
[54375.076069]  ? kfree+0xb0/0x1b0
[54375.076107]  test_kmalloc.isra.0+0x146/0x1a0 [test_overflow]
[54375.076151]  ? test_overflow_calculation+0x2f1a/0x2f1a [test_overflow]
[54375.076196]  ? sysfs_do_create_link_sd+0x8c/0x100
[54375.076243]  ? __root_device_register+0xed/0x180
[54375.076284]  test_module_init+0x27d/0x1dd2 [test_overflow]
[54375.076327]  ? test_kmalloc.isra.0+0x1a0/0x1a0 [test_overflow]
[54375.076367]  ? module_assert_mutex_or_preempt+0x40/0x70
[54375.076411]  ? sprintf+0x9b/0xd0
[54375.076439]  ? snprintf+0xc0/0xc0
[54375.076472]  ? test_kmalloc.isra.0+0x1a0/0x1a0 [test_overflow]
[54375.076516]  ? kallsyms_lookup+0x1b5/0x2a0
[54375.076556]  ? __sprint_symbol+0x10d/0x1f0
[54375.076590]  ? kallsyms_lookup+0x2a0/0x2a0
[54375.076632]  ? put_dec+0x1c/0xb0
[54375.076663]  ? number+0x7ad/0xe20
[54375.076705]  ? sched_clock+0x9/0x10
[54375.076734]  ? sched_clock+0x9/0x10
[54375.076764]  ? sched_clock_cpu+0x1b/0x1e0
[54375.076797]  ? find_held_lock+0x38/0x200
[54375.076834]  ? sched_clock+0x9/0x10
[54375.076867]  ? sched_clock+0x9/0x10
[54375.076895]  ? sched_clock+0x9/0x10
[54375.076925]  ? sched_clock_cpu+0x1b/0x1e0
[54375.076961]  ? sched_clock+0x9/0x10
[54375.076990]  ? sched_clock+0x9/0x10
[54375.077018]  ? sched_clock_cpu+0x1b/0x1e0
[54375.077051]  ? find_held_lock+0x38/0x200
[54375.077089]  ? sched_clock+0x9/0x10
[54375.077126]  ? sched_clock+0x9/0x10
[54375.077159]  ? sched_clock+0x9/0x10
[54375.077187]  ? sched_clock+0x9/0x10
[54375.077216]  ? sched_clock_cpu+0x1b/0x1e0
[54375.077249]  ? find_held_lock+0x38/0x200
[54375.077297]  ? lock_downgrade+0x7d0/0x7d0
[54375.077330]  ? lock_acquire+0xd2/0x180
[54375.077360]  ? console_unlock+0x3ce/0x810
[54375.077395]  ? kasan_check_read+0x11/0x20
[54375.077427]  ? do_raw_spin_unlock+0x54/0x220
[54375.077469]  ? sched_clock+0x9/0x10
[54375.077497]  ? sched_clock+0x9/0x10
[54375.077526]  ? sched_clock_cpu+0x1b/0x1e0
[54375.077558]  ? find_held_lock+0x38/0x200
[54375.077607]  ? lock_downgrade+0x7d0/0x7d0
[54375.077642]  ? lock_acquire+0xd2/0x180
[54375.077671]  ? do_one_initcall+0x1ed/0x2ad
[54375.077711]  ? test_kmalloc.isra.0+0x1a0/0x1a0 [test_overflow]
[54375.077753]  ? ktime_get+0xba/0x160
[54375.077788]  ? test_kmalloc.isra.0+0x1a0/0x1a0 [test_overflow]
[54375.077828]  do_one_initcall+0xab/0x2ad
[54375.077859]  ? initcall_blacklisted+0x190/0x190
[54375.077892]  ? kasan_unpoison_shadow+0x35/0x50
[54375.077930]  ? kasan_unpoison_shadow+0x35/0x50
[54375.077961]  ? kasan_unpoison_shadow+0x35/0x50
[54375.077995]  ? kasan_poison_shadow+0x2f/0x40
[54375.078028]  ? __asan_register_globals+0x5a/0x70
[54375.078072]  do_init_module+0x1c7/0x548
[54375.078111]  load_module+0x46bb/0x5da0
[54375.078188]  ? layout_and_allocate+0x2d00/0x2d00
[54375.078231]  ? kernel_read+0x90/0x130
[54375.078265]  ? kasan_check_write+0x14/0x20
[54375.078298]  ? kernel_read_file+0x247/0x630
[54375.078372]  __do_sys_finit_module+0x193/0x1b0
[54375.078406]  ? __do_sys_finit_module+0x193/0x1b0
[54375.078442]  ? __ia32_sys_init_module+0xa0/0xa0
[54375.078479]  ? vfs_statx_fd+0x45/0x80
[54375.078512]  ? kasan_check_write+0x14/0x20
[54375.078543]  ? fput_many+0x1b/0x130
[54375.078572]  ? fput+0xe/0x10
[54375.078600]  ? ksys_mmap_pgoff+0x3d9/0xb50
[54375.078658]  __x64_sys_finit_module+0x6e/0xb0
[54375.078690]  ? __x64_sys_newfstat+0x4f/0x70
[54375.078725]  do_syscall_64+0xaa/0x310
[54375.078754]  ? prepare_exit_to_usermode+0x8b/0x150
[54375.078793]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[54375.078828] RIP: 0033:0x7fe18d531129
[54375.078858] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3f 0d 2c 00 f7 d8 64 89 01 48
[54375.078954] RSP: 002b:00007ffd9e923cc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[54375.079002] RAX: ffffffffffffffda RBX: 0000557561989a50 RCX: 00007fe18d531129
[54375.079044] RDX: 0000000000000000 RSI: 0000557561777548 RDI: 0000000000000003
[54375.079087] RBP: 0000557561777548 R08: 0000000000000000 R09: 0000557561989400
[54375.079130] R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000040000
[54375.079172] R13: 0000557561989bf0 R14: 0000000000000000 R15: 0000557561989a50
[54375.079236] ---[ end trace 754acb68d8d1a1cb ]---
[54375.079313] test_overflow: kmalloc detected saturation

-- 
~Randy

Re: lib/test_overflow.c causes WARNING and tainted kernel

[Kees Cook]

Hi!

On Wed, Mar 13, 2019 at 2:29 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>
> This is v5.0-11053-gebc551f2b8f9, MAR-12 around 4:00pm PT.
>
> In the first test_kmalloc() in test_overflow_allocation():
>
> [54375.073895] test_overflow: ok: (s64)(0 << 63) == 0
> [54375.074228] WARNING: CPU: 2 PID: 5462 at ../mm/page_alloc.c:4584 __alloc_pages_nodemask+0x33f/0x540
> [...]
> [54375.079236] ---[ end trace 754acb68d8d1a1cb ]---
> [54375.079313] test_overflow: kmalloc detected saturation

Yup! This is expected and operating as intended: it is exercising the
allocator's detection of insane allocation sizes. :)

If we want to make it less noisy, perhaps we could add a global flag
the allocators could check before doing their WARNs?

-Kees

-- 
Kees Cook

Re: lib/test_overflow.c causes WARNING and tainted kernel

[Randy Dunlap]

On 3/13/19 7:53 PM, Kees Cook wrote:
> Hi!
> 
> On Wed, Mar 13, 2019 at 2:29 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>>
>> This is v5.0-11053-gebc551f2b8f9, MAR-12 around 4:00pm PT.
>>
>> In the first test_kmalloc() in test_overflow_allocation():
>>
>> [54375.073895] test_overflow: ok: (s64)(0 << 63) == 0
>> [54375.074228] WARNING: CPU: 2 PID: 5462 at ../mm/page_alloc.c:4584 __alloc_pages_nodemask+0x33f/0x540
>> [...]
>> [54375.079236] ---[ end trace 754acb68d8d1a1cb ]---
>> [54375.079313] test_overflow: kmalloc detected saturation
> 
> Yup! This is expected and operating as intended: it is exercising the
> allocator's detection of insane allocation sizes. :)
> 
> If we want to make it less noisy, perhaps we could add a global flag
> the allocators could check before doing their WARNs?
> 
> -Kees

I didn't like that global flag idea.  I also don't like the kernel becoming
tainted by this test.

Would it make sense to change the WARN_ON_ONCE() to a call to warn_alloc()
instead?  or use a plain raw printk_once()?

warn_alloc() does the _NOWARN check and does rate limiting.


--- lnx-51-rc2.orig/mm/page_alloc.c
+++ lnx-51-rc2/mm/page_alloc.c
@@ -4581,7 +4581,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, u
 	 * so bail out early if the request is out of bound.
 	 */
 	if (unlikely(order >= MAX_ORDER)) {
-		WARN_ON_ONCE(!(gfp_mask & __GFP_NOWARN));
+		warn_alloc(gfp_mask, NULL,
+				"page allocation failure: order:%u", order);
 		return NULL;
 	}
 


-- 
~Randy

Re: lib/test_overflow.c causes WARNING and tainted kernel

[Rasmus Villemoes]

On 25/05/2019 17.33, Randy Dunlap wrote:
> On 3/13/19 7:53 PM, Kees Cook wrote:
>> Hi!
>>
>> On Wed, Mar 13, 2019 at 2:29 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>>>
>>> This is v5.0-11053-gebc551f2b8f9, MAR-12 around 4:00pm PT.
>>>
>>> In the first test_kmalloc() in test_overflow_allocation():
>>>
>>> [54375.073895] test_overflow: ok: (s64)(0 << 63) == 0
>>> [54375.074228] WARNING: CPU: 2 PID: 5462 at ../mm/page_alloc.c:4584 __alloc_pages_nodemask+0x33f/0x540
>>> [...]
>>> [54375.079236] ---[ end trace 754acb68d8d1a1cb ]---
>>> [54375.079313] test_overflow: kmalloc detected saturation
>>
>> Yup! This is expected and operating as intended: it is exercising the
>> allocator's detection of insane allocation sizes. :)
>>
>> If we want to make it less noisy, perhaps we could add a global flag
>> the allocators could check before doing their WARNs?
>>
>> -Kees
> 
> I didn't like that global flag idea.  I also don't like the kernel becoming
> tainted by this test.

Me neither. Can't we pass __GFP_NOWARN from the testcases, perhaps with
a module parameter to opt-in to not pass that flag? That way one can
make the overflow module built-in (and thus run at boot) without
automatically tainting the kernel.

The vmalloc cases do not take gfp_t, would they still cause a warning?

BTW, I noticed that the 'wrap to 8K' depends on 64 bit and
pagesize==4096; for 32 bit the result is 20K, while if the pagesize is
64K one gets 128K and 512K for 32/64 bit size_t, respectively. Don't
know if that's a problem, but it's easy enough to make it independent of
pagesize (just make it 9*4096 explicitly), and if we use 5 instead of 9
it also becomes independent of sizeof(size_t) (wrapping to 16K).

Rasmus

Re: lib/test_overflow.c causes WARNING and tainted kernel

[Kees Cook]

On Mon, May 27, 2019 at 09:53:33AM +0200, Rasmus Villemoes wrote:
> On 25/05/2019 17.33, Randy Dunlap wrote:
> > On 3/13/19 7:53 PM, Kees Cook wrote:
> >> Hi!
> >>
> >> On Wed, Mar 13, 2019 at 2:29 PM Randy Dunlap <rdunlap@infradead.org> wrote:
> >>>
> >>> This is v5.0-11053-gebc551f2b8f9, MAR-12 around 4:00pm PT.
> >>>
> >>> In the first test_kmalloc() in test_overflow_allocation():
> >>>
> >>> [54375.073895] test_overflow: ok: (s64)(0 << 63) == 0
> >>> [54375.074228] WARNING: CPU: 2 PID: 5462 at ../mm/page_alloc.c:4584 __alloc_pages_nodemask+0x33f/0x540
> >>> [...]
> >>> [54375.079236] ---[ end trace 754acb68d8d1a1cb ]---
> >>> [54375.079313] test_overflow: kmalloc detected saturation
> >>
> >> Yup! This is expected and operating as intended: it is exercising the
> >> allocator's detection of insane allocation sizes. :)
> >>
> >> If we want to make it less noisy, perhaps we could add a global flag
> >> the allocators could check before doing their WARNs?
> >>
> >> -Kees
> > 
> > I didn't like that global flag idea.  I also don't like the kernel becoming
> > tainted by this test.
> 
> Me neither. Can't we pass __GFP_NOWARN from the testcases, perhaps with
> a module parameter to opt-in to not pass that flag? That way one can
> make the overflow module built-in (and thus run at boot) without
> automatically tainting the kernel.
> 
> The vmalloc cases do not take gfp_t, would they still cause a warning?

They still warn, but they don't seem to taint. I.e. this patch:

diff --git a/lib/test_overflow.c b/lib/test_overflow.c
index fc680562d8b6..c922f0d86181 100644
--- a/lib/test_overflow.c
+++ b/lib/test_overflow.c
@@ -486,11 +486,12 @@ static int __init test_overflow_shift(void)
  * Deal with the various forms of allocator arguments. See comments above
  * the DEFINE_TEST_ALLOC() instances for mapping of the "bits".
  */
-#define alloc010(alloc, arg, sz) alloc(sz, GFP_KERNEL)
-#define alloc011(alloc, arg, sz) alloc(sz, GFP_KERNEL, NUMA_NO_NODE)
+#define alloc_GFP	(GFP_KERNEL | __GFP_NOWARN)
+#define alloc010(alloc, arg, sz) alloc(sz, alloc_GFP)
+#define alloc011(alloc, arg, sz) alloc(sz, alloc_GFP, NUMA_NO_NODE)
 #define alloc000(alloc, arg, sz) alloc(sz)
 #define alloc001(alloc, arg, sz) alloc(sz, NUMA_NO_NODE)
-#define alloc110(alloc, arg, sz) alloc(arg, sz, GFP_KERNEL)
+#define alloc110(alloc, arg, sz) alloc(arg, sz, alloc_GFP | __GFP_NOWARN)
 #define free0(free, arg, ptr)	 free(ptr)
 #define free1(free, arg, ptr)	 free(arg, ptr)
 
will remove the tainting behavior but is still a bit "noisy". I can't
find a way to pass __GFP_NOWARN to a vmalloc-based allocation, though.

Randy, is removing taint sufficient for you?

> BTW, I noticed that the 'wrap to 8K' depends on 64 bit and
> pagesize==4096; for 32 bit the result is 20K, while if the pagesize is
> 64K one gets 128K and 512K for 32/64 bit size_t, respectively. Don't
> know if that's a problem, but it's easy enough to make it independent of
> pagesize (just make it 9*4096 explicitly), and if we use 5 instead of 9
> it also becomes independent of sizeof(size_t) (wrapping to 16K).

Ah! Yes, all excellent points. I've adjusted that too now. I'll send
the result to Andrew.

Thanks!

-- 
Kees Cook

Re: lib/test_overflow.c causes WARNING and tainted kernel

[Randy Dunlap]

On 5/28/19 3:47 PM, Kees Cook wrote:
> On Mon, May 27, 2019 at 09:53:33AM +0200, Rasmus Villemoes wrote:
>> On 25/05/2019 17.33, Randy Dunlap wrote:
>>> On 3/13/19 7:53 PM, Kees Cook wrote:
>>>> Hi!
>>>>
>>>> On Wed, Mar 13, 2019 at 2:29 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>>>>>
>>>>> This is v5.0-11053-gebc551f2b8f9, MAR-12 around 4:00pm PT.
>>>>>
>>>>> In the first test_kmalloc() in test_overflow_allocation():
>>>>>
>>>>> [54375.073895] test_overflow: ok: (s64)(0 << 63) == 0
>>>>> [54375.074228] WARNING: CPU: 2 PID: 5462 at ../mm/page_alloc.c:4584 __alloc_pages_nodemask+0x33f/0x540
>>>>> [...]
>>>>> [54375.079236] ---[ end trace 754acb68d8d1a1cb ]---
>>>>> [54375.079313] test_overflow: kmalloc detected saturation
>>>>
>>>> Yup! This is expected and operating as intended: it is exercising the
>>>> allocator's detection of insane allocation sizes. :)
>>>>
>>>> If we want to make it less noisy, perhaps we could add a global flag
>>>> the allocators could check before doing their WARNs?
>>>>
>>>> -Kees
>>>
>>> I didn't like that global flag idea.  I also don't like the kernel becoming
>>> tainted by this test.
>>
>> Me neither. Can't we pass __GFP_NOWARN from the testcases, perhaps with
>> a module parameter to opt-in to not pass that flag? That way one can
>> make the overflow module built-in (and thus run at boot) without
>> automatically tainting the kernel.
>>
>> The vmalloc cases do not take gfp_t, would they still cause a warning?
> 
> They still warn, but they don't seem to taint. I.e. this patch:
> 
> diff --git a/lib/test_overflow.c b/lib/test_overflow.c
> index fc680562d8b6..c922f0d86181 100644
> --- a/lib/test_overflow.c
> +++ b/lib/test_overflow.c
> @@ -486,11 +486,12 @@ static int __init test_overflow_shift(void)
>   * Deal with the various forms of allocator arguments. See comments above
>   * the DEFINE_TEST_ALLOC() instances for mapping of the "bits".
>   */
> -#define alloc010(alloc, arg, sz) alloc(sz, GFP_KERNEL)
> -#define alloc011(alloc, arg, sz) alloc(sz, GFP_KERNEL, NUMA_NO_NODE)
> +#define alloc_GFP	(GFP_KERNEL | __GFP_NOWARN)
> +#define alloc010(alloc, arg, sz) alloc(sz, alloc_GFP)
> +#define alloc011(alloc, arg, sz) alloc(sz, alloc_GFP, NUMA_NO_NODE)
>  #define alloc000(alloc, arg, sz) alloc(sz)
>  #define alloc001(alloc, arg, sz) alloc(sz, NUMA_NO_NODE)
> -#define alloc110(alloc, arg, sz) alloc(arg, sz, GFP_KERNEL)
> +#define alloc110(alloc, arg, sz) alloc(arg, sz, alloc_GFP | __GFP_NOWARN)
>  #define free0(free, arg, ptr)	 free(ptr)
>  #define free1(free, arg, ptr)	 free(arg, ptr)
>  
> will remove the tainting behavior but is still a bit "noisy". I can't
> find a way to pass __GFP_NOWARN to a vmalloc-based allocation, though.
> 
> Randy, is removing taint sufficient for you?

Yes it is.  Thanks.

>> BTW, I noticed that the 'wrap to 8K' depends on 64 bit and
>> pagesize==4096; for 32 bit the result is 20K, while if the pagesize is
>> 64K one gets 128K and 512K for 32/64 bit size_t, respectively. Don't
>> know if that's a problem, but it's easy enough to make it independent of
>> pagesize (just make it 9*4096 explicitly), and if we use 5 instead of 9
>> it also becomes independent of sizeof(size_t) (wrapping to 16K).
> 
> Ah! Yes, all excellent points. I've adjusted that too now. I'll send
> the result to Andrew.
> 
> Thanks!
> 


-- 
~Randy