linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* RE: Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble
@ 2005-07-12 23:16 Protasevich, Natalie
  2005-07-13  6:46 ` Michel Bouissou
  0 siblings, 1 reply; 16+ messages in thread
From: Protasevich, Natalie @ 2005-07-12 23:16 UTC (permalink / raw)
  To: Michel Bouissou, Alan Stern; +Cc: linux-kernel

> > Now that we know which is the offending device, it should 
> be easy to 
> > find out why the IRQ assignments go wrong.  That certainly 
> needs to be 
> > fixed, even though Michel's problem appears to be solved.
> 
> Well, it's solved by currently giving me the choice between 
> no USB 2.0 and IO-APIC, or USB 2.0 and no IO-APIC. That makes 
> a temporary solution, but I'd love to have both (and recall 
> the happy times of 2.4x that was happy with both ;-)
> 
> Natalie, could you please tell me which kernel (I have 
> started to have a number of them ;-) you'd like me <<  to 
> boot with "apic=debug" and also perhaps with "pci=routeirq" 
> and collect the trace >>, and with which BIOS setup ? (i.e. 
> USB 2.0 enabled or disabled ?).

At this point, you'll need to set the system back to its original state
that you started with, and have both "apic=debug" and "pci=routeirq" as
boot options. I'd say use the last kernel that you prepared with USB
support there (and all usual devices enabled in BIOS).
 
> When you speak about "collecting the trace", I assume you 
> mean the dmesg or /var/log/messages I'll get after booting this way ?

Yes, the dmesg will have all early setup information, including
interrupts setup. With pci=routeirq, all the PCI IRQ mappings should be
there, too, and you might need to include part of /var/log/messages
where USB devices get configured.

Thanks,
--Natalie 

^ permalink raw reply	[flat|nested] 16+ messages in thread
* RE: Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble
@ 2005-07-12 21:02 Protasevich, Natalie
  0 siblings, 0 replies; 16+ messages in thread
From: Protasevich, Natalie @ 2005-07-12 21:02 UTC (permalink / raw)
  To: Michel Bouissou, Alan Stern; +Cc: linux-kernel

> > On the other hand, _something_ was generating an interrupt request 
> > that got mapped to IRQ 21 by the hardware.  And these 
> requests do seem 
> > to be associated with USB activity.  Maybe the EHCI 
> controller is responsible?
> > One of your postings showed both uhci_hcd:usb2 and ehci_hcd:usb4 
> > mapped to IRQ 11.  That could indicate a shared signal 
> line, which is 
> > currently being mapped incorrectly.

I suspect that some device is actually on the IRQ 21, and that's how its IO-APIC line is set up. Later on, its driver tries to assign different IRQ, due to some discrepancy, and the handler gets registered on say IRQ 11, and to a wrong pin, so the actual interrupts go unattended. If this what's happening, the trace will hopefully tell the story. I suggest to boot with "apic=debug" and also perhaps with "pci=routeirq" and collect the trace. You can attach the part up to the point when it reports usb devices set up. 

--Natalie

 

^ permalink raw reply	[flat|nested] 16+ messages in thread
* Re: [SOLVED ??] Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble
@ 2005-07-12  1:54 Alan Stern
  2005-07-12  7:54 ` Michel Bouissou
  0 siblings, 1 reply; 16+ messages in thread
From: Alan Stern @ 2005-07-12  1:54 UTC (permalink / raw)
  To: Michel Bouissou; +Cc: Protasevich, Natalie, linux-kernel

On Mon, 11 Jul 2005, Michel Bouissou wrote:

> Le Lundi 11 Juillet 2005 23:21, Alan Stern a écrit :
> > Don't jump to hasty conclusions.  Problems like this are often caused by
> > unrelated things that you wouldn't suspect at first.
> 
> I know... Been working with computers for... Uh... 25 years ?

35 years for me...  My, how the time flies!

> I don't believe the hardware is broken, as it's been running for more than 2 
> years 100% stable with a 24/7/365 uptime. And with me plugging and unplugging 
> USB devices...
> 
> Never had a single problem of the kind with any 2.4.x kernel. Get those 
> problems only with 2.6.x kernels, and, in 2.6.12, only when UP IO-APIC is 
> enabled (which ran perfectly good in 2.4).
> 
> So the problem is circled to 2.6 kernel, uhci_hcd and UP IO-APIC.

Don't rule out the hardware too quickly.  2.4 and 2.6 manage the UHCI
controllers in different ways.  In particular, the usb-uhci driver in 2.4
does not suspend the controller when no devices are attached, whereas
uhci-hcd in 2.6 does.  (So does the "alternate" uhci driver in 2.4.)  I
recently handled a bug report -- from another Frenchman -- where it turned
out that his UHCI hardware instantly crashed the entire system whenever
it resumed from the suspended state!

To try and help pin things down, tomorrow (i.e., Tuesday) I'll send you a
test patch to completely disable the UHCI controllers, leaving them awake
and idle, not generating interrupts.  If you still get those spurious
IRQs, they will have to come from somewhere else.  (Assuming you can
devote server time to this sort of testing...)

And of course, if the spurious IRQs stop then we'll know where to probe 
further.

Alan Stern


^ permalink raw reply	[flat|nested] 16+ messages in thread
* RE: Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble
@ 2005-07-10 18:50 Protasevich, Natalie
  2005-07-11  9:06 ` Michel Bouissou
  0 siblings, 1 reply; 16+ messages in thread
From: Protasevich, Natalie @ 2005-07-10 18:50 UTC (permalink / raw)
  To: Michel Bouissou, linux-kernel; +Cc: stern, mingo

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of 
> Michel Bouissou
> Sent: Sunday, July 10, 2005 2:11 AM
> To: linux-kernel@vger.kernel.org
> Cc: stern@rowland.harvard.edu; mingo@redhat.com
> Subject: Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble
> 
> Hi there,
> 
> I use a Gigabyte GA7-VAXP motherboard that has been fine for 
> more than 2 years with 2.4 series kernels, and that gives 
> trouble with 2.6.12 (and previous) kernels.
> 
> The problem seems to sit between the UP IO-APIC and USB 
> uhci_hcd driver.
> 
> The GA7-VAXP MB is equipped with an Athlon 2000+ XP and a VIA 
> KT400 chipset.
> 
> Using 2.4 series kernel the IO-APIC is used just fine.  Was 
> never source of any kind of problem.
> 
> Using 2.6.12 kernel, if the IO-APIC is enabled in the kernel 
> configuration, then the interrupt controlling the uhci_hcd 
> USB controller gets disabled as soon as uhci_hcd is initialized.

Michel,
Symptoms that you describe resemble several IRQ problems with VIA
chipset reported by others (but not quite...) Could you check on
bugzilla #4843 please
http://bugzilla.kernel.org/show_bug.cgi?id=4843 and see if the patch
fixes your problem.
Thanks,
--Natalie
 
 

^ permalink raw reply	[flat|nested] 16+ messages in thread
* Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble
@ 2005-07-10  8:11 Michel Bouissou
  0 siblings, 0 replies; 16+ messages in thread
From: Michel Bouissou @ 2005-07-10  8:11 UTC (permalink / raw)
  To: linux-kernel; +Cc: stern, mingo

Hi there,

I use a Gigabyte GA7-VAXP motherboard that has been fine for more than 2 years 
with 2.4 series kernels, and that gives trouble with 2.6.12 (and previous) 
kernels.

The problem seems to sit between the UP IO-APIC and USB uhci_hcd driver.

The GA7-VAXP MB is equipped with an Athlon 2000+ XP and a VIA KT400 chipset.

Using 2.4 series kernel the IO-APIC is used just fine.  Was never source of 
any kind of problem.

Using 2.6.12 kernel, if the IO-APIC is enabled in the kernel configuration, 
then the interrupt controlling the uhci_hcd USB controller gets disabled as 
soon as uhci_hcd is initialized.

This however doesn't happen all the times. I would say that it happens at 
bootup about 50% of the times. However, if the uhci_hcd happened to 
initialize OK, then it will work just good until the machine is rebooted.

When it fails, I get the following messages in syslog:

Jul  9 12:28:06 totor kernel: usbcore: registered new driver usbfs
Jul  9 12:28:06 totor kernel: usbcore: registered new driver hub
Jul  9 12:28:06 totor kernel: USB Universal Host Controller Interface driver 
v2.2
Jul  9 12:28:06 totor kernel: PCI: Via IRQ fixup for 0000:00:10.0, from 10 to 
5
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.0: VIA Technologies, Inc. 
VT82xxxxx UHCI USB 1.1 Controller
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.0: new USB bus registered, 
assigned bus number 1
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.0: irq 21, io base 
0x0000ac00
Jul  9 12:28:06 totor kernel: hub 1-0:1.0: USB hub found
Jul  9 12:28:06 totor kernel: hub 1-0:1.0: 2 ports detected
Jul  9 12:28:06 totor kernel: PCI: Via IRQ fixup for 0000:00:10.1, from 10 to 
5
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.1: VIA Technologies, Inc. 
VT82xxxxx UHCI USB 1.1 Controller (#2)
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.1: new USB bus registered, 
assigned bus number 2
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.1: irq 21, io base 
0x0000b000
Jul  9 12:28:06 totor kernel: hub 2-0:1.0: USB hub found
Jul  9 12:28:06 totor kernel: hub 2-0:1.0: 2 ports detected
Jul  9 12:28:06 totor kernel: PCI: Via IRQ fixup for 0000:00:10.2, from 10 to 
5
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.2: VIA Technologies, Inc. 
VT82xxxxx UHCI USB 1.1 Controller (#3)
Jul  9 12:28:06 totor kernel: usb 1-1: new low speed USB device using uhci_hcd 
and address 2
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.2: new USB bus registered, 
assigned bus number 3
Jul  9 12:28:06 totor kernel: uhci_hcd 0000:00:10.2: irq 21, io base 
0x0000b400
Jul  9 12:28:06 totor kernel: hub 3-0:1.0: USB hub found
Jul  9 12:28:06 totor kernel: hub 3-0:1.0: 2 ports detected
Jul  9 12:28:06 totor kernel: PCI: Via IRQ fixup for 0000:00:10.3, from 11 to 
3
Jul  9 12:28:06 totor kernel: ehci_hcd 0000:00:10.3: VIA Technologies, Inc. 
USB 2.0
Jul  9 12:28:06 totor kernel: ehci_hcd 0000:00:10.3: new USB bus registered, 
assigned bus number 4
Jul  9 12:28:06 totor kernel: ehci_hcd 0000:00:10.3: irq 19, io mem 0xe3006000
Jul  9 12:28:06 totor kernel: ehci_hcd 0000:00:10.3: USB 2.0 initialized, EHCI 
1.00, driver 10 Dec 2004
Jul  9 12:28:06 totor kernel: usbcore: registered new driver hiddev
Jul  9 12:28:06 totor kernel: hub 4-0:1.0: USB hub found
Jul  9 12:28:06 totor kernel: hub 4-0:1.0: 6 ports detected
Jul  9 12:28:06 totor kernel: usbhid: probe of 1-1:1.0 failed with error -5
Jul  9 12:28:06 totor kernel: usbcore: registered new driver usbhid
Jul  9 12:28:06 totor kernel: drivers/usb/input/hid-core.c: v2.01:USB HID core 
driver
Jul  9 12:28:06 totor kernel: usb 1-1: USB disconnect, address 2
Jul  9 12:28:06 totor kernel: irq 21: nobody cared!
Jul  9 12:28:06 totor kernel:  [__report_bad_irq+42/144] 
__report_bad_irq+0x2a/0x90
Jul  9 12:28:06 totor kernel:  [<c0136e5a>] __report_bad_irq+0x2a/0x90
Jul  9 12:28:06 totor kernel:  [handle_IRQ_event+57/112] 
handle_IRQ_event+0x39/0x70
Jul  9 12:28:06 totor kernel:  [<c01368f9>] handle_IRQ_event+0x39/0x70
Jul  9 12:28:06 totor kernel:  [note_interrupt+163/208] 
note_interrupt+0xa3/0xd0
Jul  9 12:28:06 totor kernel:  [<c0136f83>] note_interrupt+0xa3/0xd0
Jul  9 12:28:06 totor kernel:  [__do_IRQ+240/256] __do_IRQ+0xf0/0x100
Jul  9 12:28:06 totor kernel:  [<c0136a20>] __do_IRQ+0xf0/0x100
Jul  9 12:28:06 totor kernel:  [do_IRQ+54/112] do_IRQ+0x36/0x70
Jul  9 12:28:06 totor kernel:  [<c01050e6>] do_IRQ+0x36/0x70
Jul  9 12:28:06 totor kernel:  [common_interrupt+26/32] 
common_interrupt+0x1a/0x20
Jul  9 12:28:06 totor kernel:  [<c0103526>] common_interrupt+0x1a/0x20
Jul  9 12:28:06 totor kernel: handlers:
Jul  9 12:28:06 totor kernel: [pg0+947137696/1069564928] (usb_hcd_irq+0x0/0x90 
[usbcore])
Jul  9 12:28:06 totor kernel: [<f8b3cca0>] (usb_hcd_irq+0x0/0x90 [usbcore])
Jul  9 12:28:06 totor kernel: [pg0+947137696/1069564928] (usb_hcd_irq+0x0/0x90 
[usbcore])
Jul  9 12:28:06 totor kernel: [<f8b3cca0>] (usb_hcd_irq+0x0/0x90 [usbcore])
Jul  9 12:28:06 totor kernel: [pg0+947137696/1069564928] (usb_hcd_irq+0x0/0x90 
[usbcore])
Jul  9 12:28:06 totor kernel: [<f8b3cca0>] (usb_hcd_irq+0x0/0x90 [usbcore])
Jul  9 12:28:06 totor kernel: Disabling IRQ #21
Jul  9 12:28:06 totor kernel: usb 1-1: new low speed USB device using uhci_hcd 
and address 3
Jul  9 12:28:06 totor kernel: input: USB HID v1.10 Mouse [Logitech USB 
Receiver] on usb-0000:00:10.0-1
Jul  9 12:28:06 totor kernel: usb 3-2: new full speed USB device using 
uhci_hcd and address 2


Starting there, USB devices cause trouble --  including my mouse that responds 
very slowly and erratically, probably because its interrupt got disabled ?


If the kernel is compiled without support for UP IO-APIC 
(CONFIG_X86_UP_IOAPIC), or booted with the "noapic" option, then this trouble 
doesn't happen -- but the system doesn't use its IO-APIC, which is annoying.

Here follows other syslog boot messages that may be relevant to this IRQ 
problem.  Any clue ?

N.B.: I'm not subscribed to the Linux-kernel ML. Please copy me on answers.

TIA, cheers.


Jul  9 12:28:06 totor kernel: Linux version 2.6.12-3mib4v 
(root@totor.bouissou.net) (gcc version 3.4.3 (Mandrakelinux 10.2 3.4.3-7mdk)) 
#6 Sat Jul 9 12:16:43 CEST 2005
Jul  9 12:28:06 totor kernel: BIOS-provided physical RAM map:
Jul  9 12:28:06 totor kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 
(usable)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 000000000009fc00 - 00000000000a0000 
(reserved)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 00000000000f0000 - 0000000000100000 
(reserved)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 0000000000100000 - 000000003fff0000 
(usable)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 000000003fff0000 - 000000003fff3000 
(ACPI NVS)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 000000003fff3000 - 0000000040000000 
(ACPI data)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 00000000fec00000 - 00000000fec01000 
(reserved)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 00000000fee00000 - 00000000fee01000 
(reserved)
Jul  9 12:28:06 totor kernel:  BIOS-e820: 00000000ffff0000 - 0000000100000000 
(reserved)
Jul  9 12:28:06 totor kernel: 127MB HIGHMEM available.
Jul  9 12:28:06 totor kernel: 896MB LOWMEM available.
Jul  9 12:28:06 totor kernel: found SMP MP-table at 000f4c50
Jul  9 12:28:06 totor kernel: DMI 2.3 present.
Jul  9 12:28:06 totor kernel: Intel MultiProcessor Specification v1.4
Jul  9 12:28:06 totor kernel:     Virtual Wire compatibility mode.
Jul  9 12:28:06 totor kernel: OEM ID: OEM00000 Product ID: PROD00000000 APIC 
at: 0xFEE00000
Jul  9 12:28:06 totor kernel: Processor #0 6:6 APIC version 17
Jul  9 12:28:06 totor kernel: I/O APIC #2 Version 17 at 0xFEC00000.
Jul  9 12:28:06 totor kernel: Enabling APIC mode:  Flat.  Using 1 I/O APICs
Jul  9 12:28:06 totor kernel: Processors: 1
Jul  9 12:28:06 totor kernel: Allocating PCI resources starting at 40000000 
(gap: 40000000:bec00000)
Jul  9 12:28:06 totor kernel: Built 1 zonelists
Jul  9 12:28:06 totor kernel: Initializing CPU#0
Jul  9 12:28:06 totor kernel: Kernel command line: root=/dev/evms/lv_racine ro 
devfs=nomount noquiet vga=791
Jul  9 12:28:06 totor kernel: PID hash table entries: 4096 (order: 12, 65536 
bytes)
Jul  9 12:28:06 totor kernel: Detected 1674.892 MHz processor.
Jul  9 12:28:06 totor kernel: Using tsc for high-res timesource
Jul  9 12:28:06 totor kernel: Console: colour dummy device 80x25
Jul  9 12:28:06 totor kernel: Dentry cache hash table entries: 131072 (order: 
7, 524288 bytes)
Jul  9 12:28:06 totor kernel: Inode-cache hash table entries: 65536 (order: 6, 
262144 bytes)
Jul  9 12:28:06 totor kernel: Memory: 1032408k/1048512k available (1820k 
kernel code, 15220k reserved, 740k data, 196k init, 131008k highmem,
 0k BadRAM)
Jul  9 12:28:06 totor kernel: Checking if this processor honours the WP bit 
even in supervisor mode... Ok.
Jul  9 12:28:06 totor kernel: Security Framework v1.0.0 initialized
Jul  9 12:28:06 totor kernel: Capability LSM initialized
Jul  9 12:28:06 totor kernel: Mount-cache hash table entries: 512
Jul  9 12:28:06 totor kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 
64K (64 bytes/line)
Jul  9 12:28:06 totor kernel: CPU: L2 Cache: 256K (64 bytes/line)
Jul  9 12:28:06 totor kernel: Intel machine check architecture supported.
Jul  9 12:28:06 totor kernel: Intel machine check reporting enabled on CPU#0.
Jul  9 12:28:06 totor kernel: CPU: AMD Athlon(tm) XP 2000+ stepping 02
Jul  9 12:28:06 totor kernel: Enabling fast FPU save and restore... done.
Jul  9 12:28:06 totor kernel: Enabling unmasked SIMD FPU exception support... 
done.
Jul  9 12:28:06 totor kernel: Checking 'hlt' instruction... OK.
Jul  9 12:28:06 totor kernel: ENABLING IO-APIC IRQs
Jul  9 12:28:06 totor kernel: ..TIMER: vector=0x31 pin1=2 pin2=0
Jul  9 12:28:06 totor kernel: checking if image is initramfs...it isn't (no 
cpio magic); looks like an initrd
Jul  9 12:28:06 totor kernel: Freeing initrd memory: 3097k freed
Jul  9 12:28:06 totor kernel: NET: Registered protocol family 16
Jul  9 12:28:06 totor kernel: PCI: PCI BIOS revision 2.10 entry at 0xf9ea0, 
last bus=1
Jul  9 12:28:06 totor kernel: PCI: Using configuration type 1
Jul  9 12:28:06 totor kernel: mtrr: v2.0 (20020519)
Jul  9 12:28:06 totor kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Jul  9 12:28:06 totor kernel: PnPBIOS: Disabled
Jul  9 12:28:06 totor kernel: PCI: Probing PCI hardware
Jul  9 12:28:06 totor kernel: PCI: Probing PCI hardware (bus 00)
Jul  9 12:28:06 totor kernel: PCI: Using IRQ router default [1106/3189] at 
0000:00:00.0
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:0a.0[A] -> IRQ 
18
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:0b.0[A] -> IRQ 
19
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:0f.0[A] -> IRQ 
19
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:10.0[A] -> IRQ 
21
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:10.1[B] -> IRQ 
21
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:10.2[C] -> IRQ 
21
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:10.3[D] -> IRQ 
19
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:11.1[A] -> IRQ 
22
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:11.5[C] -> IRQ 
22
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:00:13.0[A] -> IRQ 
18
Jul  9 12:28:06 totor kernel: PCI->APIC IRQ transform: 0000:01:00.0[A] -> IRQ 
16
Jul  9 12:28:06 totor kernel: Machine check exception polling timer started.
Jul  9 12:28:06 totor kernel: apm: BIOS version 1.2 Flags 0x07 (Driver version 
1.16ac)
Jul  9 12:28:06 totor kernel: audit: initializing netlink socket (disabled)
Jul  9 12:28:06 totor kernel: audit(1120904798.619:0): initialized
Jul  9 12:28:06 totor kernel: highmem bounce pool size: 64 pages
Jul  9 12:28:06 totor kernel: inotify device minor=63
[...]


-- 
Michel Bouissou <michel@bouissou.net> OpenPGP ID 0xDDE8AC6E

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2005-07-13  6:46 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-07-12 23:16 Kernel 2.6.12 + IO-APIC + uhci_hcd = Trouble Protasevich, Natalie
2005-07-13  6:46 ` Michel Bouissou
  -- strict thread matches above, loose matches on Subject: below --
2005-07-12 21:02 Protasevich, Natalie
2005-07-12  1:54 [SOLVED ??] " Alan Stern
2005-07-12  7:54 ` Michel Bouissou
2005-07-12 14:12   ` Alan Stern
2005-07-12 17:03     ` Michel Bouissou
2005-07-12 18:15       ` Alan Stern
2005-07-12 18:16       ` Michel Bouissou
2005-07-12 18:57         ` Alan Stern
2005-07-12 20:40           ` Michel Bouissou
2005-07-12 21:37             ` Alan Stern
2005-07-12 22:01               ` Michel Bouissou
2005-07-10 18:50 Protasevich, Natalie
2005-07-11  9:06 ` Michel Bouissou
2005-07-11 18:36   ` Alan Stern
2005-07-10  8:11 Michel Bouissou

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).