* More breakage in wireless-dev.git @ 2007-02-17 5:41 Pavel Roskin 2007-02-17 8:06 ` Pavel Roskin 0 siblings, 1 reply; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 5:41 UTC (permalink / raw) To: linux-wireless, Bcm43xx-dev Hello! There are more problems with today's wireless-dev.git even after I applied the two Johannes' patches. Even after updating DadWifi to the new API, it keeps crashing, and debugging shows that it doesn't happens around the changes code. One of the crashes happens in spin_lock_init() on a spinlock that has just been allocated by ieee80211_alloc_hw(). Maybe the size of the private area is miscalculated. I have most checks enabled, including Ingo's lockdep checker, but everything worked with the yesterday's tree. In another case, access to another field in the private are causes kernel oops. Looking at the code now, I see that both fields are close to the end on the structure used for private data. I guess something is either messing with the private data or not enough space is allocated. To exclude issues with DadWifi, I tried bcm43xx_d80211 from the kernel. It has always worked for me, but this time I got a message: FOUND UNSUPPORTED PHY (Analog 4, Type 0, Revision 7) Attempt to bring the interface down resulted in this: slab error in verify_redzone_free(): cache `size-64': double free detected Call Trace: [<ffffffff8027c091>] __slab_error+0x21/0x30 [<ffffffff8027c908>] cache_free_debugcheck+0xf8/0x220 [<ffffffff880371cf>] :bcm43xx_d80211:bcm43xx_wireless_core_exit+0x3f/0x90 [<ffffffff8027cc00>] kfree+0xb0/0x120 [<ffffffff880371cf>] :bcm43xx_d80211:bcm43xx_wireless_core_exit+0x3f/0x90 [<ffffffff8803789c>] :bcm43xx_d80211:bcm43xx_remove_interface+0xfc/0x140 [<ffffffff8800d086>] :80211:ieee80211_stop+0x106/0x130 [<ffffffff804612a2>] dev_close+0x62/0x90 [<ffffffff804606bd>] dev_change_flags+0x6d/0x150 [<ffffffff8049c97c>] devinet_ioctl+0x30c/0x730 [<ffffffff804623b4>] dev_ioctl+0x304/0x370 [<ffffffff802435b6>] up_read+0x26/0x30 [<ffffffff8049d08c>] inet_ioctl+0x4c/0x70 [<ffffffff804556c0>] sock_ioctl+0x210/0x240 [<ffffffff8028dcdb>] do_ioctl+0x1b/0x60 [<ffffffff8028df81>] vfs_ioctl+0x261/0x280 [<ffffffff8028dfea>] sys_ioctl+0x4a/0x80 [<ffffffff80209b1e>] system_call+0x7e/0x83 ffff81001d775c38: redzone 1:0x5a2cf071, redzone 2:0x5a2cf071. slab: double free detected in cache 'size-64', objp ffff81001d775c38 Again, phy is a private part of the network device, and both direct kfree() calls in bcm43xx_wireless_core_exit() are applied to pointers kept in phy. Copying to bcm43xx folks to alert them of the breakage. -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 5:41 More breakage in wireless-dev.git Pavel Roskin @ 2007-02-17 8:06 ` Pavel Roskin 2007-02-17 13:02 ` Michael Buesch 0 siblings, 1 reply; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 8:06 UTC (permalink / raw) To: linux-wireless; +Cc: Bcm43xx-dev On Sat, 2007-02-17 at 00:41 -0500, Pavel Roskin wrote: > There are more problems with today's wireless-dev.git even after I > applied the two Johannes' patches. Fixed in "d80211: fix incorrect hw.priv setting in ieee80211_alloc_hw()" posted in linux-wireless. In the meantime, just remove the second assignment for local->hw.priv in ieee80211_alloc_hw(). I'm still getting "bcm43xx_d80211: FOUND UNSUPPORTED PHY (Analog 4, Type 0, Revision 7)" from bcm43xx_d80211 for a card that used to work. That must be a separate problem. -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 8:06 ` Pavel Roskin @ 2007-02-17 13:02 ` Michael Buesch 2007-02-17 16:44 ` Pavel Roskin 0 siblings, 1 reply; 13+ messages in thread From: Michael Buesch @ 2007-02-17 13:02 UTC (permalink / raw) To: bcm43xx-dev; +Cc: Pavel Roskin, linux-wireless On Saturday 17 February 2007 09:06, Pavel Roskin wrote: > I'm still getting "bcm43xx_d80211: FOUND UNSUPPORTED PHY (Analog 4, Type > 0, Revision 7)" from bcm43xx_d80211 for a card that used to work. That > must be a separate problem. More info about your device, please. -- Greetings Michael. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 13:02 ` Michael Buesch @ 2007-02-17 16:44 ` Pavel Roskin 2007-02-17 16:55 ` Michael Buesch 0 siblings, 1 reply; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 16:44 UTC (permalink / raw) To: Michael Buesch; +Cc: bcm43xx-dev, linux-wireless Quoting Michael Buesch <mb@bu3sch.de>: > On Saturday 17 February 2007 09:06, Pavel Roskin wrote: > > I'm still getting "bcm43xx_d80211: FOUND UNSUPPORTED PHY (Analog 4, Type > > 0, Revision 7)" from bcm43xx_d80211 for a card that used to work. That > > must be a separate problem. > > More info about your device, please. It's the same PCIe module with PCI ID 14e4:4312. # lspci -v -s 0c:00.0 0c:00.0 Network controller: Broadcom Corporation BCM4310 UART (rev 01) Subsystem: Dell Unknown device 0007 Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at efdfc000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 2 Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/0 Enable- Capabilities: [d0] Express Legacy Endpoint IRQ 0 Capabilities: [100] Advanced Error Reporting Capabilities: [13c] Virtual Channel As it turns out, I'm getting the same problems with your (mb) branch, which doesn't have the latest wireless-dev.git changes yet. That's loading the device and bringing it up: ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 17 (level, low) -> IRQ 17 PCI: Setting latency timer of device 0000:0c:00.0 to 64 ssb: Sonics Silicon Backplane found on PCI device 0000:0c:00.0 ssb: Core 0 found: ChipCommon (cc 0x800, rev 0x11, vendor 0x4243) ssb: Core 1 found: IEEE 802.11 (cc 0x812, rev 0x0A, vendor 0x4243) ssb: Core 2 found: USB 1.1 Host (cc 0x817, rev 0x03, vendor 0x4243) ssb: Core 3 found: PCI-E (cc 0x820, rev 0x01, vendor 0x4243) ssb: Switching to ChipCommon core, index 0 ssb: Switching to PCI-E core, index 3 bcm43xx_d80211: Broadcom 4311 WLAN found ssb: Switching to IEEE 802.11 core, index 1 wmaster0: Selected rate control algorithm 'simple' bcm43xx_d80211: Adding Interface type 2 bcm43xx_d80211: Found PHY: Analog 4, Type 2, Revision 8 bcm43xx_d80211: Found Radio: Manuf 0x17F, Version 0x2050, Revision 2 ssb: Switching to PCI-E core, index 3 ssb: Switching to IEEE 802.11 core, index 1 bcm43xx_d80211: Loading firmware version 371.1122 (2006-11-08 22:02:13) ssb: Switching to ChipCommon core, index 0 ssb: Switching to IEEE 802.11 core, index 1 bcm43xx_d80211: Radio turned on bcm43xx_d80211: Radio enabled by hardware bcm43xx_d80211: Chip initialized bcm43xx_d80211: 32-bit DMA initialized bcm43xx_d80211: Wireless interface started wmaster0: Does not support passive scan, disabled bcm43xx_d80211: Using hardware based encryption for keyidx: 0, mac: ff:ff:ff:ff:ff:ff Then I do scanning. I didn't need to bring the interface down, but it's possible that some userspace utility tried to do it: wlan0: starting scan bcm43xx_d80211: Reconfiguring PHYmode to A-PHY bcm43xx_d80211: Wireless interface stopped bcm43xx_d80211: DMA-32 0x0200 (RX) max used slots: 2/64 bcm43xx_d80211: DMA-32 0x02A0 (TX) max used slots: 0/128 bcm43xx_d80211: DMA-32 0x0280 (TX) max used slots: 0/128 bcm43xx_d80211: DMA-32 0x0260 (TX) max used slots: 0/128 bcm43xx_d80211: DMA-32 0x0240 (TX) max used slots: 0/128 bcm43xx_d80211: DMA-32 0x0220 (TX) max used slots: 0/128 bcm43xx_d80211: DMA-32 0x0200 (TX) max used slots: 0/128 bcm43xx_d80211: Radio turned off ssb: Switching to ChipCommon core, index 0 ssb: Switching to IEEE 802.11 core, index 1 bcm43xx_d80211: FOUND UNSUPPORTED PHY (Analog 4, Type 0, Revision 7) bcm43xx_d80211: Fatal: Could not initialize device for new selected A-PHY mode wlan0: failed to set channel 36 (5180 MHz) for scan bcm43xx_d80211: Reconfiguring PHYmode to G-PHY bcm43xx_d80211: Reconfiguring PHYmode to A-PHY wlan0: scan completed bcm43xx_d80211: Removing Interface type 2 bcm43xx_d80211: Radio turned off ssb: Switching to ChipCommon core, index 0 slab error in verify_redzone_free(): cache `size-64': double free detected Call Trace: [<ffffffff8027c761>] __slab_error+0x21/0x30 [<ffffffff8027cfc8>] cache_free_debugcheck+0xf8/0x220 [<ffffffff8802f1cd>] :bcm43xx_d80211:bcm43xx_wireless_core_exit+0x3d/0x90 [<ffffffff8027d2d0>] kfree+0xb0/0x120 [<ffffffff8802f1cd>] :bcm43xx_d80211:bcm43xx_wireless_core_exit+0x3d/0x90 [<ffffffff8802f89b>] :bcm43xx_d80211:bcm43xx_remove_interface+0xfb/0x140 [<ffffffff880057fa>] :80211:ieee80211_stop+0xfa/0x120 [<ffffffff80463032>] dev_close+0x62/0x90 [<ffffffff8046246d>] dev_change_flags+0x6d/0x150 [<ffffffff8049f022>] devinet_ioctl+0x2f2/0x710 [<ffffffff80464276>] dev_ioctl+0x3f6/0x440 [<ffffffff80248cd5>] __lock_acquire+0xd35/0xdd0 [<ffffffff8049f71c>] inet_ioctl+0x4c/0x80 [<ffffffff804573a0>] sock_ioctl+0x210/0x240 [<ffffffff804bb9db>] _spin_unlock_irq+0x2b/0x40 [<ffffffff8028e68b>] do_ioctl+0x1b/0x60 [<ffffffff8028e931>] vfs_ioctl+0x261/0x280 [<ffffffff8028e99a>] sys_ioctl+0x4a/0x80 [<ffffffff80209b6e>] system_call+0x7e/0x83 ffff81000763e240: redzone 1:0x5a2cf071, redzone 2:0x5a2cf071. ssb: Switching to IEEE 802.11 core, index 1 bcm43xx_d80211: Adding Interface type 2 bcm43xx_d80211: FOUND UNSUPPORTED PHY (Analog 4, Type 0, Revision 7) -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 16:44 ` Pavel Roskin @ 2007-02-17 16:55 ` Michael Buesch 2007-02-17 17:30 ` Michael Buesch 0 siblings, 1 reply; 13+ messages in thread From: Michael Buesch @ 2007-02-17 16:55 UTC (permalink / raw) To: Pavel Roskin; +Cc: bcm43xx-dev, linux-wireless On Saturday 17 February 2007 17:44, Pavel Roskin wrote: > Quoting Michael Buesch <mb@bu3sch.de>: > > > On Saturday 17 February 2007 09:06, Pavel Roskin wrote: > > > I'm still getting "bcm43xx_d80211: FOUND UNSUPPORTED PHY (Analog 4, Type > > > 0, Revision 7)" from bcm43xx_d80211 for a card that used to work. That > > > must be a separate problem. > > > > More info about your device, please. > > It's the same PCIe module with PCI ID 14e4:4312. > > # lspci -v -s 0c:00.0 > 0c:00.0 Network controller: Broadcom Corporation BCM4310 UART (rev 01) > Subsystem: Dell Unknown device 0007 > Flags: bus master, fast devsel, latency 0, IRQ 17 > Memory at efdfc000 (32-bit, non-prefetchable) [size=16K] > Capabilities: [40] Power Management version 2 > Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/0 > Enable- > Capabilities: [d0] Express Legacy Endpoint IRQ 0 > Capabilities: [100] Advanced Error Reporting > Capabilities: [13c] Virtual Channel > > As it turns out, I'm getting the same problems with your (mb) branch, which > doesn't have the latest wireless-dev.git changes yet. > > That's loading the device and bringing it up: > > ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 17 (level, low) -> IRQ 17 > PCI: Setting latency timer of device 0000:0c:00.0 to 64 > ssb: Sonics Silicon Backplane found on PCI device 0000:0c:00.0 > ssb: Core 0 found: ChipCommon (cc 0x800, rev 0x11, vendor 0x4243) > ssb: Core 1 found: IEEE 802.11 (cc 0x812, rev 0x0A, vendor 0x4243) > ssb: Core 2 found: USB 1.1 Host (cc 0x817, rev 0x03, vendor 0x4243) > ssb: Core 3 found: PCI-E (cc 0x820, rev 0x01, vendor 0x4243) > ssb: Switching to ChipCommon core, index 0 > ssb: Switching to PCI-E core, index 3 > bcm43xx_d80211: Broadcom 4311 WLAN found > ssb: Switching to IEEE 802.11 core, index 1 > wmaster0: Selected rate control algorithm 'simple' > bcm43xx_d80211: Adding Interface type 2 > bcm43xx_d80211: Found PHY: Analog 4, Type 2, Revision 8 > bcm43xx_d80211: Found Radio: Manuf 0x17F, Version 0x2050, Revision 2 > ssb: Switching to PCI-E core, index 3 > ssb: Switching to IEEE 802.11 core, index 1 > bcm43xx_d80211: Loading firmware version 371.1122 (2006-11-08 22:02:13) > ssb: Switching to ChipCommon core, index 0 > ssb: Switching to IEEE 802.11 core, index 1 > bcm43xx_d80211: Radio turned on > bcm43xx_d80211: Radio enabled by hardware > bcm43xx_d80211: Chip initialized > bcm43xx_d80211: 32-bit DMA initialized > bcm43xx_d80211: Wireless interface started > wmaster0: Does not support passive scan, disabled > bcm43xx_d80211: Using hardware based encryption for keyidx: 0, mac: > ff:ff:ff:ff:ff:ff > > Then I do scanning. I didn't need to bring the interface down, but it's > possible that some userspace utility tried to do it: > > wlan0: starting scan > bcm43xx_d80211: Reconfiguring PHYmode to A-PHY Oh, heh. Well. Does your device have an A-PHY? If not, it's related to a bug I am fixing at the moment. ;) -- Greetings Michael. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 16:55 ` Michael Buesch @ 2007-02-17 17:30 ` Michael Buesch 2007-02-17 17:51 ` Pavel Roskin 0 siblings, 1 reply; 13+ messages in thread From: Michael Buesch @ 2007-02-17 17:30 UTC (permalink / raw) To: Pavel Roskin; +Cc: bcm43xx-dev, linux-wireless On Saturday 17 February 2007 17:55, Michael Buesch wrote: > > wlan0: starting scan > > bcm43xx_d80211: Reconfiguring PHYmode to A-PHY > > Oh, heh. Well. Does your device have an A-PHY? > If not, it's related to a bug I am fixing at the moment. ;) Well, anyway, this should be fixed in my tree now by ignoring A-PHYs, as we don't support them, yet. The slab error on the end of your dmesg is interresting, though. -- Greetings Michael. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 17:30 ` Michael Buesch @ 2007-02-17 17:51 ` Pavel Roskin 2007-02-17 17:56 ` Michael Buesch 0 siblings, 1 reply; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 17:51 UTC (permalink / raw) To: Michael Buesch; +Cc: bcm43xx-dev, linux-wireless Quoting Michael Buesch <mb@bu3sch.de>: > On Saturday 17 February 2007 17:55, Michael Buesch wrote: > > > wlan0: starting scan > > > bcm43xx_d80211: Reconfiguring PHYmode to A-PHY > > > > Oh, heh. Well. Does your device have an A-PHY? > > If not, it's related to a bug I am fixing at the moment. ;) > > Well, anyway, this should be fixed in my tree now by > ignoring A-PHYs, as we don't support them, yet. Actually, my card supports 802.11a. The card is marked as 1490, and Dell advertises such cards as 802.11a/b/g. But it would be fine if it supports 802.11b/g for now. > The slab error on the end of your dmesg is interresting, though. I'll have a closer look. -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 17:51 ` Pavel Roskin @ 2007-02-17 17:56 ` Michael Buesch 2007-02-17 18:10 ` Pavel Roskin 0 siblings, 1 reply; 13+ messages in thread From: Michael Buesch @ 2007-02-17 17:56 UTC (permalink / raw) To: Pavel Roskin; +Cc: bcm43xx-dev, linux-wireless On Saturday 17 February 2007 18:51, Pavel Roskin wrote: > > The slab error on the end of your dmesg is interresting, though. > > I'll have a closer look. I think that's fixed now, too. -- Greetings Michael. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 17:56 ` Michael Buesch @ 2007-02-17 18:10 ` Pavel Roskin 2007-02-17 18:14 ` Michael Buesch 0 siblings, 1 reply; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 18:10 UTC (permalink / raw) To: Michael Buesch; +Cc: bcm43xx-dev, linux-wireless Quoting Michael Buesch <mb@bu3sch.de>: > On Saturday 17 February 2007 18:51, Pavel Roskin wrote: > > > The slab error on the end of your dmesg is interresting, though. > > > > I'll have a closer look. > > I think that's fixed now, too. Awesome! I'm waiting for your public repository to update so I can test your fixes. -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 18:10 ` Pavel Roskin @ 2007-02-17 18:14 ` Michael Buesch 2007-02-17 18:26 ` Pavel Roskin 0 siblings, 1 reply; 13+ messages in thread From: Michael Buesch @ 2007-02-17 18:14 UTC (permalink / raw) To: Pavel Roskin; +Cc: bcm43xx-dev, linux-wireless On Saturday 17 February 2007 19:10, Pavel Roskin wrote: > Quoting Michael Buesch <mb@bu3sch.de>: > > > On Saturday 17 February 2007 18:51, Pavel Roskin wrote: > > > > The slab error on the end of your dmesg is interresting, though. > > > > > > I'll have a closer look. > > > > I think that's fixed now, too. > > Awesome! I'm waiting for your public repository to update so I can test your > fixes. I always update that when committing stuff, directly. So when you see mail from me "I committed foo", it means it's available in the public repository. -- Greetings Michael. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 18:14 ` Michael Buesch @ 2007-02-17 18:26 ` Pavel Roskin 2007-02-17 18:30 ` Michael Buesch 0 siblings, 1 reply; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 18:26 UTC (permalink / raw) To: Michael Buesch; +Cc: bcm43xx-dev, linux-wireless Quoting Michael Buesch <mb@bu3sch.de>: > On Saturday 17 February 2007 19:10, Pavel Roskin wrote: > > Awesome! I'm waiting for your public repository to update so I can test > your > > fixes. > > I always update that when committing stuff, directly. > So when you see mail from me "I committed foo", it means it's > available in the public repository. Then you are not running git-update-server-info every time from hooks/post-update. It needs to be run every time any of the branch heads changes, i.e. on every git-push. The git documentation was unclear about it until recently. -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 18:26 ` Pavel Roskin @ 2007-02-17 18:30 ` Michael Buesch 2007-02-17 18:58 ` Pavel Roskin 0 siblings, 1 reply; 13+ messages in thread From: Michael Buesch @ 2007-02-17 18:30 UTC (permalink / raw) To: Pavel Roskin; +Cc: bcm43xx-dev, linux-wireless On Saturday 17 February 2007 19:26, Pavel Roskin wrote: > Quoting Michael Buesch <mb@bu3sch.de>: > > > On Saturday 17 February 2007 19:10, Pavel Roskin wrote: > > > Awesome! I'm waiting for your public repository to update so I can test > > your > > > fixes. > > > > I always update that when committing stuff, directly. > > So when you see mail from me "I committed foo", it means it's > > available in the public repository. > > Then you are not running git-update-server-info every time from > hooks/post-update. It needs to be run every time any of the branch heads > changes, i.e. on every git-push. The git documentation was unclear about it > until recently. Uh, why? I thought it is just an optimization and git walks the tree, if the server info is not updated. But I ran it manually now. -- Greetings Michael. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: More breakage in wireless-dev.git 2007-02-17 18:30 ` Michael Buesch @ 2007-02-17 18:58 ` Pavel Roskin 0 siblings, 0 replies; 13+ messages in thread From: Pavel Roskin @ 2007-02-17 18:58 UTC (permalink / raw) To: Michael Buesch; +Cc: bcm43xx-dev, linux-wireless Quoting Michael Buesch <mb@bu3sch.de>: > Uh, why? I thought it is just an optimization and > git walks the tree, if the server info is not updated. > > But I ran it manually now. The driver is working now! As for git, I think all you need to make hooks/post-update executable, since it already has the right command in it. The problem is that git-fetch trusts refs/info instead of making individual requests to refs/branches/*. Git needs to request refs/info anyway because it never relies on the ability to request and parse http directory listings. And since refs/info already has the branch heads, why request more files? I know, this sounds weird. It must be caused by the git developers' attitude towards http as a dumb protocol. -- Regards, Pavel Roskin ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2007-02-17 18:58 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-02-17 5:41 More breakage in wireless-dev.git Pavel Roskin 2007-02-17 8:06 ` Pavel Roskin 2007-02-17 13:02 ` Michael Buesch 2007-02-17 16:44 ` Pavel Roskin 2007-02-17 16:55 ` Michael Buesch 2007-02-17 17:30 ` Michael Buesch 2007-02-17 17:51 ` Pavel Roskin 2007-02-17 17:56 ` Michael Buesch 2007-02-17 18:10 ` Pavel Roskin 2007-02-17 18:14 ` Michael Buesch 2007-02-17 18:26 ` Pavel Roskin 2007-02-17 18:30 ` Michael Buesch 2007-02-17 18:58 ` Pavel Roskin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.