SKY2 driver - version 0.13 - buggy but working

SKY2 driver - version 0.13 - buggy but working

[David Luyer]

Your new SKY2 driver in the latest 2.6.16-rc1 snapshots does millions
of printk()s (approximately 230,000 per second) ... but works!

Motherboard: A7V-E SE (onboard Marvel GE)
OS: Linux current snapshot (2.6.16-rc1-g0f36b018), 32-bit on AMD64
PCI options: ACPI, PCI, PCI Express, MSI enabled

dmesg|egrep 'sky2|messages suppressed':

sky2 v0.13 addr 0xdc000000 irq 66 Yukon-EC (0xb6) rev 2
sky2 eth0: addr 00:13:d4:f6:be:52
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
sky2 0000:05:00.0: pci express error (0x0)
printk: 1144326 messages suppressed.
sky2 0000:05:00.0: pci express error (0x0)
printk: 1141162 messages suppressed.
sky2 0000:05:00.0: pci express error (0x0)
printk: 1122566 messages suppressed.
sky2 0000:05:00.0: pci express error (0x0)
printk: 1125246 messages suppressed.
sky2 0000:05:00.0: pci express error (0x0)
printk: 1124271 messages suppressed.
sky2 0000:05:00.0: pci express error (0x0)
printk: 1130645 messages suppressed.

David.
-- 
Pacific Internet (Australia) Pty Ltd
Business card: http://www.luyer.net/bc.html
Important notice: http://www.pacific.net.au/disclaimer/

Re: SKY2 driver - version 0.13 - buggy but working

[Daniel Drake]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

David Luyer wrote:
> Your new SKY2 driver in the latest 2.6.16-rc1 snapshots does millions
> of printk()s (approximately 230,000 per second) ... but works!
> 
> Motherboard: A7V-E SE (onboard Marvel GE)
> OS: Linux current snapshot (2.6.16-rc1-g0f36b018), 32-bit on AMD64
> PCI options: ACPI, PCI, PCI Express, MSI enabled
> 
> dmesg|egrep 'sky2|messages suppressed':
> 
> sky2 v0.13 addr 0xdc000000 irq 66 Yukon-EC (0xb6) rev 2
> sky2 eth0: addr 00:13:d4:f6:be:52
> sky2 0000:05:00.0: pci express error (0x0)

Can you see if this patch makes any difference?

Thanks.


[-- Attachment #2: sky2-error-reporting.patch --]
[-- Type: text/x-patch, Size: 6658 bytes --]

Date: Wed, 11 Jan 2006 15:37:01 -0800
From: Stephen Hemminger <shemminger@osdl.org>
To: "Dmitrij D. Czarkoff" <czarkoff@yandex.ru>, Daniel Drake
 <dsd@gentoo.org>
Cc: netdev@vger.kernel.org
Subject: sky2: error reporting
Message-ID: <20060111153701.26d66526@dxpl.pdx.osdl.net>
In-Reply-To: <200601112116.46099.czarkoff@yandex.ru>
References: <200601112116.46099.czarkoff@yandex.ru>
X-Mailer: Sylpheed-Claws 1.9.100 (GTK+ 2.6.10; x86_64-redhat-linux-gnu)
X-Face: &@E+xe?c%:&e4D{>f1O<&U>2qwRREG5!}7R4;D<"NO^UI2mJ[eEOA2*3>(`Th.yP,VDPo9$
 /`~cw![cmj~~jWe?AHY7D1S+\}5brN0k*NE?pPh_'_d>6;XGG[\KDRViCfumZT3@[
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-MIMEDefang-Filter: osdl$Revision: 1.129 $
X-Scanned-By: MIMEDefang 2.36
X-Virus-Scan: scanned

Try this, it limits the console output and attempts to handle more
errors. Of course, I see no errors on my systems (that would make
this problem too easy)...

Index: sky2-work/drivers/net/sky2.c
===================================================================
--- sky2-work.orig/drivers/net/sky2.c	2006-01-11 13:54:23.000000000 -0800
+++ sky2-work/drivers/net/sky2.c	2006-01-11 15:19:56.000000000 -0800
@@ -1900,8 +1900,9 @@
 {
 	struct net_device *dev = hw->dev[port];
 
-	printk(KERN_INFO PFX "%s: hw error interrupt status 0x%x\n",
-	       dev->name, status);
+	if (printk_ratelimit())
+		printk(KERN_INFO PFX "%s: hw error interrupt status 0x%x\n",
+		       dev->name, status);
 
 	if (status & Y2_IS_PAR_RD1) {
 		printk(KERN_ERR PFX "%s: ram data read parity error\n",
@@ -1918,12 +1919,14 @@
 	}
 
 	if (status & Y2_IS_PAR_MAC1) {
-		printk(KERN_ERR PFX "%s: MAC parity error\n", dev->name);
+		if (printk_ratelimit())
+			printk(KERN_ERR PFX "%s: MAC parity error\n", dev->name);
 		sky2_write8(hw, SK_REG(port, TX_GMF_CTRL_T), GMF_CLI_TX_PE);
 	}
 
 	if (status & Y2_IS_PAR_RX1) {
-		printk(KERN_ERR PFX "%s: RX parity error\n", dev->name);
+		if (printk_ratelimit())
+			printk(KERN_ERR PFX "%s: RX parity error\n", dev->name);
 		sky2_write32(hw, Q_ADDR(rxqaddr[port], Q_CSR), BMU_CLR_IRQ_PAR);
 	}
 
@@ -1935,43 +1938,64 @@
 
 static void sky2_hw_intr(struct sky2_hw *hw)
 {
+	struct pci_dev *pdev = hw->pdev;
 	u32 status = sky2_read32(hw, B0_HWE_ISRC);
 
+	sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
+	/* This driver doesn't use receive timestamps */
 	if (status & Y2_IS_TIST_OV)
 		sky2_write8(hw, GMAC_TI_ST_CTRL, GMT_ST_CLR_IRQ);
 
+	if (status & Y2_IS_PCI_NEXP) {
+		if (printk_ratelimit())
+			printk(KERN_INFO PFX "%s: pci express abort\n",
+			       pci_name(pdev));
+	}
+
 	if (status & (Y2_IS_MST_ERR | Y2_IS_IRQ_STAT)) {
 		u16 pci_err;
 
-		pci_read_config_word(hw->pdev, PCI_STATUS, &pci_err);
-		printk(KERN_ERR PFX "%s: pci hw error (0x%x)\n",
-		       pci_name(hw->pdev), pci_err);
+		pci_read_config_word(pdev, PCI_STATUS, &pci_err);
+		if (printk_ratelimit())
+			printk(KERN_ERR PFX "%s: pci hw error (0x%x)\n",
+			       pci_name(pdev), pci_err);
 
-		sky2_write8(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
-		pci_write_config_word(hw->pdev, PCI_STATUS,
+		pci_write_config_word(pdev, PCI_STATUS,
 				      pci_err | PCI_STATUS_ERROR_BITS);
-		sky2_write8(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
 	}
 
-	if (status & Y2_IS_PCI_EXP) {
-		/* PCI-Express uncorrectable Error occurred */
-		u32 pex_err;
+	if (status & Y2_IS_PCI_EXP) {		/* PCI-Express error */
+		u32 err;
 
-		pci_read_config_dword(hw->pdev, PEX_UNC_ERR_STAT, &pex_err);
+		pci_read_config_dword(pdev, PEX_UNC_ERR_STAT, &err);
+		if (err) {
+			if (net_ratelimit())
+				printk(KERN_INFO PFX
+				       "%s: pci express uncorrectable error (0x%x)\n",
+				       pci_name(pdev), err);
 
-		printk(KERN_ERR PFX "%s: pci express error (0x%x)\n",
-		       pci_name(hw->pdev), pex_err);
+			if (err & PEX_FATAL_ERRORS) {
+				u32 hwmsk = sky2_read32(hw, B0_HWE_IMSK);
 
-		/* clear the interrupt */
-		sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
-		pci_write_config_dword(hw->pdev, PEX_UNC_ERR_STAT,
-				       0xffffffffUL);
-		sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
+				printk(KERN_ERR PFX "%s: adapter failed\n", pci_name(pdev));
 
-		if (pex_err & PEX_FATAL_ERRORS) {
-			u32 hwmsk = sky2_read32(hw, B0_HWE_IMSK);
-			hwmsk &= ~Y2_IS_PCI_EXP;
-			sky2_write32(hw, B0_HWE_IMSK, hwmsk);
+				hwmsk &= ~Y2_IS_PCI_EXP;
+				sky2_write32(hw, B0_HWE_IMSK, hwmsk);
+
+				/* Should we turn off all interrupts?? */
+			}
+
+			/* clear the interrupt */
+			pci_write_config_dword(pdev, PEX_UNC_ERR_STAT, 0xffffffffUL);
+		}
+
+		pci_read_config_dword(pdev, PEX_COR_ERR_STAT, &err);
+		if (err) {
+			if (printk_ratelimit())
+				printk(KERN_INFO PFX
+				       "%s: pci express correctable error (0x%x)\n",
+				       pci_name(pdev), err);
+			pci_write_config_dword(pdev, PEX_COR_ERR_STAT, 0xffffffffUL);
 		}
 	}
 
@@ -1980,6 +2004,8 @@
 	status >>= 8;
 	if (status & Y2_HWE_L1_MASK)
 		sky2_hw_error(hw, 1, status);
+
+	sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
 }
 
 static void sky2_mac_intr(struct sky2_hw *hw, unsigned port)
Index: sky2-work/drivers/net/sky2.h
===================================================================
--- sky2-work.orig/drivers/net/sky2.h	2006-01-11 13:54:23.000000000 -0800
+++ sky2-work/drivers/net/sky2.h	2006-01-11 15:18:12.000000000 -0800
@@ -10,9 +10,25 @@
 #define PCI_DEV_STATUS  0x7c
 #define PCI_OS_PCI_X    (1<<26)
 
-#define PEX_LNK_STAT	0xf2
-#define PEX_UNC_ERR_STAT 0x104
-#define PEX_DEV_CTRL	0xe8
+#define PEX_CAP_ID	0xe0	/*  8 bit	PEX Capability ID */
+#define PEX_NITEM	0xe1	/*  8 bit	PEX Next Item Pointer */
+#define PEX_CAP_REG	0xe2	/* 16 bit	PEX Capability Register */
+#define PEX_DEV_CAP	0xe4	/* 32 bit	PEX Device Capabilities */
+#define PEX_DEV_CTRL	0xe8	/* 16 bit	PEX Device Control */
+#define PEX_DEV_STAT	0xea	/* 16 bit	PEX Device Status */
+#define PEX_LNK_CAP	0xec	/* 32 bit	PEX Link Capabilities */
+#define PEX_LNK_CTRL	0xf0	/* 16 bit	PEX Link Control */
+#define PEX_LNK_STAT	0xf2	/* 16 bit	PEX Link Status */
+
+/* PCI Express Extended Capabilities */
+#define PEX_ADV_ERR_REP	 0x100	/* 32 bit	PEX Advanced Error Reporting */
+#define PEX_UNC_ERR_STAT 0x104	/* 32 bit	PEX Uncorr. Errors Status */
+#define PEX_UNC_ERR_MASK 0x108	/* 32 bit	PEX Uncorr. Errors Mask */
+#define PEX_UNC_ERR_SEV	 0x10c	/* 32 bit	PEX Uncorr. Errors Severity */
+#define PEX_COR_ERR_STAT 0x110	/* 32 bit	PEX Correc. Errors Status */
+#define PEX_COR_ERR_MASK 0x114	/* 32 bit	PEX Correc. Errors Mask */
+#define PEX_ADV_ERR_CAP_C 0x118	/* 32 bit	PEX Advanced Error Cap./Ctrl */
+#define PEX_HEADER_LOG	0x11c	/* 4x32 bit	PEX Header Log Register */
 
 /* Yukon-2 */
 enum pci_dev_reg_1 {

[RFT] sky2: pci express error fix

[Stephen Hemminger]

For all those people suffering with pci express errors
on the sky2 driver.  The problem is the PCI subsystem sometimes
won't let the sky2 driver write to PCI express registers. It depends
on the phase of the moon (actually ACPI) and number of devices.

Anyway, this should fix it. Please tell me if it solves it for you.


--- sky2-2.6.orig/drivers/net/sky2.c
+++ sky2-2.6/drivers/net/sky2.c
@@ -2003,19 +2003,16 @@ static void sky2_hw_intr(struct sky2_hw 
 
 	if (status & Y2_IS_PCI_EXP) {
 		/* PCI-Express uncorrectable Error occurred */
-		u32 pex_err;
-
-		pci_read_config_dword(hw->pdev, PEX_UNC_ERR_STAT, &pex_err);
+		u32 pex_err = sky2_read32(hw, PCI_C(PEX_UNC_ERR_STAT));
 
 		if (net_ratelimit())
 			printk(KERN_ERR PFX "%s: pci express error (0x%x)\n",
 			       pci_name(hw->pdev), pex_err);
 
 		/* clear the interrupt */
-		sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
-		pci_write_config_dword(hw->pdev, PEX_UNC_ERR_STAT,
-				       0xffffffffUL);
-		sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
+		sky2_write8(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
+		sky2_write32(hw, PCI_CI(PEX_UNC_ERR_STAT), 0xffffffffUL);
+		sky2_write8(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
 
 		if (pex_err & PEX_FATAL_ERRORS) {
 			u32 hwmsk = sky2_read32(hw, B0_HWE_IMSK);
@@ -2181,12 +2178,8 @@ static int sky2_reset(struct sky2_hw *hw
 	sky2_write8(hw, B0_CTST, CS_MRST_CLR);
 
 	/* clear any PEX errors */
-	if (is_pciex(hw)) {
-		u16 lstat;
-		pci_write_config_dword(hw->pdev, PEX_UNC_ERR_STAT,
-				       0xffffffffUL);
-		pci_read_config_word(hw->pdev, PEX_LNK_STAT, &lstat);
-	}
+	if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP))
+		sky2_write32(hw, PCI_C(PEX_UNC_ERR_STAT), 0xffffffffUL);
 
 	pmd_type = sky2_read8(hw, B2_PMD_TYP);
 	hw->copper = !(pmd_type == 'L' || pmd_type == 'S');
--- sky2-2.6.orig/drivers/net/sky2.h
+++ sky2-2.6/drivers/net/sky2.h
@@ -183,6 +183,12 @@ enum csr_regs {
 	Y2_CFG_SPC	= 0x1c00,
 };
 
+/* Workaround for ACPI limitations in pci support.
+ * Sometimes it is impossible to access registers > 256 with
+ * pci_{read/write}_config_dword
+ */
+#define PCI_C(reg)	(Y2_CFG_SPC + reg)
+
 /*	B0_CTST			16 bit	Control/Status register */
 enum {
 	Y2_VMAIN_AVAIL	= 1<<17,/* VMAIN available (YUKON-2 only) */

Re: [RFT] sky2: pci express error fix

[Kalin KOZHUHAROV]

Stephen Hemminger wrote:
> For all those people suffering with pci express errors
> on the sky2 driver.  The problem is the PCI subsystem sometimes
> won't let the sky2 driver write to PCI express registers. It depends
> on the phase of the moon (actually ACPI) and number of devices.
> 
> Anyway, this should fix it. Please tell me if it solves it for you.

Can you describe the bug a bit more? What happens?

I had a few times something like this:

[   24.145040] sky2 eth0: phy interrupt status 0x1c40 0xbc0c

[ 3647.341757] sky2 eth0: phy interrupt status 0x1c40 0xbc4c

after which all network was dead. (and it wasn't a module so had to
restart). As you can see from the above two logs, sometimes it failed on
boot, sometimes after an hour. Sourry, I didn't remember the phase of the
moon, but I can check :-)

I have two Asus P5GDC-V Deluxe boards, with these chips. One of them is
happily working with sk98lin (the binary one), the other is dying miserably,
so now I use r8169 card to be able to isolate the problem (separate mail).

Kalin.

> 
> 
> --- sky2-2.6.orig/drivers/net/sky2.c
> +++ sky2-2.6/drivers/net/sky2.c
> @@ -2003,19 +2003,16 @@ static void sky2_hw_intr(struct sky2_hw 
>  
>  	if (status & Y2_IS_PCI_EXP) {
>  		/* PCI-Express uncorrectable Error occurred */
> -		u32 pex_err;
> -
> -		pci_read_config_dword(hw->pdev, PEX_UNC_ERR_STAT, &pex_err);
> +		u32 pex_err = sky2_read32(hw, PCI_C(PEX_UNC_ERR_STAT));
>  
>  		if (net_ratelimit())
>  			printk(KERN_ERR PFX "%s: pci express error (0x%x)\n",
>  			       pci_name(hw->pdev), pex_err);
>  
>  		/* clear the interrupt */
> -		sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
> -		pci_write_config_dword(hw->pdev, PEX_UNC_ERR_STAT,
> -				       0xffffffffUL);
> -		sky2_write32(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
> +		sky2_write8(hw, B2_TST_CTRL1, TST_CFG_WRITE_ON);
> +		sky2_write32(hw, PCI_CI(PEX_UNC_ERR_STAT), 0xffffffffUL);
> +		sky2_write8(hw, B2_TST_CTRL1, TST_CFG_WRITE_OFF);
>  
>  		if (pex_err & PEX_FATAL_ERRORS) {
>  			u32 hwmsk = sky2_read32(hw, B0_HWE_IMSK);
> @@ -2181,12 +2178,8 @@ static int sky2_reset(struct sky2_hw *hw
>  	sky2_write8(hw, B0_CTST, CS_MRST_CLR);
>  
>  	/* clear any PEX errors */
> -	if (is_pciex(hw)) {
> -		u16 lstat;
> -		pci_write_config_dword(hw->pdev, PEX_UNC_ERR_STAT,
> -				       0xffffffffUL);
> -		pci_read_config_word(hw->pdev, PEX_LNK_STAT, &lstat);
> -	}
> +	if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP))
> +		sky2_write32(hw, PCI_C(PEX_UNC_ERR_STAT), 0xffffffffUL);
>  
>  	pmd_type = sky2_read8(hw, B2_PMD_TYP);
>  	hw->copper = !(pmd_type == 'L' || pmd_type == 'S');
> --- sky2-2.6.orig/drivers/net/sky2.h
> +++ sky2-2.6/drivers/net/sky2.h
> @@ -183,6 +183,12 @@ enum csr_regs {
>  	Y2_CFG_SPC	= 0x1c00,
>  };
>  
> +/* Workaround for ACPI limitations in pci support.
> + * Sometimes it is impossible to access registers > 256 with
> + * pci_{read/write}_config_dword
> + */
> +#define PCI_C(reg)	(Y2_CFG_SPC + reg)
> +
>  /*	B0_CTST			16 bit	Control/Status register */
>  enum {
>  	Y2_VMAIN_AVAIL	= 1<<17,/* VMAIN available (YUKON-2 only) */


-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|

Re: [RFT] sky2: pci express error fix

[Stephen Hemminger]

On Fri, 27 Jan 2006 01:11:20 +0900
Kalin KOZHUHAROV <kalin@thinrope.net> wrote:

> Stephen Hemminger wrote:
> > For all those people suffering with pci express errors
> > on the sky2 driver.  The problem is the PCI subsystem sometimes
> > won't let the sky2 driver write to PCI express registers. It depends
> > on the phase of the moon (actually ACPI) and number of devices.
> > 
> > Anyway, this should fix it. Please tell me if it solves it for you.
> 
> Can you describe the bug a bit more? What happens?
> 
> I had a few times something like this:
> 
> [   24.145040] sky2 eth0: phy interrupt status 0x1c40 0xbc0c

> 
> [ 3647.341757] sky2 eth0: phy interrupt status 0x1c40 0xbc4c
>

Looks like a noisy crappy cable causing PHY link status changes.


> after which all network was dead. (and it wasn't a module so had to
> restart). As you can see from the above two logs, sometimes it failed on
> boot, sometimes after an hour. Sourry, I didn't remember the phase of the
> moon, but I can check :-)
> 
> I have two Asus P5GDC-V Deluxe boards, with these chips. One of them is
> happily working with sk98lin (the binary one), the other is dying miserably,
> so now I use r8169 card to be able to isolate the problem (separate mail).
> 
>

Re: [RFT] sky2: pci express error fix

[Kalin KOZHUHAROV]

Stephen Hemminger wrote:
> On Fri, 27 Jan 2006 01:11:20 +0900
> Kalin KOZHUHAROV <kalin@thinrope.net> wrote:
> 
>> Stephen Hemminger wrote:
>>> For all those people suffering with pci express errors
>>> on the sky2 driver.  The problem is the PCI subsystem sometimes
>>> won't let the sky2 driver write to PCI express registers. It depends
>>> on the phase of the moon (actually ACPI) and number of devices.
>>>
>>> Anyway, this should fix it. Please tell me if it solves it for you.
>> Can you describe the bug a bit more? What happens?
>>
>> I had a few times something like this:
>>
>> [   24.145040] sky2 eth0: phy interrupt status 0x1c40 0xbc0c
> 
>> [ 3647.341757] sky2 eth0: phy interrupt status 0x1c40 0xbc4c
>>
> 
> Looks like a noisy crappy cable causing PHY link status changes.

I will check that particular cable on monday, but it *should* be CAT6 not
very cheap cable (Japanese brand: Elecom). In the worst case it is a CAT5e
from probably the same brand. (just because there are no other cables in the
office)

Can these errors be caused by bad hub?
>From what I've seen the network on this machine dies at the same time (not
sure before or after) as the error in dmesg.

>> after which all network was dead. (and it wasn't a module so had to
>> restart). As you can see from the above two logs, sometimes it failed on
>> boot, sometimes after an hour. Sourry, I didn't remember the phase of the
>> moon, but I can check :-)
>>
>> I have two Asus P5GDC-V Deluxe boards, with these chips. One of them is
>> happily working with sk98lin (the binary one), the other is dying miserably,
>> so now I use r8169 card to be able to isolate the problem (separate mail).
Forgot to post it, now it went under the subject:
	libata errors in 2.6.15.1 ICH6 AHCI (SATA drive WD740GD)

Kalin.

-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|

Re: [RFT] sky2: pci express error fix

[Ansgar Esztermann]

Stephen,

> For all those people suffering with pci express errors
> on the sky2 driver.  The problem is the PCI subsystem sometimes
> won't let the sky2 driver write to PCI express registers. It depends
> on the phase of the moon (actually ACPI) and number of devices.
> 
> Anyway, this should fix it. Please tell me if it solves it for you.

I've applied your patch to an -mm4 soure tree, and the error messages
did stop. Thanks!

BTW, I think there is a typo in this line:
+ sky2_write32(hw, PCI_CI(PEX_UNC_ERR_STAT), 0xffffffffUL);
(it should be PCI_C, not PCI_CI).

A.
-- 
Ansgar Esztermann
Researcher & Sysadmin
http://www2.thphy.uni-duesseldorf.de/~ansgar