Bogus frames transmitted with r8169 & fragmentation & large mtu

Bogus frames transmitted with r8169 & fragmentation & large mtu

[Nick Bowler]

[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]

Hi folks,

We were testing IPsec with large mtu sizes (9000 bytes) and noticed the
occasional failure with large datagrams (requiring several fragments,
>=25k bytes or so).  Investigating further, I was able to reproduce the
issue without using IPsec at all.  Looking at the wireshark capture I
see that one of the fragments transmitted is totally bogus: portions of
the payload data have made it onto the wire as the headers.  My test
case was this:

  ping -c 1 -s 30000 -p 42 birch

which is split into 4 frames, 3 of which (first, second and last) look
correct but the fourth consists entirely of 0x42 octets, including the
ethernet and IP headers (so the source address is 42:42:42:42:42:42, the
destination address is 42:42:42:42:42:42, ethertype is 0x4242, etc.)
I've attached the wireshark capture (gzipped) since it's small enough.
Nevertheless, the total length is correct for the missing fragment.

It seems to be specific to the particular network chipset on this
machine (using the r8169 driver):

  Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)

because the other machine (which has a Marvell controller) transmits
fine as far as I can tell.  I'll see about putting a discrete network
card in this computer to see if that helps any.

It's not entirely consistent in how things fail.  Sufficiently small
pings (say, <=20k bytes) seem to work 100% of the time, sufficiently
large pings (say, >= 40k bytes) seem to fail 100% of the time, and in
between most pings fail but the occasional one makes it through.
Everything works at all sizes with 1500 byte MTU.

There are no unusual messages in dmesg and this was all run with latest
Linus' git as of this morning.  It occurred with somewhat older kernels
(2.6.39) as well, even though the driver only let me go to 7200 byte mtu
on that version.

Let me know if you need any more info,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

[-- Attachment #2: fragment-9000.cap.gz --]
[-- Type: application/octet-stream, Size: 527 bytes --]

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Eric Dumazet]

Le mercredi 15 février 2012 à 11:37 -0500, Nick Bowler a écrit :
> Hi folks,
> 
> We were testing IPsec with large mtu sizes (9000 bytes) and noticed the
> occasional failure with large datagrams (requiring several fragments,
> >=25k bytes or so).  Investigating further, I was able to reproduce the
> issue without using IPsec at all.  Looking at the wireshark capture I
> see that one of the fragments transmitted is totally bogus: portions of
> the payload data have made it onto the wire as the headers.  My test
> case was this:
> 
>   ping -c 1 -s 30000 -p 42 birch
> 
> which is split into 4 frames, 3 of which (first, second and last) look
> correct but the fourth consists entirely of 0x42 octets, including the
> ethernet and IP headers (so the source address is 42:42:42:42:42:42, the
> destination address is 42:42:42:42:42:42, ethertype is 0x4242, etc.)
> I've attached the wireshark capture (gzipped) since it's small enough.
> Nevertheless, the total length is correct for the missing fragment.
> 
> It seems to be specific to the particular network chipset on this
> machine (using the r8169 driver):
> 
>   Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
> 
> because the other machine (which has a Marvell controller) transmits
> fine as far as I can tell.  I'll see about putting a discrete network
> card in this computer to see if that helps any.
> 
> It's not entirely consistent in how things fail.  Sufficiently small
> pings (say, <=20k bytes) seem to work 100% of the time, sufficiently
> large pings (say, >= 40k bytes) seem to fail 100% of the time, and in
> between most pings fail but the occasional one makes it through.
> Everything works at all sizes with 1500 byte MTU.
> 
> There are no unusual messages in dmesg and this was all run with latest
> Linus' git as of this morning.  It occurred with somewhat older kernels
> (2.6.39) as well, even though the driver only let me go to 7200 byte mtu
> on that version.
> 
> Let me know if you need any more info,

Interesting, but 2nd frame is also corrupted (not at the start but in
the middle)

Where was taken this trace exactly ?

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Nick Bowler]

On 2012-02-15 20:13 +0100, Eric Dumazet wrote:
> Le mercredi 15 février 2012 à 11:37 -0500, Nick Bowler a écrit :
> > We were testing IPsec with large mtu sizes (9000 bytes) and noticed the
> > occasional failure with large datagrams (requiring several fragments,
> > >=25k bytes or so).  Investigating further, I was able to reproduce the
> > issue without using IPsec at all.  Looking at the wireshark capture I
> > see that one of the fragments transmitted is totally bogus: portions of
> > the payload data have made it onto the wire as the headers.  My test
> > case was this:
> > 
> >   ping -c 1 -s 30000 -p 42 birch
> > 
> > which is split into 4 frames, 3 of which (first, second and last) look
> > correct but the fourth consists entirely of 0x42 octets, including the
> > ethernet and IP headers (so the source address is 42:42:42:42:42:42, the
> > destination address is 42:42:42:42:42:42, ethertype is 0x4242, etc.)
> > I've attached the wireshark capture (gzipped) since it's small enough.
> > Nevertheless, the total length is correct for the missing fragment.
[...]
> Interesting, but 2nd frame is also corrupted (not at the start but in
> the middle)

So it is, I missed that.  That looks like the missing headers stuck in
the middle of that frame, although the fragment offset appears to be
wrong (and the more fragments flag is clear).  There's also two 0 bytes
slightly before the headers.

> Where was taken this trace exactly ?

Wireshark was run on the receiving machine (birch), but it was captured
using a separate NIC via a monitor port on the switch.

Thanks,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Nick Bowler]

On 2012-02-15 14:58 -0500, Nick Bowler wrote:
> On 2012-02-15 20:13 +0100, Eric Dumazet wrote:
> > Le mercredi 15 février 2012 à 11:37 -0500, Nick Bowler a écrit :
> > > We were testing IPsec with large mtu sizes (9000 bytes) and noticed the
> > > occasional failure with large datagrams (requiring several fragments,
> > > >=25k bytes or so).  Investigating further, I was able to reproduce the
> > > issue without using IPsec at all.  Looking at the wireshark capture I
> > > see that one of the fragments transmitted is totally bogus: portions of
> > > the payload data have made it onto the wire as the headers.  My test
> > > case was this:
> > > 
> > >   ping -c 1 -s 30000 -p 42 birch
> > > 
> > > which is split into 4 frames, 3 of which (first, second and last) look
> > > correct but the fourth consists entirely of 0x42 octets, including the
> > > ethernet and IP headers (so the source address is 42:42:42:42:42:42, the
> > > destination address is 42:42:42:42:42:42, ethertype is 0x4242, etc.)
> > > I've attached the wireshark capture (gzipped) since it's small enough.
> > > Nevertheless, the total length is correct for the missing fragment.
> [...]
> > Interesting, but 2nd frame is also corrupted (not at the start but in
> > the middle)
> 
> So it is, I missed that.  That looks like the missing headers stuck in
> the middle of that frame, although the fragment offset appears to be
> wrong (and the more fragments flag is clear).  There's also two 0 bytes
> slightly before the headers.

Ah, I was misreading the fragment offset field.  That data in the
second packet looks like the headers for the final fragment (offset of
26928, MF clear), not the missing one.

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Francois Romieu]

Nick Bowler <nbowler@elliptictech.com> :
[...]
> Let me know if you need any more info,

Please send the XID line from the r8169 driver for this device.

Did you enable some offload feature (see 'ethtool -k') ?

-- 
Ueimor

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Francois Romieu]

Eric Dumazet <eric.dumazet@gmail.com> :
> Le mercredi 15 février 2012 à 11:37 -0500, Nick Bowler a écrit :
[...]
> > Let me know if you need any more info,
> 
> Interesting, but 2nd frame is also corrupted (not at the start but in
> the middle)

And the corruption looks like a packet header. Compare :

0000  00 25 11 8c 1c 3d bc ae  c5 27 19 6f 08 00 45 00   .%...=.. .'.o..E.
0010  23 24 5d 69 20 00 40 01  46 a5 c0 a8 09 41 c0 a8   #$]i .@. F....A..
0020  09 39 08 00 d9 b6 0c 6b  00 01 d1 d2 3b 4f ad 69   .9.....k ....;O.i
0030  06 00 42 42 42 42 42 42  42 42 42 42 42 42 42 42   ..BBBBBB BBBBBBBB

with :

(lots of 42)
1f00  00 25 11 8c 1c 3d bc ae  c5 27 19 6f 08 00 45 00   .%...=.. .'.o..E.
1f10  0c 1c 5d 69 0d 26 40 01  70 87 c0 a8 09 41 c0 a8   ..]i.&@. p....A..
                 (^^^^^ fragment offset 6694)
1f20  09 39 42 42 42 42 42 42  42 42 42 42 42 42 42 42   .9BBBBBB BBBBBBBB

Mac addresses and ip addresses match.

Hayes, what is the story with ICMP packets and IP flags ?

-- 
Ueimor

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Nick Bowler]

On 2012-02-15 23:41 +0100, Francois Romieu wrote:
> Nick Bowler <nbowler@elliptictech.com> :
> [...]
> > Let me know if you need any more info,
> 
> Please send the XID line from the r8169 driver for this device.

This one?

  [   10.344571] r8169 0000:02:00.0: eth0: RTL8168d/8111d at 0xf8064000, bc:ae:c5:27:19:6f, XID 083000c0 IRQ 41

> Did you enable some offload feature (see 'ethtool -k') ?

Not beyond the defaults as far as I know.  Here's the ethtool -k output:

  Offload parameters for p10p1:
  rx-checksumming: on
  tx-checksumming: off
  scatter-gather: off
  tcp-segmentation-offload: off
  udp-fragmentation-offload: off
  generic-segmentation-offload: off
  generic-receive-offload: on
  large-receive-offload: off
  rx-vlan-offload: on
  tx-vlan-offload: on
  ntuple-filters: off
  receive-hashing: off

Thanks,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Eric Dumazet]

Le mercredi 15 février 2012 à 18:34 -0500, Nick Bowler a écrit :
> On 2012-02-15 23:41 +0100, Francois Romieu wrote:
> > Nick Bowler <nbowler@elliptictech.com> :
> > [...]
> > > Let me know if you need any more info,
> > 
> > Please send the XID line from the r8169 driver for this device.
> 
> This one?
> 
>   [   10.344571] r8169 0000:02:00.0: eth0: RTL8168d/8111d at 0xf8064000, bc:ae:c5:27:19:6f, XID 083000c0 IRQ 41
> 
> > Did you enable some offload feature (see 'ethtool -k') ?
> 
> Not beyond the defaults as far as I know.  Here's the ethtool -k output:
> 
>   Offload parameters for p10p1:
>   rx-checksumming: on
>   tx-checksumming: off
>   scatter-gather: off
>   tcp-segmentation-offload: off
>   udp-fragmentation-offload: off
>   generic-segmentation-offload: off
>   generic-receive-offload: on
>   large-receive-offload: off
>   rx-vlan-offload: on
>   tx-vlan-offload: on
>   ntuple-filters: off
>   receive-hashing: off
> 
> Thanks,

1) Are you using SLUB, SLAB, or SLOB ?

2) Problem with MTU=9000 is that frames span several 4K pages.

Maybe some versions of r8169 hardware have problems with that ...

3) We could check in xmit handler if the frames are correct before
sending them.  (Adding a trace of ICMP frames, and check if the payload
is all 0x42 filler or not) 

RE: Bogus frames transmitted with r8169 & fragmentation & large mtu

[hayeswang]

 

> -----Original Message-----
> From: Francois Romieu [mailto:romieu@fr.zoreil.com] 
> 
> Mac addresses and ip addresses match.
> 
> Hayes, what is the story with ICMP packets and IP flags ?
> 

Please add Cfg9346_Unlock before calling rtl_hw_jumbo_enable or
hw_jumbo_disable.
And, add Cfg9346_lock after calling rlt_hw_jumbo_enable or hw_jumbo_disable. 

Best Regards,
Hayes

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Francois Romieu]

hayeswang <hayeswang@realtek.com> :
[...]
> Please add Cfg9346_Unlock before calling rtl_hw_jumbo_enable or
> hw_jumbo_disable.
> And, add Cfg9346_lock after calling rlt_hw_jumbo_enable or hw_jumbo_disable. 

Thanks.

I am testing it now.

diff --git a/drivers/net/ethernet/realtek/r8169.c b/drivers/net/ethernet/realtek/r8169.c
index 5eb6858..81e6ea2 100644
--- a/drivers/net/ethernet/realtek/r8169.c
+++ b/drivers/net/ethernet/realtek/r8169.c
@@ -3833,12 +3833,20 @@ static void rtl8169_init_ring_indexes(struct rtl8169_private *tp)
 
 static void rtl_hw_jumbo_enable(struct rtl8169_private *tp)
 {
+	void __iomem *ioaddr = tp->mmio_addr;
+
+	RTL_W8(Cfg9346, Cfg9346_Unlock);
 	rtl_generic_op(tp, tp->jumbo_ops.enable);
+	RTL_W8(Cfg9346, Cfg9346_Lock);
 }
 
 static void rtl_hw_jumbo_disable(struct rtl8169_private *tp)
 {
+	void __iomem *ioaddr = tp->mmio_addr;
+
+	RTL_W8(Cfg9346, Cfg9346_Unlock);
 	rtl_generic_op(tp, tp->jumbo_ops.disable);
+	RTL_W8(Cfg9346, Cfg9346_Lock);
 }
 
 static void r8168c_hw_jumbo_enable(struct rtl8169_private *tp)

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[joeyli]

Hi Nick, 

於 三，2012-02-15 於 18:34 -0500，Nick Bowler 提到：
> On 2012-02-15 23:41 +0100, Francois Romieu wrote:
> > Nick Bowler <nbowler@elliptictech.com> :
> > [...]
> > > Let me know if you need any more info,
> > 
> > Please send the XID line from the r8169 driver for this device.
> 
> This one?
> 
>   [   10.344571] r8169 0000:02:00.0: eth0: RTL8168d/8111d at 0xf8064000, bc:ae:c5:27:19:6f, XID 083000c0 IRQ 41
> 

Did you see any message from r8169 in dmesg said lost firmware file on
your system?


Thanks
Joey Lee

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Francois Romieu]

joeyli <jlee@suse.com> :
[...]
> Did you see any message from r8169 in dmesg said lost firmware file on
> your system?

I can reproduce the problem with the same chipset as Nick (same XID) and
the firmware is loaded.

-- 
Ueimor

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Francois Romieu]

Francois Romieu <romieu@fr.zoreil.com> :
[...]
> I am testing it now.

It does not fix the problem. It seems to make a difference but I still
see a few percents packet loss. The pattern has changed a bit too: there
are more bogus IP fragment offsets.

Trying with 'ping -qf -c 100 -l 2 -s 65507 10.0.7.1'

Good:

239   0.020107     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=0, ID=9f97)
240   0.020116     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=8976, ID=9f97)
241   0.020122     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=17952, ID=9f97)
242   0.020128     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=26928, ID=9f97)
243   0.020134     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=35904, ID=9f97)
244   0.020139     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=44880, ID=9f97)
245   0.020145     10.0.7.1 -> 10.0.7.5     IP Fragmented IP protocol (proto=ICMP 0x01, off=53856, ID=9f97)
246   0.020150     10.0.7.1 -> 10.0.7.5     ICMP Echo (ping) reply    (id=0x04c7, seq(be/le)=16/4096, ttl=64)

Bad:

247   0.020809     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=0, ID=dce9)
248   0.020977     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=8976, ID=dce9)
249   0.021046     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=17952, ID=dce9)
250   0.021192     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=26928, ID=dce9)
251   0.021199     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=35904, ID=dce9)
252   0.021291     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=44880, ID=dce9)
253   0.021298     10.0.7.5 -> 10.0.7.1     IP Fragmented IP protocol (proto=ICMP 0x01, off=62832, ID=dce9)

So far I have only seen it with the 2nd and last fragment.

-- 
Ueimor

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Nick Bowler]

On 2012-02-16 05:19 +0100, Eric Dumazet wrote:
> 1) Are you using SLUB, SLAB, or SLOB ?

SLUB.

> 2) Problem with MTU=9000 is that frames span several 4K pages.
> 
> Maybe some versions of r8169 hardware have problems with that ...

Could be, but probably not the whole story.  I tested some >4k MTU values
(all with 60K packets here because it failed more reliably than 30K).
6500 byte MTU appears to work OK (<1% loss).  7500 fails (>99% loss).
A largish range of MTU settings inbetween achieve a roughly 50% loss
rate.

The corruption is not always as significant as the original trace: some
of my tests had all the fragments correct but a mere two bytes of payload
were zero (resulting in the reassembled datagram being dropped due to a
checksum failure).

Recall that the original trace also had this "two bytes zeroed" problem
in the second fragment, in addition to all its other problems.

Cheers,
-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)

RE: Bogus frames transmitted with r8169 & fragmentation & large mtu

[hayeswang]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

> From: Francois Romieu [mailto:romieu@fr.zoreil.com] 
> Sent: Thursday, February 16, 2012 8:05 PM
> To: Hayeswang
> Cc: 'Eric Dumazet'; 'Nick Bowler'; netdev@vger.kernel.org
> Subject: Re: Bogus frames transmitted with r8169 & 
> fragmentation & large mtu
> 
> Francois Romieu <romieu@fr.zoreil.com> :
> [...]
> > I am testing it now.
> 
> It does not fix the problem. It seems to make a difference but I still
> see a few percents packet loss. The pattern has changed a bit 
> too: there
> are more bogus IP fragment offsets.
> 
> Trying with 'ping -qf -c 100 -l 2 -s 65507 10.0.7.1'

It works fine for me. I use Linux platform (Fedora 16 with kernel 3.2.0 with
that patch) with the realtek nic to ping Windows platform with the NVIDIA nic,
and I connect the two PCs directly without the switch. It works well except for
the first packet after I set the new mtu.

> 
> So far I have only seen it with the 2nd and last fragment.
> 
 
Best Regards,
Hayes

[-- Attachment #2: ping.log --]
[-- Type: application/octet-stream, Size: 1257 bytes --]

[   45.415894] r8169 0000:04:00.0: eth0: RTL8168d/8111d at 0xf7e58000, 00:e0:4c:68:00:04, XID 083000c0 IRQ 46
p33p1     Link encap:Ethernet  HWaddr 00:E0:4C:68:00:04  
          inet addr:192.168.94.44  Bcast:192.168.94.255  Mask:255.255.255.0
          inet6 addr: fe80::2e0:4cff:fe68:4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:3714 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3670 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:28794250 (27.4 MiB)  TX bytes:28919110 (27.5 MiB)
          Interrupt:46 Base address:0x8000 
PING 192.168.94.20 (192.168.94.20) 65507(65535) bytes of data.

--- 192.168.94.20 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 89ms
rtt min/avg/max/mdev = 1.368/1.643/1.954/0.099 ms, pipe 2, ipg/ewma 0.904/1.705 ms
PING 192.168.94.20 (192.168.94.20) 65507(65535) bytes of data.

--- 192.168.94.20 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 90ms
rtt min/avg/max/mdev = 1.419/1.658/1.943/0.078 ms, pipe 2, ipg/ewma 0.912/1.673 ms
Linux fc16.localdomain 3.2.0 #1 SMP Tue Jan 10 16:11:29 CST 2012 i686 i686 i386 GNU/Linux

Re: Bogus frames transmitted with r8169 & fragmentation & large mtu

[Francois Romieu]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

hayeswang <hayeswang@realtek.com> :
> > Francois Romieu <romieu@fr.zoreil.com> :
[...]
> > Trying with 'ping -qf -c 100 -l 2 -s 65507 10.0.7.1'
> 
> It works fine for me. I use Linux platform (Fedora 16 with kernel 3.2.0 with
> that patch) with the realtek nic to ping Windows platform with the NVIDIA nic,
> and I connect the two PCs directly without the switch. It works well except for
> the first packet after I set the new mtu.

No switch, old PCI e1000, Fedora 15, d5ef8a4d87ab21d575ac86366599c9152a28028d
(post -rc3) from davem here. Imvho you are not trying hard enough to break
your toys :o)

What about :

# sysctl -w net.core.rmem_max=1000000
# ping -qf -c 10000 -l 8 -s 65507 10.0.7.1

If it's not enough you can increase '-c' and '-l' further. I see no loss
with '-l 1', event after 100k packets.

> > So far I have only seen it with the 2nd and last fragment.

... which should have read as "the hardware problem is gone, stop using SLUB
if you do not want to see intra-packet corruption".

However I still see it with SLAB (.config attached):

# ping -qf -c 10000 -l 2 -s 65507  10.0.7.1
PING 10.0.7.1 (10.0.7.1) 65507(65535) bytes of data.

--- 10.0.7.1 ping statistics ---
10000 packets transmitted, 9916 received, 0% packet loss, time 15898ms
rtt min/avg/max/mdev = 2.154/2.847/3.256/0.138 ms, pipe 2, ipg/ewma 1.589/2.867 ms

Almost mundane .5 %. Alas:

# ping -qf -c 6 -l 4 -s 65507  10.0.7.1
PING 10.0.7.1 (10.0.7.1) 65507(65535) bytes of data.

--- 10.0.7.1 ping statistics ---
6 packets transmitted, 4 received, 33% packet loss, time 4ms
rtt min/avg/max/mdev = 2.831/3.872/4.896/0.765 ms, pipe 4, ipg/ewma 0.874/3.309 ms

I'll follow Eric's suggestion with checking the packet content in start_xmit.

-- 
Ueimor

[-- Attachment #2: config-3.3.0-rc3-dmn+.gz --]
[-- Type: application/x-gzip, Size: 21527 bytes --]