[PATCH] netdev: pktgen xmit packet through vlan interface

[PATCH] netdev: pktgen xmit packet through vlan interface

[Zhouyi Zhou]

As http://www.spinics.net/lists/netdev/msg165015.html
pktgen generates shared packet through vlan interface will cause
oops because of duplicate entering tc queue.

Try to solve this problem by means of packet clone instead of sharing.

Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
---
 net/core/pktgen.c |   20 +++++++++++++++++---
 1 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 0304f98..ced07fc 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -3297,6 +3297,7 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
 	netdev_tx_t (*xmit)(struct sk_buff *, struct net_device *)
 		= odev->netdev_ops->ndo_start_xmit;
 	struct netdev_queue *txq;
+	struct sk_buff *nskb = NULL;
 	u16 queue_map;
 	int ret;
 
@@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
 		pkt_dev->last_ok = 0;
 		goto unlock;
 	}
-	atomic_inc(&(pkt_dev->skb->users));
-	ret = (*xmit)(pkt_dev->skb, odev);
+
+	if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
+		nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
+		ret = -ENOMEM;
+		if (nskb)
+			ret = (*xmit)(nskb, odev);
+		else
+			nskb = ERR_PTR(ret);
+	} else {
+		atomic_inc(&(pkt_dev->skb->users));
+		ret = (*xmit)(pkt_dev->skb, odev);
+	}
 
 	switch (ret) {
 	case NETDEV_TX_OK:
@@ -3372,7 +3383,10 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
 	case NETDEV_TX_LOCKED:
 	case NETDEV_TX_BUSY:
 		/* Retry it next time */
-		atomic_dec(&(pkt_dev->skb->users));
+		if (nskb && !IS_ERR(nskb))
+			kfree_skb(nskb);
+		else
+			atomic_dec(&(pkt_dev->skb->users));
 		pkt_dev->last_ok = 0;
 	}
 unlock:
-- 
1.7.1

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Jesper Dangaard Brouer]

On Fri,  2 May 2014 15:18:12 +0800
Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:

> As http://www.spinics.net/lists/netdev/msg165015.html
> pktgen generates shared packet through vlan interface will cause
> oops because of duplicate entering tc queue.
>
> Try to solve this problem by means of packet clone instead of sharing.

I really don't like adding this stuff to the fast path of pktgen.

Why would you use pktgen on a VLAN?

Why don't you use the "vlan_id" feature available in pktgen, and send
in the lower real device?



> Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
> ---
>  net/core/pktgen.c |   20 +++++++++++++++++---
>  1 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
> index 0304f98..ced07fc 100644
> --- a/net/core/pktgen.c
> +++ b/net/core/pktgen.c
> @@ -3297,6 +3297,7 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>  	netdev_tx_t (*xmit)(struct sk_buff *, struct net_device *)
>  		= odev->netdev_ops->ndo_start_xmit;
>  	struct netdev_queue *txq;
> +	struct sk_buff *nskb = NULL;
>  	u16 queue_map;
>  	int ret;
>  
> @@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>  		pkt_dev->last_ok = 0;
>  		goto unlock;
>  	}
> -	atomic_inc(&(pkt_dev->skb->users));
> -	ret = (*xmit)(pkt_dev->skb, odev);
> +
> +	if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
> +		nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
> +		ret = -ENOMEM;
> +		if (nskb)
> +			ret = (*xmit)(nskb, odev);
> +		else
> +			nskb = ERR_PTR(ret);
> +	} else {
> +		atomic_inc(&(pkt_dev->skb->users));
> +		ret = (*xmit)(pkt_dev->skb, odev);
> +	}
>  
>  	switch (ret) {
>  	case NETDEV_TX_OK:
> @@ -3372,7 +3383,10 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>  	case NETDEV_TX_LOCKED:
>  	case NETDEV_TX_BUSY:
>  		/* Retry it next time */
> -		atomic_dec(&(pkt_dev->skb->users));
> +		if (nskb && !IS_ERR(nskb))
> +			kfree_skb(nskb);
> +		else
> +			atomic_dec(&(pkt_dev->skb->users));
>  		pkt_dev->last_ok = 0;
>  	}
>  unlock:



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[John Fastabend]

On 5/2/2014 6:19 AM, Jesper Dangaard Brouer wrote:
> On Fri,  2 May 2014 15:18:12 +0800
> Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
>
>> As http://www.spinics.net/lists/netdev/msg165015.html
>> pktgen generates shared packet through vlan interface will cause
>> oops because of duplicate entering tc queue.
>>
>> Try to solve this problem by means of packet clone instead of sharing.
>
> I really don't like adding this stuff to the fast path of pktgen.
>
> Why would you use pktgen on a VLAN?

Its a good way to test qdiscs. When you run pktgen over the VLAN
you exercise the lower devices qdisc.

Although I never submitted a patch like this because I figured it
was a corner case and we would want to keep the hotpath clean.

>
> Why don't you use the "vlan_id" feature available in pktgen, and send
> in the lower real device?
>
>
>
>> Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
>> ---
>>   net/core/pktgen.c |   20 +++++++++++++++++---
>>   1 files changed, 17 insertions(+), 3 deletions(-)
>>
>> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
>> index 0304f98..ced07fc 100644
>> --- a/net/core/pktgen.c
>> +++ b/net/core/pktgen.c
>> @@ -3297,6 +3297,7 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>>   	netdev_tx_t (*xmit)(struct sk_buff *, struct net_device *)
>>   		= odev->netdev_ops->ndo_start_xmit;
>>   	struct netdev_queue *txq;
>> +	struct sk_buff *nskb = NULL;
>>   	u16 queue_map;
>>   	int ret;
>>
>> @@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>>   		pkt_dev->last_ok = 0;
>>   		goto unlock;
>>   	}
>> -	atomic_inc(&(pkt_dev->skb->users));
>> -	ret = (*xmit)(pkt_dev->skb, odev);
>> +
>> +	if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
>> +		nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
>> +		ret = -ENOMEM;
>> +		if (nskb)
>> +			ret = (*xmit)(nskb, odev);
>> +		else
>> +			nskb = ERR_PTR(ret);
>> +	} else {
>> +		atomic_inc(&(pkt_dev->skb->users));
>> +		ret = (*xmit)(pkt_dev->skb, odev);
>> +	}
>>
>>   	switch (ret) {
>>   	case NETDEV_TX_OK:
>> @@ -3372,7 +3383,10 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>>   	case NETDEV_TX_LOCKED:
>>   	case NETDEV_TX_BUSY:
>>   		/* Retry it next time */
>> -		atomic_dec(&(pkt_dev->skb->users));
>> +		if (nskb && !IS_ERR(nskb))
>> +			kfree_skb(nskb);
>> +		else
>> +			atomic_dec(&(pkt_dev->skb->users));
>>   		pkt_dev->last_ok = 0;
>>   	}
>>   unlock:
>
>
>

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Zhouyi Zhou]

Thanks for reviewing

On Fri, May 2, 2014 at 10:00 PM, John Fastabend
<john.r.fastabend@intel.com> wrote:
> On 5/2/2014 6:19 AM, Jesper Dangaard Brouer wrote:
>>
>> On Fri,  2 May 2014 15:18:12 +0800
>> Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
>>
>>> As http://www.spinics.net/lists/netdev/msg165015.html
>>> pktgen generates shared packet through vlan interface will cause
>>> oops because of duplicate entering tc queue.
>>>
>>> Try to solve this problem by means of packet clone instead of sharing.
>>
>>
>> I really don't like adding this stuff to the fast path of pktgen.


>>
>> Why would you use pktgen on a VLAN?

I use pktgen on a VLAN under a special circumstance.

>
>
> Its a good way to test qdiscs. When you run pktgen over the VLAN
> you exercise the lower devices qdisc.
>
> Although I never submitted a patch like this because I figured it
> was a corner case and we would want to keep the hotpath clean.
>

I think is_vlan_dev(odev) have great chances to be in a cache line
that seldom got reset. Could we only judge is_vlan_dev(odev) in my
first if statement?


Also I guess if (nskb && !IS_ERR(nskb)) happens in rarely reached
cases when the interface is not overload.

>
>>
>> Why don't you use the "vlan_id" feature available in pktgen, and send
>> in the lower real device?

Could we automatically convert the VLAN device's joining effort to
real device + vlan_id settings in pkt_dev.
or could we issue a warning or a suggestion
when user try to add a VLAN device to the pktgen thread?

>>
>>
>>
>>> Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
>>> ---
>>>   net/core/pktgen.c |   20 +++++++++++++++++---
>>>   1 files changed, 17 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
>>> index 0304f98..ced07fc 100644
>>> --- a/net/core/pktgen.c
>>> +++ b/net/core/pktgen.c
>>> @@ -3297,6 +3297,7 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>>>         netdev_tx_t (*xmit)(struct sk_buff *, struct net_device *)
>>>                 = odev->netdev_ops->ndo_start_xmit;
>>>         struct netdev_queue *txq;
>>> +       struct sk_buff *nskb = NULL;
>>>         u16 queue_map;
>>>         int ret;
>>>
>>> @@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev
>>> *pkt_dev)
>>>                 pkt_dev->last_ok = 0;
>>>                 goto unlock;
>>>         }
>>> -       atomic_inc(&(pkt_dev->skb->users));
>>> -       ret = (*xmit)(pkt_dev->skb, odev);
>>> +
>>> +       if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
>>> +               nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
>>> +               ret = -ENOMEM;
>>> +               if (nskb)
>>> +                       ret = (*xmit)(nskb, odev);
>>> +               else
>>> +                       nskb = ERR_PTR(ret);
>>> +       } else {
>>> +               atomic_inc(&(pkt_dev->skb->users));
>>> +               ret = (*xmit)(pkt_dev->skb, odev);
>>> +       }
>>>
>>>         switch (ret) {
>>>         case NETDEV_TX_OK:
>>> @@ -3372,7 +3383,10 @@ static void pktgen_xmit(struct pktgen_dev
>>> *pkt_dev)
>>>         case NETDEV_TX_LOCKED:
>>>         case NETDEV_TX_BUSY:
>>>                 /* Retry it next time */
>>> -               atomic_dec(&(pkt_dev->skb->users));
>>> +               if (nskb && !IS_ERR(nskb))
>>> +                       kfree_skb(nskb);
>>> +               else
>>> +                       atomic_dec(&(pkt_dev->skb->users));
>>>                 pkt_dev->last_ok = 0;
>>>         }
>>>   unlock:
>>
>>
>>
>>
>

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Sergei Shtylyov]

Hello.

On 05/02/2014 11:18 AM, Zhouyi Zhou wrote:

> As http://www.spinics.net/lists/netdev/msg165015.html
> pktgen generates shared packet through vlan interface will cause
> oops because of duplicate entering tc queue.

> Try to solve this problem by means of packet clone instead of sharing.

> Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
> ---
>   net/core/pktgen.c |   20 +++++++++++++++++---
>   1 files changed, 17 insertions(+), 3 deletions(-)

> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
> index 0304f98..ced07fc 100644
> --- a/net/core/pktgen.c
> +++ b/net/core/pktgen.c
[...]
> @@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
>   		pkt_dev->last_ok = 0;
>   		goto unlock;
>   	}
> -	atomic_inc(&(pkt_dev->skb->users));
> -	ret = (*xmit)(pkt_dev->skb, odev);
> +
> +	if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
> +		nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
> +		ret = -ENOMEM;
> +		if (nskb)
> +			ret = (*xmit)(nskb, odev);

    You can just do:

			ret = xmit(nskb, odev);

> +		else
> +			nskb = ERR_PTR(ret);
> +	} else {
> +		atomic_inc(&(pkt_dev->skb->users));
> +		ret = (*xmit)(pkt_dev->skb, odev);

    Same here.

WBR, Sergei

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Zhouyi Zhou]

Thank Sergei for reviewing.

I think
On Sat, May 3, 2014 at 12:18 AM, Sergei Shtylyov
<sergei.shtylyov@cogentembedded.com> wrote:

>> +
>> +       if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
>> +               nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
>> +               ret = -ENOMEM;
>> +               if (nskb)
>> +                       ret = (*xmit)(nskb, odev);
if (is_vlan_dev(odev) && pkt_dev->clone_skb) {
        nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
        ret = -ENOMEM;
        if (nskb)
             ret = (*xmit)(nskb, odev);
}
>
and

case NETDEV_TX_LOCKED:
case NETDEV_TX_BUSY:
     /* Retry it next time */
     atomic_dec(&(pkt_dev->skb->users));
     if (is_vlan_dev(odev) && pkt_dev->clone_skb && nskb)
                 kfree_skb(nskb);
     else
                 atomic_dec(&(pkt_dev->skb->users));

is better, because is_vlan_dev(odev) is probably in  read most cache line.

Zhouyi

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Jesper Dangaard Brouer]

On Fri, 02 May 2014 07:00:00 -0700
John Fastabend <john.r.fastabend@intel.com> wrote:

> On 5/2/2014 6:19 AM, Jesper Dangaard Brouer wrote:
> > On Fri,  2 May 2014 15:18:12 +0800
> > Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
> >
> >> As http://www.spinics.net/lists/netdev/msg165015.html
> >> pktgen generates shared packet through vlan interface will cause
> >> oops because of duplicate entering tc queue.
> >>
> >> Try to solve this problem by means of packet clone instead of sharing.
> >
> > I really don't like adding this stuff to the fast path of pktgen.
> >
> > Why would you use pktgen on a VLAN?
> 
> Its a good way to test qdiscs. When you run pktgen over the VLAN
> you exercise the lower devices qdisc.

I do (personally) need a faster way/tool to exercise the qdisc path.
I'm currently using trafgen, but it is not fast enough for my 10G
testing.

Perhaps we could add a pktgen option, that explicitly enable
transmitting on qdisc path.  And when adding a VLAN device, auto enable
that mode?


> Although I never submitted a patch like this because I figured it
> was a corner case and we would want to keep the hotpath clean.

I'm worried about the overhead of skb_clone() on for every pktgen
pkt_dev->clone_skb counter... does it "invalidate" using the pktgen
CLONE_SKB counter (kind-of confusing pktgen uses "clone_skb" for its
counter).


> > Why don't you use the "vlan_id" feature available in pktgen, and send
> > in the lower real device?

Guess, we can use it for testing/stressing the qdisc code path.


> >> Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
> >> ---
> >>   net/core/pktgen.c |   20 +++++++++++++++++---
> >>   1 files changed, 17 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
[...]
> >> @@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
> >>   		pkt_dev->last_ok = 0;
> >>   		goto unlock;
> >>   	}
> >> -	atomic_inc(&(pkt_dev->skb->users));
> >> -	ret = (*xmit)(pkt_dev->skb, odev);
> >> +
> >> +	if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
> >> +		nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
> >> +		ret = -ENOMEM;
> >> +		if (nskb)
> >> +			ret = (*xmit)(nskb, odev);
> >> +		else
> >> +			nskb = ERR_PTR(ret);
> >> +	} else {
> >> +		atomic_inc(&(pkt_dev->skb->users));
> >> +		ret = (*xmit)(pkt_dev->skb, odev);
> >> +	}


-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Jiri Pirko]

Fri, May 02, 2014 at 09:18:12AM CEST, zhouzhouyi@gmail.com wrote:
>As http://www.spinics.net/lists/netdev/msg165015.html
>pktgen generates shared packet through vlan interface will cause
>oops because of duplicate entering tc queue.
>
>Try to solve this problem by means of packet clone instead of sharing.

Isn't this prohibited by:
dev->priv_flags         &= ~(IFF_TX_SKB_SHARING);
?

>
>Signed-off-by: Zhouyi Zhou <yizhouzhou@ict.ac.cn>
>---
> net/core/pktgen.c |   20 +++++++++++++++++---
> 1 files changed, 17 insertions(+), 3 deletions(-)
>
>diff --git a/net/core/pktgen.c b/net/core/pktgen.c
>index 0304f98..ced07fc 100644
>--- a/net/core/pktgen.c
>+++ b/net/core/pktgen.c
>@@ -3297,6 +3297,7 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
> 	netdev_tx_t (*xmit)(struct sk_buff *, struct net_device *)
> 		= odev->netdev_ops->ndo_start_xmit;
> 	struct netdev_queue *txq;
>+	struct sk_buff *nskb = NULL;
> 	u16 queue_map;
> 	int ret;
> 
>@@ -3347,8 +3348,18 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
> 		pkt_dev->last_ok = 0;
> 		goto unlock;
> 	}
>-	atomic_inc(&(pkt_dev->skb->users));
>-	ret = (*xmit)(pkt_dev->skb, odev);
>+
>+	if (pkt_dev->clone_skb && is_vlan_dev(odev)) {
>+		nskb = skb_clone(pkt_dev->skb, GFP_ATOMIC);
>+		ret = -ENOMEM;
>+		if (nskb)
>+			ret = (*xmit)(nskb, odev);
>+		else
>+			nskb = ERR_PTR(ret);
>+	} else {
>+		atomic_inc(&(pkt_dev->skb->users));
>+		ret = (*xmit)(pkt_dev->skb, odev);
>+	}
> 
> 	switch (ret) {
> 	case NETDEV_TX_OK:
>@@ -3372,7 +3383,10 @@ static void pktgen_xmit(struct pktgen_dev *pkt_dev)
> 	case NETDEV_TX_LOCKED:
> 	case NETDEV_TX_BUSY:
> 		/* Retry it next time */
>-		atomic_dec(&(pkt_dev->skb->users));
>+		if (nskb && !IS_ERR(nskb))
>+			kfree_skb(nskb);
>+		else
>+			atomic_dec(&(pkt_dev->skb->users));
> 		pkt_dev->last_ok = 0;
> 	}
> unlock:
>-- 
>1.7.1
>
>--
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Ben Greear]

On 05/05/2014 08:12 AM, Jesper Dangaard Brouer wrote:
> On Fri, 02 May 2014 07:00:00 -0700
> John Fastabend <john.r.fastabend@intel.com> wrote:
> 
>> On 5/2/2014 6:19 AM, Jesper Dangaard Brouer wrote:
>>> On Fri,  2 May 2014 15:18:12 +0800
>>> Zhouyi Zhou <zhouzhouyi@gmail.com> wrote:
>>>
>>>> As http://www.spinics.net/lists/netdev/msg165015.html
>>>> pktgen generates shared packet through vlan interface will cause
>>>> oops because of duplicate entering tc queue.
>>>>
>>>> Try to solve this problem by means of packet clone instead of sharing.
>>>
>>> I really don't like adding this stuff to the fast path of pktgen.
>>>
>>> Why would you use pktgen on a VLAN?
>>
>> Its a good way to test qdiscs. When you run pktgen over the VLAN
>> you exercise the lower devices qdisc.
> 
> I do (personally) need a faster way/tool to exercise the qdisc path.
> I'm currently using trafgen, but it is not fast enough for my 10G
> testing.
> 
> Perhaps we could add a pktgen option, that explicitly enable
> transmitting on qdisc path.  And when adding a VLAN device, auto enable
> that mode?

You could just force pktgen to not support multi-skb on vlan interfaces?

I thought we went through this a year or two ago and came up with
something like a 'pktgen-challenged' network interface flag?

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [PATCH] netdev: pktgen xmit packet through vlan interface

[Zhouyi Zhou]

> You could just force pktgen to not support multi-skb on vlan interfaces?
>
> I thought we went through this a year or two ago and came up with
> something like a 'pktgen-challenged' network interface flag?

Ah yes,  IFF_TX_SKB_SHARING does the job, very sorry for missing that,
 as matter of fact, I have
tailed pktgen model for my personal use.

By the way, would skb_clone save the CPU cycles for
memset(skb_put(skb, datalen), 0, datalen) thing ? especially for
Jesper's qdisc test scenery.





>
> Thanks,
> Ben
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>