Re: [ovs-dev] [PATCH net-next v4 05/10] net: openvswitch: optimize flow-mask looking up

[William Tu <u9012063@gmail.com>]

> > Hi Tonghao,
> >
> > Does this improve performance? After all, the original code simply
> > check whether the mask is NULL, then goto next mask.
> I tested the performance, but I disable the mask cache, and use the
> dpdk-pktgen to generate packets:
> The test ovs flow:
> ovs-dpctl add-dp system@ovs-system
> ovs-dpctl add-if system@ovs-system eth6
> ovs-dpctl add-if system@ovs-system eth7
>
> for m in $(seq 1 100 | xargs printf '%.2x\n'); do
>        ovs-dpctl add-flow ovs-system
> "in_port(1),eth(dst=00:$m:00:00:00:00/ff:ff:ff:ff:ff:$m),eth_type(0x0800),ipv4(frag=no)"
> 2
> done
>
> ovs-dpctl add-flow ovs-system
> "in_port(1),eth(dst=98:03:9b:6e:4a:f5/ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no)"
> 2
> ovs-dpctl add-flow ovs-system
> "in_port(2),eth(dst=98:03:9b:6e:4a:f4/ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no)"
> 1
>
> for m in $(seq 101 160 | xargs printf '%.2x\n'); do
>         ovs-dpctl add-flow ovs-system
> "in_port(1),eth(dst=00:$m:00:00:00:00/ff:ff:ff:ff:ff:$m),eth_type(0x0800),ipv4(frag=no)"
> 2
> done
>
> for m in $(seq 1 100 | xargs printf '%.2x\n'); do
>          ovs-dpctl del-flow ovs-system
> "in_port(1),eth(dst=00:$m:00:00:00:00/ff:ff:ff:ff:ff:$m),eth_type(0x0800),ipv4(frag=no)"
> done
>
> Without this patch: 982481pps (64B)
> With this patch: 1112495 pps (64B), about 13% improve
>

Hi Tonghao,

Thanks for doing the measurement.
Based on the result (skipping 100 NULL mask lookup with 13% improvement),
and with additional overhead of mask cache being invalidate while
refilling these 100
gap, I'd argue that this patch is not necessary. But let's wait for
others comments.

Regards,
William

> > However, with your patch,  isn't this invalidated the mask cache entry which
> > point to the "M" you swap to the front? See my commands inline.
> >
> > >
> > > Signed-off-by: Tonghao Zhang <xiangxia.m.yue@gmail.com>
> > > Tested-by: Greg Rose <gvrose8192@gmail.com>
> > > ---

<snip>

> > >  static struct table_instance *table_instance_expand(struct table_instance *ti,
> > > @@ -704,21 +704,33 @@ static struct table_instance *table_instance_expand(struct table_instance *ti,
> > >         return table_instance_rehash(ti, ti->n_buckets * 2, ufid);
> > >  }
> > >
> > > -static void tbl_mask_array_delete_mask(struct mask_array *ma,
> > > -                                      struct sw_flow_mask *mask)
> > > +static void tbl_mask_array_del_mask(struct flow_table *tbl,
> > > +                                   struct sw_flow_mask *mask)
> > >  {
> > > -       int i;
> > > +       struct mask_array *ma = ovsl_dereference(tbl->mask_array);
> > > +       int i, ma_count = READ_ONCE(ma->count);
> > >
> > >         /* Remove the deleted mask pointers from the array */
> > > -       for (i = 0; i < ma->max; i++) {
> > > -               if (mask == ovsl_dereference(ma->masks[i])) {
> > > -                       RCU_INIT_POINTER(ma->masks[i], NULL);
> > > -                       ma->count--;
> > > -                       kfree_rcu(mask, rcu);
> > > -                       return;
> > > -               }
> > > +       for (i = 0; i < ma_count; i++) {
> > > +               if (mask == ovsl_dereference(ma->masks[i]))
> > > +                       goto found;
> > >         }
> > > +
> > >         BUG();
> > > +       return;
> > > +
> > > +found:
> > > +       WRITE_ONCE(ma->count, ma_count -1);
> > > +
> > > +       rcu_assign_pointer(ma->masks[i], ma->masks[ma_count -1]);
> > > +       RCU_INIT_POINTER(ma->masks[ma_count -1], NULL);
> >
> > So when you swap the ma->masks[ma_count -1], the mask cache entry
> > who's 'mask_index == ma_count' become all invalid?
> Yes, a little tricky.
> > Regards,
> > William
> >