Re: [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue

[Maciej Fijalkowski <maciej.fijalkowski@intel.com>]

On Thu, Jan 21, 2021 at 11:54:24AM +0800, Hangbin Liu wrote:
> Hi Maciej,
> On Wed, Jan 20, 2021 at 11:42:38PM +0100, Maciej Fijalkowski wrote:
> > > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog,
> > > +				struct xdp_frame **frames, int n,
> > > +				struct net_device *dev)
> > > +{
> > > +	struct xdp_txq_info txq = { .dev = dev };
> > > +	struct xdp_buff xdp;
> > > +	int i, nframes = 0;
> > > +
> > > +	for (i = 0; i < n; i++) {
> > > +		struct xdp_frame *xdpf = frames[i];
> > > +		u32 act;
> > > +		int err;
> > > +
> > > +		xdp_convert_frame_to_buff(xdpf, &xdp);
> > > +		xdp.txq = &txq;
> > > +
> > > +		act = bpf_prog_run_xdp(xdp_prog, &xdp);
> > > +		switch (act) {
> > > +		case XDP_PASS:
> > > +			err = xdp_update_frame_from_buff(&xdp, xdpf);
> > 
> > Bump on John's question.
> 
> Hi Jesper, would you please help answer John's question?
> > >  
> > > -	sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags);
> > > +	/* Init sent to cnt in case there is no xdp_prog */
> > > +	sent = cnt;
> > > +	if (bq->xdp_prog) {
> > > +		sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
> > > +		if (!sent)
> > > +			goto out;
> > 
> > Sorry, but 'sent' is a bit confusing to me, actual sending happens below
> > via ndo_xdp_xmit, right? This hook will not actually send frames.
> > Can we do a subtle change to have it in separate variable 'to_send' ?
> 
> Makes sense to me.
> > 
> > Although I'm a huge goto advocate, I feel like this particular usage could
> > be simplified. Not sure why we had that in first place.
> > 
> > I gave a shot at rewriting/refactoring whole bq_xmit_all and I feel like
> > it's more readable. I introduced 'to_send' variable and got rid of 'error'
> > label.
> > 
> > Thoughts?
> > 
> > I might have missed something, though.
> > 
> > static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
> > {
> > 	struct net_device *dev = bq->dev;
> > 	unsigned int cnt = bq->count;
> > 	int drops = 0, err = 0;
> > 	int to_send = 0;
> 
> The to_send also need to init to cnt.

So I missed something indeed :P you're correct

> 
> > 	int sent = cnt;
> > 	int i;
> > 
> > 	if (unlikely(!cnt))
> > 		return;
> > 
> > 	for (i = 0; i < cnt; i++) {
> > 		struct xdp_frame *xdpf = bq->q[i];
> > 
> > 		prefetch(xdpf);
> > 	}
> > 
> > 	if (bq->xdp_prog) {
> > 		to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
> > 		if (!to_send) {
> > 			sent = 0;
> > 			goto out;
> > 		}
> > 	}
> > 
> > 	drops = cnt - to_send;
> 
> This line could move in to the xdp_prog brackets to save time when no xdp_prog.

Hmm, looks like we can do it.
For scenario where there was no bq->xdp_prog and failure of ndo_xdp_xmit,
we didn't alter the count of frames to be sent, so we would basically free
all of the frames (as drops is 0, cnt = bq->count). After that we
recalculate drops and correct value will be reported in tracepoint.

(needed to explain it to myself)

> 
> 	if (bq->xdp_prog) {
> 		to_send = ...
> 		if (!to_send) {
> 			...
> 		}
> 		drops = cnt - to_send;
> 	}
> 
> > 	sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags);
> 
> If we don't have xdp_prog, the to_send should be cnt.

Yes, we should init to_send to cnt as you're suggesting above.

> 
> > 	if (sent < 0) {
> > 		err = sent;
> > 		sent = 0;
> > 
> > 		/* If ndo_xdp_xmit fails with an errno, no frames have been
> > 		 * xmit'ed and it's our responsibility to them free all.
> > 		 */
> > 		for (i = 0; i < cnt - drops; i++) {
> > 			struct xdp_frame *xdpf = bq->q[i];
> > 
> > 			xdp_return_frame_rx_napi(xdpf);
> > 		}
> > 	}
> > out:
> > 	drops = cnt - sent;
> > 	bq->count = 0;
> > 
> > 	trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
> > 	bq->dev_rx = NULL;
> > 	bq->xdp_prog = NULL;
> > 	__list_del_clearprev(&bq->flush_node);
> > 
> > 	return;
> > }
> 
> Thanks for your code, looks much clear now.

Good to hear! I agree on your points as well.

> 
> Hangbin