Re: [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue

[Hangbin Liu <liuhangbin@gmail.com>]

Hi Maciej,
On Wed, Jan 20, 2021 at 11:42:38PM +0100, Maciej Fijalkowski wrote:
> > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog,
> > +				struct xdp_frame **frames, int n,
> > +				struct net_device *dev)
> > +{
> > +	struct xdp_txq_info txq = { .dev = dev };
> > +	struct xdp_buff xdp;
> > +	int i, nframes = 0;
> > +
> > +	for (i = 0; i < n; i++) {
> > +		struct xdp_frame *xdpf = frames[i];
> > +		u32 act;
> > +		int err;
> > +
> > +		xdp_convert_frame_to_buff(xdpf, &xdp);
> > +		xdp.txq = &txq;
> > +
> > +		act = bpf_prog_run_xdp(xdp_prog, &xdp);
> > +		switch (act) {
> > +		case XDP_PASS:
> > +			err = xdp_update_frame_from_buff(&xdp, xdpf);
> 
> Bump on John's question.

Hi Jesper, would you please help answer John's question?
> >  
> > -	sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags);
> > +	/* Init sent to cnt in case there is no xdp_prog */
> > +	sent = cnt;
> > +	if (bq->xdp_prog) {
> > +		sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
> > +		if (!sent)
> > +			goto out;
> 
> Sorry, but 'sent' is a bit confusing to me, actual sending happens below
> via ndo_xdp_xmit, right? This hook will not actually send frames.
> Can we do a subtle change to have it in separate variable 'to_send' ?

Makes sense to me.
> 
> Although I'm a huge goto advocate, I feel like this particular usage could
> be simplified. Not sure why we had that in first place.
> 
> I gave a shot at rewriting/refactoring whole bq_xmit_all and I feel like
> it's more readable. I introduced 'to_send' variable and got rid of 'error'
> label.
> 
> Thoughts?
> 
> I might have missed something, though.
> 
> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
> {
> 	struct net_device *dev = bq->dev;
> 	unsigned int cnt = bq->count;
> 	int drops = 0, err = 0;
> 	int to_send = 0;

The to_send also need to init to cnt.

> 	int sent = cnt;
> 	int i;
> 
> 	if (unlikely(!cnt))
> 		return;
> 
> 	for (i = 0; i < cnt; i++) {
> 		struct xdp_frame *xdpf = bq->q[i];
> 
> 		prefetch(xdpf);
> 	}
> 
> 	if (bq->xdp_prog) {
> 		to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
> 		if (!to_send) {
> 			sent = 0;
> 			goto out;
> 		}
> 	}
> 
> 	drops = cnt - to_send;

This line could move in to the xdp_prog brackets to save time when no xdp_prog.

	if (bq->xdp_prog) {
		to_send = ...
		if (!to_send) {
			...
		}
		drops = cnt - to_send;
	}

> 	sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags);

If we don't have xdp_prog, the to_send should be cnt.

> 	if (sent < 0) {
> 		err = sent;
> 		sent = 0;
> 
> 		/* If ndo_xdp_xmit fails with an errno, no frames have been
> 		 * xmit'ed and it's our responsibility to them free all.
> 		 */
> 		for (i = 0; i < cnt - drops; i++) {
> 			struct xdp_frame *xdpf = bq->q[i];
> 
> 			xdp_return_frame_rx_napi(xdpf);
> 		}
> 	}
> out:
> 	drops = cnt - sent;
> 	bq->count = 0;
> 
> 	trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
> 	bq->dev_rx = NULL;
> 	bq->xdp_prog = NULL;
> 	__list_del_clearprev(&bq->flush_node);
> 
> 	return;
> }

Thanks for your code, looks much clear now.

Hangbin