Re: [PATCH v4 1/7] service cores: header and implementation

["Van Haaren, Harry" <harry.van.haaren@intel.com>]

<lots of snips to make responses consumable!>

> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org; thomas@monjalon.net; Wiles, Keith <keith.wiles@intel.com>; Richardson,
> Bruce <bruce.richardson@intel.com>
> Subject: Re: [PATCH v4 1/7] service cores: header and implementation
> 
<snip>
> 
> Remove above info from the git commit.

Done


> Fix the below mentioned documentation warning.
> 
> +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:338:
> warning: argument 'enabled' of command @param is not found in the
> argument list of rte_service_set_stats_enable(int enable)
> +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:346:
> warning: The following parameters of rte_service_set_stats_enable(int
> enable) are not documented:
> +  parameter 'enable'
> +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:349:
> warning: argument 'The' of command @param is not found in the argument
> list of rte_service_lcore_list(uint32_t array[], uint32_t n)
> +/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:367:
> warning: The following parameters of rte_service_lcore_list(uint32_t
> array[], uint32_t n) are not documented:
> +  parameter 'n'

Done


> command to reproduce:
> ./devtools/test-build.sh -j8 x86_64-native-linuxapp-gcc+shared x86_64-native-linuxapp-
> gcc+debug

Thanks - noted.


> > +
> > +/* the internal values of a service core */
> > +struct core_state {
> 
> Change to lcore_state.

Done.


> > +
> > +void rte_service_set_stats_enable(int enabled)
> 
> IMO, It should be per service  i.e
> rte_service_set_stats_enable(const struct rte_service_spec *spec, int enable)

Improved service library to handle statistics collection on a per service basis.


> > +			/* check if this is the only core mapped, else use
> > +			 * atomic to serialize cores mapped to this service
> > +			 */
> > +			uint32_t *lock = (uint32_t *)&s->execute_lock;
> > +			if ((s->spec.capabilities & RTE_SERVICE_CAP_MT_SAFE) ||
> > +					(s->num_mapped_cores == 1 ||
> > +					rte_atomic32_cmpset(lock, 0, 1))) {
> > +				void *userdata = s->spec.callback_userdata;
> > +
> > +				if (cs->collect_statistics) {
> > +					uint64_t start = rte_rdtsc();
> > +					s->spec.callback(userdata);
> > +					uint64_t end = rte_rdtsc();
> > +					s->cycles_spent += end - start;
> > +					cs->calls_per_service[i]++;
> > +					s->calls++;
> > +				} else
> > +					s->spec.callback(userdata);
> > +
> > +				if ((s->spec.capabilities &
> > +						RTE_SERVICE_CAP_MT_SAFE) == 0 &&
> > +						s->num_mapped_cores > 1)
> 
> How about computing the non rte_atomic32_cmpset() mode value first and
> using in both place i.e here and in the top "if" loop
> 
> 	const int need_cmpset = (s->spec.capabilities & RTE_SERVICE_CAP_MT_SAFE)...
> 	if (need_cmpset || rte_atomic32_cmpset(lock, 0, 1))
> 	..
> 	if (need_cmpset)
> 		rte_atomic32_clear()..


Yes good idea. Indeed I wasn't happy with that, this is a good fix.
The checks to detect if we need_cmpset are a little complex, but it's the only solution I see.

I've added another unit test to verify both MT safe and MT unsafe callback operations.


> > +int32_t
> > +rte_service_set_default_mapping(void)
> > +{
> > +	/* create a default mapping from cores to services, then start the
> > +	 * services to make them transparent to unaware applications.
> > +	 */
> > +	uint32_t i;
> > +	int ret;
> > +	uint32_t count = rte_service_get_count();
> > +
> > +	int32_t lcore_iter = 0;
> > +	uint32_t ids[RTE_MAX_LCORE];
> > +	int32_t lcore_count = rte_service_lcore_list(ids, RTE_MAX_LCORE);
> > +
> > +	for (i = 0; i < count; i++) {
> > +		struct rte_service_spec *s = rte_service_get_by_id(i);
> > +		if (!s)
> > +			return -EINVAL;
> > +
> > +		/* if no lcores available as services cores, don't setup map.
> > +		 * This means app logic must add cores, and setup mappings
> > +		 */
> > +		if (lcore_count > 0) {
> 
> 
> > +			/* do 1:1 core mapping here, with each service getting
> > +			 * assigned a single core by default. Adding multiple
> > +			 * services should multiplex to a single core, or 1:1
> > +			 * if services == cores
> > +			 */
> > +			ret = rte_service_enable_on_lcore(s, ids[lcore_iter]);
> > +			if (ret)
> > +				return -ENODEV;
> > +		}
> > +
> > +		lcore_iter++;
> > +		if (lcore_iter >= lcore_count)
> > +			lcore_iter = 0;
> > +
> > +		ret = rte_service_start(s);
> 
> IMO, we don't need to start the service if lcore_count == 0. How about
> moving the "if (lcore_count > 0)" check on top of for the loop and exist
> from the function if lcore_count == 0.

Good point, added   if() check for lcore_count at start of function, and return if no service cores are available.


> > +
> > +int32_t
> > +rte_service_lcore_add(uint32_t lcore)
> > +{
> > +	if (lcore >= RTE_MAX_LCORE)
> > +		return -EINVAL;
> > +	if (cores_state[lcore].is_service_core)
> > +		return -EALREADY;
> > +
> > +	set_lcore_state(lcore, ROLE_SERVICE);
> > +
> > +	/* ensure that after adding a core the mask and state are defaults */
> > +	cores_state[lcore].service_mask = 0;
> > +	cores_state[lcore].runstate = RUNSTATE_STOPPED;
> 
> If worker core can call rte_service_lcore_add() then add rte_smp_wmb()
> here. Applies to rte_service_lcore_del() as well.

Added barriers to both add() and del().