[RFC] eal: add cgroup-aware resource self discovery

[RFC] eal: add cgroup-aware resource self discovery

[Jianfeng Tan]

Current issue: DPDK is not that friendly to container environment usage.
It's because that it pre-alloc resource like cores and hugepages from cmd
line options. So for a DPDK application, it's necessary to check how much
resource is allocated to a container and then use that as an reference.

To address that, this patch introduces two APIs:
   a. rte_eal_res_self_discovery, to query how much resource can be used.
   b. rte_eal_res_self_discovery_apply, to apply self-discovered resource
      into DPDK.

Currently only Linux CGroup is added, similarly, we can add BSD jail as
well in the future. And even in Linux, there could be other way to query
and apply resources, like through a centralized daemon.

Known issue: current way to read individual attributes of cgroups directly
instead of via systemd's API is not a long-term solution. Please refer to
http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
for more information.

Test example:
    a. cgcreate -g cpuset,hugetlb:/test-subgroup
    b. cgset -r cpuset.cpus=2-3 test-subgroup
    c. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
    d. cgexec -g cpuset,hugetlb:test-subgroup \
	    ./examples/l2fwd/build/l2fwd --self-discovery=cgroup -n 4 -- -p 3

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 lib/librte_eal/common/eal_common_options.c      |  39 ++++
 lib/librte_eal/common/eal_internal_cfg.h        |   1 +
 lib/librte_eal/common/eal_options.h             |   2 +
 lib/librte_eal/common/include/rte_eal.h         |  34 +++
 lib/librte_eal/linuxapp/eal/Makefile            |   1 +
 lib/librte_eal/linuxapp/eal/eal_cgroup.c        | 294 ++++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_hugepage_info.c |   5 +
 7 files changed, 376 insertions(+)
 create mode 100644 lib/librte_eal/linuxapp/eal/eal_cgroup.c

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 29942ea..7235473 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+	{OPT_SELF_DISCOVERY,    1, NULL, OPT_SELF_DISCOVERY_NUM   },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -128,6 +129,7 @@ eal_reset_internal_config(struct internal_config *internal_cfg)
 	internal_cfg->force_nchannel = 0;
 	internal_cfg->hugefile_prefix = HUGEFILE_PREFIX_DEFAULT;
 	internal_cfg->hugepage_dir = NULL;
+	internal_cfg->self_discovery = NULL;
 	internal_cfg->force_sockets = 0;
 	/* zero out the NUMA config */
 	for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
@@ -755,6 +757,24 @@ eal_parse_proc_type(const char *arg)
 }
 
 int
+__attribute__((weak))
+rte_eal_res_self_discovery(const char *type __rte_unused,
+			   char **p_corelist __rte_unused,
+			   uint64_t *p_memory __rte_unused)
+{
+	return -1;
+}
+
+int
+__attribute__((weak))
+rte_eal_res_self_discovery_apply(const char *type __rte_unused,
+				 int enable_core __rte_unused,
+				 int enable_mem __rte_unused)
+{
+	return -1;
+}
+
+int
 eal_parse_common_option(int opt, const char *optarg,
 			struct internal_config *conf)
 {
@@ -897,6 +917,25 @@ eal_parse_common_option(int opt, const char *optarg,
 		}
 		break;
 
+	case OPT_SELF_DISCOVERY_NUM: {
+		char *corelist;
+
+		if (rte_eal_res_self_discovery(optarg, &corelist, NULL) < 0) {
+			RTE_LOG(ERR, EAL, "invalid parameter for --"
+				OPT_SELF_DISCOVERY "\n");
+			return -1;
+		}
+
+		if (eal_parse_corelist(corelist) < 0) {
+			RTE_LOG(ERR, EAL, "invalid core list\n");
+			return -1;
+		}
+		/* Save it here for memory limit */
+		internal_config.self_discovery = strdup(optarg);
+
+		break;
+	}
+
 	/* don't know what to do, leave this to caller */
 	default:
 		return 1;
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..f3c8e31 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -83,6 +83,7 @@ struct internal_config {
 	volatile enum rte_intr_mode vfio_intr_mode;
 	const char *hugefile_prefix;      /**< the base filename of hugetlbfs files */
 	const char *hugepage_dir;         /**< specific hugetlbfs directory to use */
+	const char *self_discovery;       /**< specific type of self_discovery */
 
 	unsigned num_hugepage_sizes;      /**< how many sizes on this system */
 	struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES];
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..a499d73 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#define OPT_SELF_DISCOVERY    "self-discovery"
+	OPT_SELF_DISCOVERY_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index d2816a8..ff81484 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -220,6 +220,40 @@ int rte_eal_has_hugepages(void);
 int rte_sys_gettid(void);
 
 /**
+ * An API to query resource self discovery.
+ *
+ * @type
+ *   Type of self resource discovery.
+ * @p_corelist
+ *   If succeed, fill core list which can be used. Caller to free.
+ * @p_memory
+ *   If succeed, fill how many (bytes) memory can be used.
+ *
+ * @return
+ *   - (-1), if failed.
+ *   - 0, if succeed.
+ */
+int rte_eal_res_self_discovery(const char *type,
+			       char **p_corelist, uint64_t *p_memory);
+/**
+ * An API to apply resource through self discovery.
+ *
+ * @type
+ *   Type of self resource discovery.
+ * @enable_core
+ *   If succeed, apply core resource.
+ * @p_memory
+ *   If succeed, apply memory resource.
+ *
+ * @return
+ *   - (-1), if failed.
+ *   - 0, if succeed.
+ */
+int rte_eal_res_self_discovery_apply(const char *type,
+				     int enable_core, int enable_mem);
+
+
+/**
  * Get system unique thread id.
  *
  * @return
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 26eced5..834ae2f 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -87,6 +87,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_devargs.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_dev.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_options.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_thread.c
+SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_cgroup.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += rte_malloc.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += malloc_heap.c
diff --git a/lib/librte_eal/linuxapp/eal/eal_cgroup.c b/lib/librte_eal/linuxapp/eal/eal_cgroup.c
new file mode 100644
index 0000000..d6a04ee
--- /dev/null
+++ b/lib/librte_eal/linuxapp/eal/eal_cgroup.c
@@ -0,0 +1,294 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2010-2016 Intel Corporation. All rights reserved.
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <unistd.h>
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+#include <sys/file.h>
+#include <sys/types.h>
+#include <dirent.h>
+#include <mntent.h>
+#include <inttypes.h>
+
+#include <rte_log.h>
+#include <rte_eal.h>
+#include <rte_common.h>
+
+#include "eal_internal_cfg.h"
+
+static int pid;
+
+static char *
+find_controller_dfs(const char *dir_path)
+{
+	FILE *f;
+	char *line;
+	char *ret;
+	size_t len;
+	ssize_t read;
+	DIR *dir;
+	struct dirent *ent;
+	char filepath[PATH_MAX];
+
+	// 1. check if this process belongs to this cgroup
+	snprintf(filepath, sizeof(filepath)-1, "%s/tasks", dir_path);
+	f = fopen(filepath, "r");
+	if (f == NULL)
+		return NULL;
+	len = 0;
+	line = NULL;
+	while ((read = getline(&line, &len, f)) != -1) {
+		int _pid = atoi(line);
+		free(line);
+		if (_pid == pid)
+			break;
+		len = 0;
+		line = NULL;
+	}
+	fclose(f);
+	if (read != -1)
+		return strdup(dir_path);
+
+	// 2. check its child cgroup
+	if (!(dir = opendir(dir_path)))
+		return NULL;
+
+	ret = NULL;
+	while ((ent = readdir(dir)) != NULL) {
+		if (ent->d_type != DT_DIR)
+			continue;
+		if (strcmp(ent->d_name, ".") == 0 ||
+		    strcmp(ent->d_name, "..") == 0)
+			continue;
+
+		snprintf(filepath, sizeof(filepath)-1, "%s/%s",
+			 dir_path, ent->d_name);
+
+		ret = find_controller_dfs(filepath);
+		if (ret != NULL)
+			break;
+	}
+
+	closedir(dir);
+	return ret;
+}
+
+static char *
+find_controller(const char *controller)
+{
+	FILE *f;
+	char *path;
+	struct mntent *ent;
+
+	static const char *proc_mounts = "/proc/mounts";
+	static const char *fs_type = "cgroup";
+
+	f = setmntent(proc_mounts, "r");
+	if (f == NULL) {
+		RTE_LOG(ERR, EAL, "Cannot open %s\n", proc_mounts);
+		return NULL;
+	}
+
+	while (NULL != (ent = getmntent(f))) {
+		if (strcmp(ent->mnt_type, fs_type) != 0)
+			continue;
+		if (hasmntopt(ent, controller) == NULL)
+			continue;
+		break;
+	}
+
+	if (ent == NULL) {
+		path = NULL;
+		goto end;
+	}
+
+	path = find_controller_dfs(ent->mnt_dir);
+end:
+	endmntent(f);
+	return path;
+}
+
+static inline char *
+get_oneline_from_file(const char *path)
+{
+	FILE *f;
+	char *line = NULL;
+	size_t len = 0;
+
+	if (NULL == (f = fopen(path, "r")))
+		return NULL;
+	if (getline(&line, &len, f) == -1)
+		line = NULL;
+	line[strcspn(line, "\n")] = 0;
+	fclose(f);
+	return line;
+}
+
+static int
+cgroup_cpuset(char **p_corelist, int enable __rte_unused)
+{
+	char filepath[PATH_MAX];
+	char *controller;
+
+       	controller = find_controller("cpuset");
+	if (controller == NULL)
+		return -1;
+
+	snprintf(filepath, sizeof(filepath)-1, "%s/cpuset.cpus", controller);
+	*p_corelist = get_oneline_from_file(filepath);
+	RTE_LOG(INFO, EAL, "cgroup cpuset: %s\n", *p_corelist);
+	return 0;
+
+}
+
+static inline uint64_t
+get_hugetlb_limit(const char *path)
+{
+	uint64_t limit;
+	char *str;
+
+       	str = get_oneline_from_file(path);
+	sscanf(str, "%"PRIu64, &limit);
+	free(str);
+	return limit;
+}
+
+static int
+cgroup_hugetlb(uint64_t *p_memory, int enable)
+{
+	unsigned i;
+	char filepath[PATH_MAX];
+	char *controller;
+	DIR *dir;
+	struct dirent *ent;
+	uint64_t memory = 0;
+	static char prefix[] = "hugetlb";
+	static int prefix_len = sizeof(prefix) - 1;
+	static char suffix[] = "limit_in_bytes";
+
+       	controller = find_controller("hugetlb");
+	if (controller == NULL)
+		return -1;
+
+	if (!(dir = opendir(controller)))
+		return -1;
+
+	while ((ent = readdir(dir)) != NULL) {
+		if (strncmp(ent->d_name, prefix, prefix_len) != 0)
+			continue;
+
+		char *sz_beg = ent->d_name + prefix_len + 1;
+		char *sz_end = strchr(sz_beg, '.');
+
+		if (strcmp(sz_end + 1, suffix) != 0)
+			continue;
+
+		char *tmp = strndup(sz_beg, sz_end - sz_beg);
+		uint64_t pagesize = rte_str_to_size(tmp);
+		free(tmp);
+
+		snprintf(filepath, sizeof(filepath)-1, "%s/%s",
+			 controller, ent->d_name);
+		uint64_t m_limit = get_hugetlb_limit(filepath);
+		memory += m_limit;
+
+		/* Record those information into internal_config if hugepages
+		 * are already initialized.
+		 */
+		if (! enable)
+			continue;
+		for (i = 0; i < internal_config.num_hugepage_sizes; ++i) {
+			struct hugepage_info *hp;
+
+		       	hp = &internal_config.hugepage_info[i];
+			if (hp->hugepage_sz != pagesize)
+				continue;
+
+			if (m_limit < hp->hugepage_sz * hp->num_pages[0])
+				hp->num_pages[0] = m_limit / hp->hugepage_sz;
+		}
+	}
+
+	closedir(dir);
+	*p_memory = memory;
+	RTE_LOG(INFO, EAL, "cgroup hugetlb: %"PRIx64"\n", *p_memory);
+	return 0;
+}
+
+static int
+resource_self_discovery(const char *type, char **p_corelist, int enable_core,
+			uint64_t *p_memory, int enable_mem)
+{
+	if (strcmp(type, "cgroup") != 0) {
+		RTE_LOG(ERR, EAL, "type not supported: %s\n", type);
+		return -1;
+	}
+
+	pid = getpid();
+
+	if (p_corelist != NULL && cgroup_cpuset(p_corelist, enable_core) < 0) {
+		RTE_LOG(ERR, EAL, "Failed when discover resource cpuset\n");
+		return -1;
+	}
+	if (p_memory != NULL && cgroup_hugetlb(p_memory, enable_mem) < 0) {
+		RTE_LOG(ERR, EAL, "Failed when discover resource hugetlb\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+int
+rte_eal_res_self_discovery(const char *type, char **p_corelist,
+			   uint64_t *p_memory)
+{
+	return resource_self_discovery(type, p_corelist, 0, p_memory, 0);
+}
+
+int
+rte_eal_res_self_discovery_apply(const char *type, int enable_core,
+				 int enable_mem)
+{
+	char *corelist, **pc = NULL;
+	uint64_t mem, *pm = NULL;
+	
+	if (enable_core)
+		pc = &corelist;
+	if (enable_mem)
+		pm = &mem;
+
+	return resource_self_discovery(type, pc, enable_core,
+				       pm, enable_mem);
+}
diff --git a/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c b/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
index 18858e2..a6b6548 100644
--- a/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
+++ b/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
@@ -354,6 +354,11 @@ eal_hugepage_info_init(void)
 	qsort(&internal_config.hugepage_info[0], num_sizes,
 	      sizeof(internal_config.hugepage_info[0]), compare_hpi);
 
+	/* Apply cgroup hugetlb limit before we really use hugepages */
+	if (internal_config.self_discovery)
+		rte_eal_res_self_discovery_apply(internal_config.self_discovery,
+						 0, 1);
+
 	/* now we have all info, check we have at least one valid size */
 	for (i = 0; i < num_sizes; i++)
 		if (internal_config.hugepage_info[i].hugedir != NULL &&
-- 
2.1.4

Re: [RFC] eal: add cgroup-aware resource self discovery

[Neil Horman]

On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote:
> Current issue: DPDK is not that friendly to container environment usage.
> It's because that it pre-alloc resource like cores and hugepages from cmd
> line options. So for a DPDK application, it's necessary to check how much
> resource is allocated to a container and then use that as an reference.
> 
> To address that, this patch introduces two APIs:
>    a. rte_eal_res_self_discovery, to query how much resource can be used.
>    b. rte_eal_res_self_discovery_apply, to apply self-discovered resource
>       into DPDK.
> 
> Currently only Linux CGroup is added, similarly, we can add BSD jail as
> well in the future. And even in Linux, there could be other way to query
> and apply resources, like through a centralized daemon.
> 
> Known issue: current way to read individual attributes of cgroups directly
> instead of via systemd's API is not a long-term solution. Please refer to
> http://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
> for more information.
> 
> Test example:
>     a. cgcreate -g cpuset,hugetlb:/test-subgroup
>     b. cgset -r cpuset.cpus=2-3 test-subgroup
>     c. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>     d. cgexec -g cpuset,hugetlb:test-subgroup \
> 	    ./examples/l2fwd/build/l2fwd --self-discovery=cgroup -n 4 -- -p 3
> 
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> ---
>  lib/librte_eal/common/eal_common_options.c      |  39 ++++
>  lib/librte_eal/common/eal_internal_cfg.h        |   1 +
>  lib/librte_eal/common/eal_options.h             |   2 +
>  lib/librte_eal/common/include/rte_eal.h         |  34 +++
>  lib/librte_eal/linuxapp/eal/Makefile            |   1 +
>  lib/librte_eal/linuxapp/eal/eal_cgroup.c        | 294 ++++++++++++++++++++++++
>  lib/librte_eal/linuxapp/eal/eal_hugepage_info.c |   5 +
>  7 files changed, 376 insertions(+)
>  create mode 100644 lib/librte_eal/linuxapp/eal/eal_cgroup.c
> 
> diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
> index 29942ea..7235473 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -95,6 +95,7 @@ eal_long_options[] = {
>  	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
>  	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
>  	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
> +	{OPT_SELF_DISCOVERY,    1, NULL, OPT_SELF_DISCOVERY_NUM   },
>  	{0,                     0, NULL, 0                        }
>  };
>  
> @@ -128,6 +129,7 @@ eal_reset_internal_config(struct internal_config *internal_cfg)
>  	internal_cfg->force_nchannel = 0;
>  	internal_cfg->hugefile_prefix = HUGEFILE_PREFIX_DEFAULT;
>  	internal_cfg->hugepage_dir = NULL;
> +	internal_cfg->self_discovery = NULL;
>  	internal_cfg->force_sockets = 0;
>  	/* zero out the NUMA config */
>  	for (i = 0; i < RTE_MAX_NUMA_NODES; i++)
> @@ -755,6 +757,24 @@ eal_parse_proc_type(const char *arg)
>  }
>  
>  int
> +__attribute__((weak))
> +rte_eal_res_self_discovery(const char *type __rte_unused,
> +			   char **p_corelist __rte_unused,
> +			   uint64_t *p_memory __rte_unused)
> +{
> +	return -1;
> +}
> +
> +int
> +__attribute__((weak))
> +rte_eal_res_self_discovery_apply(const char *type __rte_unused,
> +				 int enable_core __rte_unused,
> +				 int enable_mem __rte_unused)
> +{
> +	return -1;
> +}
> +
> +int
>  eal_parse_common_option(int opt, const char *optarg,
>  			struct internal_config *conf)
>  {
> @@ -897,6 +917,25 @@ eal_parse_common_option(int opt, const char *optarg,
>  		}
>  		break;
>  
> +	case OPT_SELF_DISCOVERY_NUM: {
> +		char *corelist;
> +
> +		if (rte_eal_res_self_discovery(optarg, &corelist, NULL) < 0) {
> +			RTE_LOG(ERR, EAL, "invalid parameter for --"
> +				OPT_SELF_DISCOVERY "\n");
> +			return -1;
> +		}
> +
> +		if (eal_parse_corelist(corelist) < 0) {
> +			RTE_LOG(ERR, EAL, "invalid core list\n");
> +			return -1;
> +		}
> +		/* Save it here for memory limit */
> +		internal_config.self_discovery = strdup(optarg);
> +
> +		break;
> +	}
> +
>  	/* don't know what to do, leave this to caller */
>  	default:
>  		return 1;
> diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
> index 5f1367e..f3c8e31 100644
> --- a/lib/librte_eal/common/eal_internal_cfg.h
> +++ b/lib/librte_eal/common/eal_internal_cfg.h
> @@ -83,6 +83,7 @@ struct internal_config {
>  	volatile enum rte_intr_mode vfio_intr_mode;
>  	const char *hugefile_prefix;      /**< the base filename of hugetlbfs files */
>  	const char *hugepage_dir;         /**< specific hugetlbfs directory to use */
> +	const char *self_discovery;       /**< specific type of self_discovery */
>  
>  	unsigned num_hugepage_sizes;      /**< how many sizes on this system */
>  	struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES];
> diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
> index a881c62..a499d73 100644
> --- a/lib/librte_eal/common/eal_options.h
> +++ b/lib/librte_eal/common/eal_options.h
> @@ -83,6 +83,8 @@ enum {
>  	OPT_VMWARE_TSC_MAP_NUM,
>  #define OPT_XEN_DOM0          "xen-dom0"
>  	OPT_XEN_DOM0_NUM,
> +#define OPT_SELF_DISCOVERY    "self-discovery"
> +	OPT_SELF_DISCOVERY_NUM,
>  	OPT_LONG_MAX_NUM
>  };
>  
> diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
> index d2816a8..ff81484 100644
> --- a/lib/librte_eal/common/include/rte_eal.h
> +++ b/lib/librte_eal/common/include/rte_eal.h
> @@ -220,6 +220,40 @@ int rte_eal_has_hugepages(void);
>  int rte_sys_gettid(void);
>  
>  /**
> + * An API to query resource self discovery.
> + *
> + * @type
> + *   Type of self resource discovery.
> + * @p_corelist
> + *   If succeed, fill core list which can be used. Caller to free.
> + * @p_memory
> + *   If succeed, fill how many (bytes) memory can be used.
> + *
> + * @return
> + *   - (-1), if failed.
> + *   - 0, if succeed.
> + */
> +int rte_eal_res_self_discovery(const char *type,
> +			       char **p_corelist, uint64_t *p_memory);
> +/**
> + * An API to apply resource through self discovery.
> + *
> + * @type
> + *   Type of self resource discovery.
> + * @enable_core
> + *   If succeed, apply core resource.
> + * @p_memory
> + *   If succeed, apply memory resource.
> + *
> + * @return
> + *   - (-1), if failed.
> + *   - 0, if succeed.
> + */
> +int rte_eal_res_self_discovery_apply(const char *type,
> +				     int enable_core, int enable_mem);
> +
> +
> +/**
>   * Get system unique thread id.
>   *
>   * @return
> diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
> index 26eced5..834ae2f 100644
> --- a/lib/librte_eal/linuxapp/eal/Makefile
> +++ b/lib/librte_eal/linuxapp/eal/Makefile
> @@ -87,6 +87,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_devargs.c
>  SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_dev.c
>  SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_options.c
>  SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_common_thread.c
> +SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_cgroup.c
>  SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += rte_malloc.c
>  SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += malloc_elem.c
>  SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += malloc_heap.c
> diff --git a/lib/librte_eal/linuxapp/eal/eal_cgroup.c b/lib/librte_eal/linuxapp/eal/eal_cgroup.c
> new file mode 100644
> index 0000000..d6a04ee
> --- /dev/null
> +++ b/lib/librte_eal/linuxapp/eal/eal_cgroup.c
> @@ -0,0 +1,294 @@
> +/*-
> + *   BSD LICENSE
> + *
> + *   Copyright(c) 2010-2016 Intel Corporation. All rights reserved.
> + *   All rights reserved.
> + *
> + *   Redistribution and use in source and binary forms, with or without
> + *   modification, are permitted provided that the following conditions
> + *   are met:
> + *
> + *     * Redistributions of source code must retain the above copyright
> + *       notice, this list of conditions and the following disclaimer.
> + *     * Redistributions in binary form must reproduce the above copyright
> + *       notice, this list of conditions and the following disclaimer in
> + *       the documentation and/or other materials provided with the
> + *       distribution.
> + *     * Neither the name of Intel Corporation nor the names of its
> + *       contributors may be used to endorse or promote products derived
> + *       from this software without specific prior written permission.
> + *
> + *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> + *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> + *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> + *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> + *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> + *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> + *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> + *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> + *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> + */
> +
> +#include <unistd.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <stdlib.h>
> +#include <sys/stat.h>
> +#include <sys/file.h>
> +#include <sys/types.h>
> +#include <dirent.h>
> +#include <mntent.h>
> +#include <inttypes.h>
> +
> +#include <rte_log.h>
> +#include <rte_eal.h>
> +#include <rte_common.h>
> +
> +#include "eal_internal_cfg.h"
> +
> +static int pid;
> +
> +static char *
> +find_controller_dfs(const char *dir_path)
> +{
> +	FILE *f;
> +	char *line;
> +	char *ret;
> +	size_t len;
> +	ssize_t read;
> +	DIR *dir;
> +	struct dirent *ent;
> +	char filepath[PATH_MAX];
> +
> +	// 1. check if this process belongs to this cgroup
> +	snprintf(filepath, sizeof(filepath)-1, "%s/tasks", dir_path);
> +	f = fopen(filepath, "r");
> +	if (f == NULL)
> +		return NULL;
> +	len = 0;
> +	line = NULL;
> +	while ((read = getline(&line, &len, f)) != -1) {
> +		int _pid = atoi(line);
> +		free(line);
> +		if (_pid == pid)
> +			break;
> +		len = 0;
> +		line = NULL;
> +	}
> +	fclose(f);
> +	if (read != -1)
> +		return strdup(dir_path);
> +
> +	// 2. check its child cgroup
> +	if (!(dir = opendir(dir_path)))
> +		return NULL;
> +
> +	ret = NULL;
> +	while ((ent = readdir(dir)) != NULL) {
> +		if (ent->d_type != DT_DIR)
> +			continue;
> +		if (strcmp(ent->d_name, ".") == 0 ||
> +		    strcmp(ent->d_name, "..") == 0)
> +			continue;
> +
> +		snprintf(filepath, sizeof(filepath)-1, "%s/%s",
> +			 dir_path, ent->d_name);
> +
> +		ret = find_controller_dfs(filepath);
> +		if (ret != NULL)
> +			break;
> +	}
> +
> +	closedir(dir);
> +	return ret;
> +}
> +
> +static char *
> +find_controller(const char *controller)
> +{
> +	FILE *f;
> +	char *path;
> +	struct mntent *ent;
> +
> +	static const char *proc_mounts = "/proc/mounts";
> +	static const char *fs_type = "cgroup";
> +
> +	f = setmntent(proc_mounts, "r");
> +	if (f == NULL) {
> +		RTE_LOG(ERR, EAL, "Cannot open %s\n", proc_mounts);
> +		return NULL;
> +	}
> +
> +	while (NULL != (ent = getmntent(f))) {
> +		if (strcmp(ent->mnt_type, fs_type) != 0)
> +			continue;
> +		if (hasmntopt(ent, controller) == NULL)
> +			continue;
> +		break;
> +	}
> +
> +	if (ent == NULL) {
> +		path = NULL;
> +		goto end;
> +	}
> +
> +	path = find_controller_dfs(ent->mnt_dir);
> +end:
> +	endmntent(f);
> +	return path;
> +}
> +
> +static inline char *
> +get_oneline_from_file(const char *path)
> +{
> +	FILE *f;
> +	char *line = NULL;
> +	size_t len = 0;
> +
> +	if (NULL == (f = fopen(path, "r")))
> +		return NULL;
> +	if (getline(&line, &len, f) == -1)
> +		line = NULL;
> +	line[strcspn(line, "\n")] = 0;
> +	fclose(f);
> +	return line;
> +}
> +
> +static int
> +cgroup_cpuset(char **p_corelist, int enable __rte_unused)
> +{
> +	char filepath[PATH_MAX];
> +	char *controller;
> +
> +       	controller = find_controller("cpuset");
> +	if (controller == NULL)
> +		return -1;
> +
> +	snprintf(filepath, sizeof(filepath)-1, "%s/cpuset.cpus", controller);
> +	*p_corelist = get_oneline_from_file(filepath);
> +	RTE_LOG(INFO, EAL, "cgroup cpuset: %s\n", *p_corelist);
> +	return 0;
> +
> +}
> +
> +static inline uint64_t
> +get_hugetlb_limit(const char *path)
> +{
> +	uint64_t limit;
> +	char *str;
> +
> +       	str = get_oneline_from_file(path);
> +	sscanf(str, "%"PRIu64, &limit);
> +	free(str);
> +	return limit;
> +}
> +
> +static int
> +cgroup_hugetlb(uint64_t *p_memory, int enable)
> +{
> +	unsigned i;
> +	char filepath[PATH_MAX];
> +	char *controller;
> +	DIR *dir;
> +	struct dirent *ent;
> +	uint64_t memory = 0;
> +	static char prefix[] = "hugetlb";
> +	static int prefix_len = sizeof(prefix) - 1;
> +	static char suffix[] = "limit_in_bytes";
> +
> +       	controller = find_controller("hugetlb");
> +	if (controller == NULL)
> +		return -1;
> +
> +	if (!(dir = opendir(controller)))
> +		return -1;
> +
> +	while ((ent = readdir(dir)) != NULL) {
> +		if (strncmp(ent->d_name, prefix, prefix_len) != 0)
> +			continue;
> +
> +		char *sz_beg = ent->d_name + prefix_len + 1;
> +		char *sz_end = strchr(sz_beg, '.');
> +
> +		if (strcmp(sz_end + 1, suffix) != 0)
> +			continue;
> +
> +		char *tmp = strndup(sz_beg, sz_end - sz_beg);
> +		uint64_t pagesize = rte_str_to_size(tmp);
> +		free(tmp);
> +
> +		snprintf(filepath, sizeof(filepath)-1, "%s/%s",
> +			 controller, ent->d_name);
> +		uint64_t m_limit = get_hugetlb_limit(filepath);
> +		memory += m_limit;
> +
> +		/* Record those information into internal_config if hugepages
> +		 * are already initialized.
> +		 */
> +		if (! enable)
> +			continue;
> +		for (i = 0; i < internal_config.num_hugepage_sizes; ++i) {
> +			struct hugepage_info *hp;
> +
> +		       	hp = &internal_config.hugepage_info[i];
> +			if (hp->hugepage_sz != pagesize)
> +				continue;
> +
> +			if (m_limit < hp->hugepage_sz * hp->num_pages[0])
> +				hp->num_pages[0] = m_limit / hp->hugepage_sz;
> +		}
> +	}
> +
> +	closedir(dir);
> +	*p_memory = memory;
> +	RTE_LOG(INFO, EAL, "cgroup hugetlb: %"PRIx64"\n", *p_memory);
> +	return 0;
> +}
> +
> +static int
> +resource_self_discovery(const char *type, char **p_corelist, int enable_core,
> +			uint64_t *p_memory, int enable_mem)
> +{
> +	if (strcmp(type, "cgroup") != 0) {
> +		RTE_LOG(ERR, EAL, "type not supported: %s\n", type);
> +		return -1;
> +	}
> +
> +	pid = getpid();
> +
> +	if (p_corelist != NULL && cgroup_cpuset(p_corelist, enable_core) < 0) {
> +		RTE_LOG(ERR, EAL, "Failed when discover resource cpuset\n");
> +		return -1;
> +	}
> +	if (p_memory != NULL && cgroup_hugetlb(p_memory, enable_mem) < 0) {
> +		RTE_LOG(ERR, EAL, "Failed when discover resource hugetlb\n");
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +int
> +rte_eal_res_self_discovery(const char *type, char **p_corelist,
> +			   uint64_t *p_memory)
> +{
> +	return resource_self_discovery(type, p_corelist, 0, p_memory, 0);
> +}
> +
> +int
> +rte_eal_res_self_discovery_apply(const char *type, int enable_core,
> +				 int enable_mem)
> +{
> +	char *corelist, **pc = NULL;
> +	uint64_t mem, *pm = NULL;
> +	
> +	if (enable_core)
> +		pc = &corelist;
> +	if (enable_mem)
> +		pm = &mem;
> +
> +	return resource_self_discovery(type, pc, enable_core,
> +				       pm, enable_mem);
> +}
> diff --git a/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c b/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
> index 18858e2..a6b6548 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_hugepage_info.c
> @@ -354,6 +354,11 @@ eal_hugepage_info_init(void)
>  	qsort(&internal_config.hugepage_info[0], num_sizes,
>  	      sizeof(internal_config.hugepage_info[0]), compare_hpi);
>  
> +	/* Apply cgroup hugetlb limit before we really use hugepages */
> +	if (internal_config.self_discovery)
> +		rte_eal_res_self_discovery_apply(internal_config.self_discovery,
> +						 0, 1);
> +
>  	/* now we have all info, check we have at least one valid size */
>  	for (i = 0; i < num_sizes; i++)
>  		if (internal_config.hugepage_info[i].hugedir != NULL &&
> -- 
> 2.1.4
> 
> 


This doesn't make a whole lot of sense, for several reasons:

1) Applications, as a general rule shouldn't be interrogating the cgroups
interface at all.  

2) Cgroups aren't the only way in which a cpuset or memoryset can be restricted
(the isolcpus command line argument, or a taskset on a parent process for
instance, but there are several others).

Instead of trying to figure out what cpuset is valid for your process by
interrogating the cgroups heirarchy, instead you should follow the proscribed
method of calling sched_getaffinity after calling sched_setaffinity.  That will
give you the canonical cpuset that you are executing on, taking all cpuset
filters into account (including cgroups and any other restrictions).  Its far
simpler as well, as it doesn't require a ton of file/string processing.

Neil

Re: [RFC] eal: add cgroup-aware resource self discovery

[Tan, Jianfeng]

Hi Neil,

On 1/25/2016 9:46 PM, Neil Horman wrote:
> On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote:
...
>> -- 
>> 2.1.4
>>
>>
>
> This doesn't make a whole lot of sense, for several reasons:
>
> 1) Applications, as a general rule shouldn't be interrogating the cgroups
> interface at all.

The main reason to do this in DPDK is that DPDK obtains resource 
information from sysfs and proc, which are not well containerized so 
far. And DPDK pre-allocates resource instead of on-demand gradual 
allocating.

>
> 2) Cgroups aren't the only way in which a cpuset or memoryset can be restricted
> (the isolcpus command line argument, or a taskset on a parent process for
> instance, but there are several others).

Yes, I agree. To enable that, I'd like design the new API for resource 
self discovery in a flexible way. A parameter "type" is used to specify 
the solution to discovery way. In addition, I'm considering to add a 
callback function pointer so that users can write their own resource 
discovery functions.

>
> Instead of trying to figure out what cpuset is valid for your process by
> interrogating the cgroups heirarchy, instead you should follow the proscribed
> method of calling sched_getaffinity after calling sched_setaffinity.  That will
> give you the canonical cpuset that you are executing on, taking all cpuset
> filters into account (including cgroups and any other restrictions).  Its far
> simpler as well, as it doesn't require a ton of file/string processing.

Yes, this way is much better for cpuset discovery. But is there such a 
syscall for hugepages?

Thanks,
Jianfeng

>
> Neil
>

Re: [RFC] eal: add cgroup-aware resource self discovery

[Neil Horman]

On Tue, Jan 26, 2016 at 10:22:18AM +0800, Tan, Jianfeng wrote:
> 
> Hi Neil,
> 
> On 1/25/2016 9:46 PM, Neil Horman wrote:
> >On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote:
> ...
> >>-- 
> >>2.1.4
> >>
> >>
> >
> >This doesn't make a whole lot of sense, for several reasons:
> >
> >1) Applications, as a general rule shouldn't be interrogating the cgroups
> >interface at all.
> 
> The main reason to do this in DPDK is that DPDK obtains resource information
> from sysfs and proc, which are not well containerized so far. And DPDK
> pre-allocates resource instead of on-demand gradual allocating.
> 
Not disagreeing with this, just suggesting that:

1) Interrogating cgroups really isn't the best way to collect that information
2) Pre-allocating those resources isn't particularly wise without some mechanism
to reallocate it, as resource constraints can change (consider your cpuset
getting rewritten)

> >
> >2) Cgroups aren't the only way in which a cpuset or memoryset can be restricted
> >(the isolcpus command line argument, or a taskset on a parent process for
> >instance, but there are several others).
> 
> Yes, I agree. To enable that, I'd like design the new API for resource self
> discovery in a flexible way. A parameter "type" is used to specify the
> solution to discovery way. In addition, I'm considering to add a callback
> function pointer so that users can write their own resource discovery
> functions.
> 
Why?  You don't need an API for this, or if you really want one, it can be very
generic if you use POSIX apis to gather the information.  What you have here is
going to be very linux specific, and will need reimplementing for BSD or other
operating systems.  To use the cpuset example, instead of reading and parsing
the mask files in the cgroup filesystem module to find your task and
corresponding mask, just call sched_setaffinity with an all f's mask, then call
sched_getaffinity.  The returned mask will be all the cpus your process is
allowed to execute on, taking into account every limiting filter the system you
are running on offers.

There are simmilar OS level POSIX apis for most resources out there.  You really
don't need to dig through cgroups just to learn what some of those reources are.

> >
> >Instead of trying to figure out what cpuset is valid for your process by
> >interrogating the cgroups heirarchy, instead you should follow the proscribed
> >method of calling sched_getaffinity after calling sched_setaffinity.  That will
> >give you the canonical cpuset that you are executing on, taking all cpuset
> >filters into account (including cgroups and any other restrictions).  Its far
> >simpler as well, as it doesn't require a ton of file/string processing.
> 
> Yes, this way is much better for cpuset discovery. But is there such a
> syscall for hugepages?
> 
In what capacity?  Interrogating how many hugepages you have, or to what node
they are affined to?  Capacity would require reading the requisite proc file, as
theres no posix api for this resource.  Node affinity can be implied by setting
the numa policy of the dpdk and then writing to /proc/nr_hugepages, as the
kernel will attempt to distribute hugepages evenly among the tasks' numa policy
configuration.

That said, I would advise that you strongly consider not exporting hugepages as
a resource, as:

a) Applications generally don't need to know that they are using hugepages, and
so they dont need to know where said hugepages live, they just allocate memory
via your allocation api and you give them something appropriate

b) Hugepages are a resource that are very specific to Linux, and to X86 Linux at
that.  Some OS implement simmilar resources, but they may have very different
semantics.  And other Arches may or may not implement various forms of compound
paging at all.  As the DPDK expands to support more OS'es and arches, it would
be nice to ensure that the programming surfaces that you expose have a more
broad level of support.

Neil

> Thanks,
> Jianfeng
> 
> >
> >Neil
> >
> 
> 

Re: [RFC] eal: add cgroup-aware resource self discovery

[Tan, Jianfeng]

Hi Neil,

On 1/26/2016 10:19 PM, Neil Horman wrote:
> On Tue, Jan 26, 2016 at 10:22:18AM +0800, Tan, Jianfeng wrote:
>> Hi Neil,
>>
>> On 1/25/2016 9:46 PM, Neil Horman wrote:
>>> On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote:
>> ...
>>>> -- 
>>>> 2.1.4
>>>>
>>>>
>>> This doesn't make a whole lot of sense, for several reasons:
>>>
>>> 1) Applications, as a general rule shouldn't be interrogating the cgroups
>>> interface at all.
>> The main reason to do this in DPDK is that DPDK obtains resource information
>> from sysfs and proc, which are not well containerized so far. And DPDK
>> pre-allocates resource instead of on-demand gradual allocating.
>>
> Not disagreeing with this, just suggesting that:
>
> 1) Interrogating cgroups really isn't the best way to collect that information
> 2) Pre-allocating those resources isn't particularly wise without some mechanism
> to reallocate it, as resource constraints can change (consider your cpuset
> getting rewritten)

In the case of reallocate,
For cpuset, DPDK panics in the initialization if set_affinity fails, but 
after that, cpuset rewritten will not bring any problem I believe.
For memory, a running application uses 2G hugepages, then admin 
decreases hugetlb cgroup into 1G, the application will not get killed, 
unless it tries to access more hugepages (I'll double check this).

So another way to address this problem is to add an option that DPDK 
tries best to allocate those resources, and if fails, it just posts a 
warning and uses those allocated resources, instead of panic. What do 
you think?

>
>>> 2) Cgroups aren't the only way in which a cpuset or memoryset can be restricted
>>> (the isolcpus command line argument, or a taskset on a parent process for
>>> instance, but there are several others).
>> Yes, I agree. To enable that, I'd like design the new API for resource self
>> discovery in a flexible way. A parameter "type" is used to specify the
>> solution to discovery way. In addition, I'm considering to add a callback
>> function pointer so that users can write their own resource discovery
>> functions.
>>
> Why?  You don't need an API for this, or if you really want one, it can be very
> generic if you use POSIX apis to gather the information.  What you have here is
> going to be very linux specific, and will need reimplementing for BSD or other
> operating systems.  To use the cpuset example, instead of reading and parsing
> the mask files in the cgroup filesystem module to find your task and
> corresponding mask, just call sched_setaffinity with an all f's mask, then call
> sched_getaffinity.  The returned mask will be all the cpus your process is
> allowed to execute on, taking into account every limiting filter the system you
> are running on offers.

Yes, it makes sense on cpu's side.

>
> There are simmilar OS level POSIX apis for most resources out there.  You really
> don't need to dig through cgroups just to learn what some of those reources are.
>
>>> Instead of trying to figure out what cpuset is valid for your process by
>>> interrogating the cgroups heirarchy, instead you should follow the proscribed
>>> method of calling sched_getaffinity after calling sched_setaffinity.  That will
>>> give you the canonical cpuset that you are executing on, taking all cpuset
>>> filters into account (including cgroups and any other restrictions).  Its far
>>> simpler as well, as it doesn't require a ton of file/string processing.
>> Yes, this way is much better for cpuset discovery. But is there such a
>> syscall for hugepages?
>>
> In what capacity?  Interrogating how many hugepages you have, or to what node
> they are affined to?  Capacity would require reading the requisite proc file, as
> theres no posix api for this resource.  Node affinity can be implied by setting
> the numa policy of the dpdk and then writing to /proc/nr_hugepages, as the
> kernel will attempt to distribute hugepages evenly among the tasks' numa policy
> configuration.

For memory affinity, I believe the existing way of reading 
/proc/self/pagemap already handle the problem. What I was asking is how 
much memory (or hugepages in Linux's case) can be used. By the way, what 
is /proc/nr_hugepages?

>
> That said, I would advise that you strongly consider not exporting hugepages as
> a resource, as:
>
> a) Applications generally don't need to know that they are using hugepages, and
> so they dont need to know where said hugepages live, they just allocate memory
> via your allocation api and you give them something appropriate

But the allocation api provider, DPDK library, needs to know if it's 
using hugepages or not.

> b) Hugepages are a resource that are very specific to Linux, and to X86 Linux at
> that.  Some OS implement simmilar resources, but they may have very different
> semantics.  And other Arches may or may not implement various forms of compound
> paging at all.  As the DPDK expands to support more OS'es and arches, it would
> be nice to ensure that the programming surfaces that you expose have a more
> broad level of support.

That's why I put current implement in lib/librte_eal/linuxapp/. And the 
new API uses the words of cores and memory, which is very generic IMO. 
In Linux's context, memory is interpreted into hugepages (maybe not 
correct because DPDK can be used with 4K memory). For other OSes, we 
could add similar limitation in their semantics.


Thanks,
Jianfeng

>
> Neil
>
>> Thanks,
>> Jianfeng
>>
>>> Neil
>>>
>>

Re: [RFC] eal: add cgroup-aware resource self discovery

[Neil Horman]

On Wed, Jan 27, 2016 at 08:02:27PM +0800, Tan, Jianfeng wrote:
> Hi Neil,
> 
> On 1/26/2016 10:19 PM, Neil Horman wrote:
> >On Tue, Jan 26, 2016 at 10:22:18AM +0800, Tan, Jianfeng wrote:
> >>Hi Neil,
> >>
> >>On 1/25/2016 9:46 PM, Neil Horman wrote:
> >>>On Mon, Jan 25, 2016 at 02:49:53AM +0800, Jianfeng Tan wrote:
> >>...
> >>>>-- 
> >>>>2.1.4
> >>>>
> >>>>
> >>>This doesn't make a whole lot of sense, for several reasons:
> >>>
> >>>1) Applications, as a general rule shouldn't be interrogating the cgroups
> >>>interface at all.
> >>The main reason to do this in DPDK is that DPDK obtains resource information
> >>from sysfs and proc, which are not well containerized so far. And DPDK
> >>pre-allocates resource instead of on-demand gradual allocating.
> >>
> >Not disagreeing with this, just suggesting that:
> >
> >1) Interrogating cgroups really isn't the best way to collect that information
> >2) Pre-allocating those resources isn't particularly wise without some mechanism
> >to reallocate it, as resource constraints can change (consider your cpuset
> >getting rewritten)
> 
> In the case of reallocate,
> For cpuset, DPDK panics in the initialization if set_affinity fails, but
> after that, cpuset rewritten will not bring any problem I believe.
Yes, that seems reasonable, but I think you need to update
rte_thread_set_affinity to not assume that success in pthread_setaffinity_np
means that all cpus in the provided mask are available.  That is to say, cpusetp
is subsequently stored in lore information after the set, but may not reflect
the actual working set of processors, you should follow a successful set with a
call to pthread_getaffinity_np to retrieve the actual working cpuset

As for subsequent changes to the cpuset, I'm not sure how you want to handle
that. I would think that you might want to run a check periodically or alow for
a SIGHUP or some other signal to trigger a rescan of your working cpuset so as
to keep the application in sync with the system.

> For memory, a running application uses 2G hugepages, then admin decreases
> hugetlb cgroup into 1G, the application will not get killed, unless it tries
> to access more hugepages (I'll double check this).
> 
No, the semantics should be identical to malloc/mmap (if you use the alloc_hugepages
api or the mmap api).  You should get a NULL return or other no fatal indicator
if you allocate more than is available.

> So another way to address this problem is to add an option that DPDK tries
> best to allocate those resources, and if fails, it just posts a warning and
> uses those allocated resources, instead of panic. What do you think?
> 
Yes, that makes sense

> >
> >>>2) Cgroups aren't the only way in which a cpuset or memoryset can be restricted
> >>>(the isolcpus command line argument, or a taskset on a parent process for
> >>>instance, but there are several others).
> >>Yes, I agree. To enable that, I'd like design the new API for resource self
> >>discovery in a flexible way. A parameter "type" is used to specify the
> >>solution to discovery way. In addition, I'm considering to add a callback
> >>function pointer so that users can write their own resource discovery
> >>functions.
> >>
> >Why?  You don't need an API for this, or if you really want one, it can be very
> >generic if you use POSIX apis to gather the information.  What you have here is
> >going to be very linux specific, and will need reimplementing for BSD or other
> >operating systems.  To use the cpuset example, instead of reading and parsing
> >the mask files in the cgroup filesystem module to find your task and
> >corresponding mask, just call sched_setaffinity with an all f's mask, then call
> >sched_getaffinity.  The returned mask will be all the cpus your process is
> >allowed to execute on, taking into account every limiting filter the system you
> >are running on offers.
> 
> Yes, it makes sense on cpu's side.
> 
> >
> >There are simmilar OS level POSIX apis for most resources out there.  You really
> >don't need to dig through cgroups just to learn what some of those reources are.
> >
> >>>Instead of trying to figure out what cpuset is valid for your process by
> >>>interrogating the cgroups heirarchy, instead you should follow the proscribed
> >>>method of calling sched_getaffinity after calling sched_setaffinity.  That will
> >>>give you the canonical cpuset that you are executing on, taking all cpuset
> >>>filters into account (including cgroups and any other restrictions).  Its far
> >>>simpler as well, as it doesn't require a ton of file/string processing.
> >>Yes, this way is much better for cpuset discovery. But is there such a
> >>syscall for hugepages?
> >>
> >In what capacity?  Interrogating how many hugepages you have, or to what node
> >they are affined to?  Capacity would require reading the requisite proc file, as
> >theres no posix api for this resource.  Node affinity can be implied by setting
> >the numa policy of the dpdk and then writing to /proc/nr_hugepages, as the
> >kernel will attempt to distribute hugepages evenly among the tasks' numa policy
> >configuration.
> 
> For memory affinity, I believe the existing way of reading
> /proc/self/pagemap already handle the problem. What I was asking is how much
> memory (or hugepages in Linux's case) can be used. By the way, what is
> /proc/nr_hugepages?
> 
For affinity, you can parse /proc/self/pagemap or any number of other procfiles,
but again, doing so is going to be very OS specific, and doesn't get you much in
terms or resource management. It only tells you where the pages reside now.

/proc/nr_hugepages is the proc tunable that lets you allocate/realocate
hugepages.

> >
> >That said, I would advise that you strongly consider not exporting hugepages as
> >a resource, as:
> >
> >a) Applications generally don't need to know that they are using hugepages, and
> >so they dont need to know where said hugepages live, they just allocate memory
> >via your allocation api and you give them something appropriate
> 
> But the allocation api provider, DPDK library, needs to know if it's using
> hugepages or not.
> 
Right, but you're purpose was to expose thie library to applications.  I'm
saying you really don't need to expose such a library API to applications. If
you just want to use it internally to dpdk, thats fine.

> >b) Hugepages are a resource that are very specific to Linux, and to X86 Linux at
> >that.  Some OS implement simmilar resources, but they may have very different
> >semantics.  And other Arches may or may not implement various forms of compound
> >paging at all.  As the DPDK expands to support more OS'es and arches, it would
> >be nice to ensure that the programming surfaces that you expose have a more
> >broad level of support.
> 
> That's why I put current implement in lib/librte_eal/linuxapp/. And the new
> API uses the words of cores and memory, which is very generic IMO. In
> Linux's context, memory is interpreted into hugepages (maybe not correct
> because DPDK can be used with 4K memory). For other OSes, we could add
> similar limitation in their semantics.
> 
> 
> Thanks,
> Jianfeng
> 
> >
> >Neil
> >
> >>Thanks,
> >>Jianfeng
> >>
> >>>Neil
> >>>
> >>
> 
> 

[PATCH] eal: make resource initialization more robust

[Jianfeng Tan]

Current issue: DPDK is not that friendly to container environment, which
caused by that it pre-alloc resource like cores and hugepages. But there
are this or that resource limitations, for examples, cgroup, rlimit,
cpuset, etc.

For cores, this patch makes use of pthread_getaffinity_np to further
narrow down detected cores before parsing coremask (-c), corelist (-l),
and coremap (--lcores).

For hugepages, this patch adds a recover mechanism to the case that
there are no that many hugepages can be used. It relys on a mem access
to fault-in hugepages, and if fails with SIGBUS, recover to previously
saved stack environment with siglongjmp().

Test example:
    a. cgcreate -g cpuset,hugetlb:/test-subgroup
    b. cgset -r cpuset.cpus=2-3 test-subgroup
    c. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
    d. cgexec -g cpuset,hugetlb:test-subgroup \
	    ./examples/l2fwd/build/l2fwd -n 4 -- -p 3

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 lib/librte_eal/common/eal_common_lcore.c | 10 +++-
 lib/librte_eal/linuxapp/eal/eal_memory.c | 78 ++++++++++++++++++++++++++++----
 2 files changed, 79 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
index a4263ba..8e9c675 100644
--- a/lib/librte_eal/common/eal_common_lcore.c
+++ b/lib/librte_eal/common/eal_common_lcore.c
@@ -57,6 +57,13 @@ rte_eal_cpu_init(void)
 	struct rte_config *config = rte_eal_get_configuration();
 	unsigned lcore_id;
 	unsigned count = 0;
+	rte_cpuset_t cpuset;
+	pthread_t tid;
+
+	tid = pthread_self();
+	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cpuset) != 0)
+		for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++)
+			CPU_SET(lcore_id, &cpuset);
 
 	/*
 	 * Parse the maximum set of logical cores, detect the subset of running
@@ -70,7 +77,8 @@ rte_eal_cpu_init(void)
 
 		/* in 1:1 mapping, record related cpu detected state */
 		lcore_config[lcore_id].detected = eal_cpu_detected(lcore_id);
-		if (lcore_config[lcore_id].detected == 0) {
+		if (lcore_config[lcore_id].detected == 0 ||
+		    !CPU_ISSET(lcore_id, &cpuset)) {
 			config->lcore_role[lcore_id] = ROLE_OFF;
 			lcore_config[lcore_id].core_index = -1;
 			continue;
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 846fd31..837fd9e 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,12 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf jmpenv;
+
+static void sigbus_handler(int signo __rte_unused)
+{
+	siglongjmp(jmpenv, 1);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
@@ -396,7 +404,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		if (fd < 0) {
 			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
@@ -405,11 +413,26 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
 			hugepg_tbl[i].orig_va = virtaddr;
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (sigsetjmp(jmpenv, 0)) {
+				RTE_LOG(ERR, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				return i;
+			}
 			memset(virtaddr, 0, hugepage_sz);
 		}
 		else {
@@ -421,7 +444,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -429,7 +452,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1075,6 +1098,31 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static struct sigaction action_old;
+static int need_recover = 0;
+
+static void
+register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = sigbus_handler;
+
+	need_recover = !sigaction(SIGBUS, &action, &action_old);
+}
+
+static void
+recover_sigbus(void)
+{
+	if (need_recover)
+		sigaction(SIGBUS, &action_old, NULL);
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1161,8 +1209,11 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		int pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1176,10 +1227,19 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
-			goto fail;
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+			internal_config.memory -=
+				hpi->hugepage_sz * (pages_old - pages_new);
+			nr_hugepages -= (pages_old - pages_new);
+			hpi->num_pages[0] = pages_new;
+			if (pages_new == 0)
+				continue;
 		}
 
 		/* find physical addresses and sockets for each hugepage */
@@ -1226,6 +1286,8 @@ rte_eal_hugepage_init(void)
 #endif
 	}
 
+	recover_sigbus();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
-- 
2.1.4

Re: [PATCH] eal: make resource initialization more robust

[Neil Horman]

On Fri, Jan 29, 2016 at 07:22:02PM +0800, Jianfeng Tan wrote:
> Current issue: DPDK is not that friendly to container environment, which
> caused by that it pre-alloc resource like cores and hugepages. But there
> are this or that resource limitations, for examples, cgroup, rlimit,
> cpuset, etc.
> 
> For cores, this patch makes use of pthread_getaffinity_np to further
> narrow down detected cores before parsing coremask (-c), corelist (-l),
> and coremap (--lcores).
> 
> For hugepages, this patch adds a recover mechanism to the case that
> there are no that many hugepages can be used. It relys on a mem access
> to fault-in hugepages, and if fails with SIGBUS, recover to previously
> saved stack environment with siglongjmp().
> 
> Test example:
>     a. cgcreate -g cpuset,hugetlb:/test-subgroup
>     b. cgset -r cpuset.cpus=2-3 test-subgroup
>     c. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>     d. cgexec -g cpuset,hugetlb:test-subgroup \
> 	    ./examples/l2fwd/build/l2fwd -n 4 -- -p 3
> 
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> ---
>  lib/librte_eal/common/eal_common_lcore.c | 10 +++-
>  lib/librte_eal/linuxapp/eal/eal_memory.c | 78 ++++++++++++++++++++++++++++----
>  2 files changed, 79 insertions(+), 9 deletions(-)
> 

This looks alot better.  One minor comment, the sigbus handler, you should
probably store the previous bus handler and restore it after you map all the
hugepages you want (lest you overwrite something an application is doing with
sigbus).


Other than that, nice work.
Neil
 
> diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
> index a4263ba..8e9c675 100644
> --- a/lib/librte_eal/common/eal_common_lcore.c
> +++ b/lib/librte_eal/common/eal_common_lcore.c
> @@ -57,6 +57,13 @@ rte_eal_cpu_init(void)
>  	struct rte_config *config = rte_eal_get_configuration();
>  	unsigned lcore_id;
>  	unsigned count = 0;
> +	rte_cpuset_t cpuset;
> +	pthread_t tid;
> +
> +	tid = pthread_self();
> +	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cpuset) != 0)
> +		for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++)
> +			CPU_SET(lcore_id, &cpuset);
>  
>  	/*
>  	 * Parse the maximum set of logical cores, detect the subset of running
> @@ -70,7 +77,8 @@ rte_eal_cpu_init(void)
>  
>  		/* in 1:1 mapping, record related cpu detected state */
>  		lcore_config[lcore_id].detected = eal_cpu_detected(lcore_id);
> -		if (lcore_config[lcore_id].detected == 0) {
> +		if (lcore_config[lcore_id].detected == 0 ||
> +		    !CPU_ISSET(lcore_id, &cpuset)) {
>  			config->lcore_role[lcore_id] = ROLE_OFF;
>  			lcore_config[lcore_id].core_index = -1;
>  			continue;
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 846fd31..837fd9e 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -80,6 +80,8 @@
>  #include <errno.h>
>  #include <sys/ioctl.h>
>  #include <sys/time.h>
> +#include <signal.h>
> +#include <setjmp.h>
>  
>  #include <rte_log.h>
>  #include <rte_memory.h>
> @@ -309,6 +311,12 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
>  	return addr;
>  }
>  
> +static sigjmp_buf jmpenv;
> +
> +static void sigbus_handler(int signo __rte_unused)
> +{
> +	siglongjmp(jmpenv, 1);
> +}
>  /*
>   * Mmap all hugepages of hugepage table: it first open a file in
>   * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
> @@ -396,7 +404,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>  		if (fd < 0) {
>  			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
>  					strerror(errno));
> -			return -1;
> +			return i;
>  		}
>  
>  		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
> @@ -405,11 +413,26 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>  			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
>  					strerror(errno));
>  			close(fd);
> -			return -1;
> +			return i;
>  		}
>  
>  		if (orig) {
>  			hugepg_tbl[i].orig_va = virtaddr;
> +			/* In linux, hugetlb limitations, like cgroup, are
> +			 * enforced at fault time instead of mmap(), even
> +			 * with the option of MAP_POPULATE. Kernel will send
> +			 * a SIGBUS signal. To avoid to be killed, save stack
> +			 * environment here, if SIGBUS happens, we can jump
> +			 * back here.
> +			 */
> +			if (sigsetjmp(jmpenv, 0)) {
> +				RTE_LOG(ERR, EAL, "SIGBUS: Cannot mmap more "
> +					"hugepages of size %u MB\n",
> +					(unsigned)(hugepage_sz / 0x100000));
> +				munmap(virtaddr, hugepage_sz);
> +				close(fd);
> +				return i;
> +			}
>  			memset(virtaddr, 0, hugepage_sz);
>  		}
>  		else {
> @@ -421,7 +444,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>  			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
>  				__func__, strerror(errno));
>  			close(fd);
> -			return -1;
> +			return i;
>  		}
>  
>  		close(fd);
> @@ -429,7 +452,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>  		vma_addr = (char *)vma_addr + hugepage_sz;
>  		vma_len -= hugepage_sz;
>  	}
> -	return 0;
> +	return i;
>  }
>  
>  #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
> @@ -1075,6 +1098,31 @@ calc_num_pages_per_socket(uint64_t * memory,
>  	return total_num_pages;
>  }
>  
> +static struct sigaction action_old;
> +static int need_recover = 0;
> +
> +static void
> +register_sigbus(void)
> +{
> +	sigset_t mask;
> +	struct sigaction action;
> +
> +	sigemptyset(&mask);
> +	sigaddset(&mask, SIGBUS);
> +	action.sa_flags = 0;
> +	action.sa_mask = mask;
> +	action.sa_handler = sigbus_handler;
> +
> +	need_recover = !sigaction(SIGBUS, &action, &action_old);
> +}
> +
> +static void
> +recover_sigbus(void)
> +{
> +	if (need_recover)
> +		sigaction(SIGBUS, &action_old, NULL);
> +}
> +
>  /*
>   * Prepare physical memory mapping: fill configuration structure with
>   * these infos, return 0 on success.
> @@ -1161,8 +1209,11 @@ rte_eal_hugepage_init(void)
>  
>  	hp_offset = 0; /* where we start the current page size entries */
>  
> +	register_sigbus();
> +
>  	/* map all hugepages and sort them */
>  	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
> +		int pages_old, pages_new;
>  		struct hugepage_info *hpi;
>  
>  		/*
> @@ -1176,10 +1227,19 @@ rte_eal_hugepage_init(void)
>  			continue;
>  
>  		/* map all hugepages available */
> -		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
> -			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
> -					(unsigned)(hpi->hugepage_sz / 0x100000));
> -			goto fail;
> +		pages_old = hpi->num_pages[0];
> +		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
> +		if (pages_new < pages_old) {
> +			RTE_LOG(DEBUG, EAL,
> +				"%d not %d hugepages of size %u MB allocated\n",
> +				pages_new, pages_old,
> +				(unsigned)(hpi->hugepage_sz / 0x100000));
> +			internal_config.memory -=
> +				hpi->hugepage_sz * (pages_old - pages_new);
> +			nr_hugepages -= (pages_old - pages_new);
> +			hpi->num_pages[0] = pages_new;
> +			if (pages_new == 0)
> +				continue;
>  		}
>  
>  		/* find physical addresses and sockets for each hugepage */
> @@ -1226,6 +1286,8 @@ rte_eal_hugepage_init(void)
>  #endif
>  	}
>  
> +	recover_sigbus();
> +
>  #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
>  	nr_hugefiles = 0;
>  	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
> -- 
> 2.1.4
> 
> 

Re: [PATCH] eal: make resource initialization more robust

[Tan, Jianfeng]

Hi Neil,

Sorry that for my previous misconfiguration of email agent, I missed 
this email.

> This looks alot better.  One minor comment, the sigbus handler, you should
> probably store the previous bus handler and restore it after you map 
> all the
> hugepages you want (lest you overwrite something an application is 
> doing with
> sigbus).
>

I did not catch your point. I did store it "static struct sigaction 
action_old" and recover it after mapping all the hugepages. Can you give 
more details on this?

Thanks,
Jianfeng

Re: [PATCH] eal: make resource initialization more robust

[Neil Horman]

On Mon, Feb 22, 2016 at 02:08:51PM +0800, Tan, Jianfeng wrote:
> Hi Neil,
> 
> Sorry that for my previous misconfiguration of email agent, I missed this
> email.
> 
> >This looks alot better.  One minor comment, the sigbus handler, you should
> >probably store the previous bus handler and restore it after you map all
> >the
> >hugepages you want (lest you overwrite something an application is doing
> >with
> >sigbus).
> >
> 
> I did not catch your point. I did store it "static struct sigaction
> action_old" and recover it after mapping all the hugepages. Can you give
> more details on this?
> 
Nope, I can't because I missed the fact you had done that.  Apologies, it looks
good.

Acked-by: Neil Horman <nhorman@tuxdriver.com>

> Thanks,
> Jianfeng
> 
> 

Re: [PATCH] eal: make resource initialization more robust

[Thomas Monjalon]

Hi,

2016-01-29 19:22, Jianfeng Tan:
> Current issue: DPDK is not that friendly to container environment, which
> caused by that it pre-alloc resource like cores and hugepages. But there
> are this or that resource limitations, for examples, cgroup, rlimit,
> cpuset, etc.
> 
> For cores, this patch makes use of pthread_getaffinity_np to further
> narrow down detected cores before parsing coremask (-c), corelist (-l),
> and coremap (--lcores).
> 
> For hugepages, this patch adds a recover mechanism to the case that
> there are no that many hugepages can be used. It relys on a mem access
> to fault-in hugepages, and if fails with SIGBUS, recover to previously
> saved stack environment with siglongjmp().

They are some interesting ideas.
However, I am not sure a library should try to be so smart silently.
It needs more feedback to decide wether it can be the default behaviour
or an option.

Please send coremask and hugepage mapping as separate patches as they
are totally different and may be integrated separately.

Thanks

Re: [PATCH] eal: make resource initialization more robust

[Tan, Jianfeng]

Hi Thomas,

On 2/29/2016 5:12 AM, Thomas Monjalon wrote:
> Hi,
>
> 2016-01-29 19:22, Jianfeng Tan:
>> Current issue: DPDK is not that friendly to container environment, which
>> caused by that it pre-alloc resource like cores and hugepages. But there
>> are this or that resource limitations, for examples, cgroup, rlimit,
>> cpuset, etc.
>>
>> For cores, this patch makes use of pthread_getaffinity_np to further
>> narrow down detected cores before parsing coremask (-c), corelist (-l),
>> and coremap (--lcores).
>>
>> For hugepages, this patch adds a recover mechanism to the case that
>> there are no that many hugepages can be used. It relys on a mem access
>> to fault-in hugepages, and if fails with SIGBUS, recover to previously
>> saved stack environment with siglongjmp().
> They are some interesting ideas.
> However, I am not sure a library should try to be so smart silently.
> It needs more feedback to decide wether it can be the default behaviour
> or an option.
>
> Please send coremask and hugepage mapping as separate patches as they
> are totally different and may be integrated separately.

Good advise, thanks! I'll do it.

And one more thing FYI, coremask using pthread_getaffinity_np() may have 
issue on some Linux versions or distros: it excludes isolcpus. This is 
reported by Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>, 
and I'm still working it out.

Thanks,
Jianfeng

>
> Thanks

[PATCH] eal: add option --avail-cores to detect lcores

[Jianfeng Tan]

This patch adds option, --avail-cores, to use lcores which are available
by calling pthread_getaffinity_np() to narrow down detected cores before
parsing coremask (-c), corelist (-l), and coremap (--lcores).

Test example:
$ taskset 0xc0000 ./examples/helloworld/build/helloworld \
		--avail-cores -m 1024

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
 lib/librte_eal/common/eal_common_options.c | 52 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_options.h        |  2 ++
 2 files changed, 54 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 29942ea..dc4882d 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+	{OPT_AVAIL_CORES,       0, NULL, OPT_AVAIL_CORES_NUM      },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -681,6 +682,37 @@ err:
 }
 
 static int
+eal_parse_avail_cores(void)
+{
+	int i, count;
+	pthread_t tid;
+	rte_cpuset_t cpuset;
+	struct rte_config *cfg = rte_eal_get_configuration();
+
+	tid = pthread_self();
+	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cpuset) != 0)
+		return -1;
+
+	for (i = 0, count = 0; i < RTE_MAX_LCORE; i++) {
+		if (lcore_config[i].detected && !CPU_ISSET(i, &cpuset)) {
+			RTE_LOG(DEBUG, EAL, "Flag lcore %u as undetected\n", i);
+			lcore_config[i].detected = 0;
+			lcore_config[i].core_index = -1;
+			cfg->lcore_role[i] = ROLE_OFF;
+			count++;
+		}
+	}
+	cfg->lcore_count -= count;
+	if (cfg->lcore_count == 0) {
+		RTE_LOG(ERR, EAL, "No lcores available\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+
+static int
 eal_parse_syslog(const char *facility, struct internal_config *conf)
 {
 	int i;
@@ -754,6 +786,10 @@ eal_parse_proc_type(const char *arg)
 	return RTE_PROC_INVALID;
 }
 
+static int param_coremask;
+static int param_corelist;
+static int param_coremap;
+
 int
 eal_parse_common_option(int opt, const char *optarg,
 			struct internal_config *conf)
@@ -775,6 +811,7 @@ eal_parse_common_option(int opt, const char *optarg,
 		break;
 	/* coremask */
 	case 'c':
+		param_coremask = 1;
 		if (eal_parse_coremask(optarg) < 0) {
 			RTE_LOG(ERR, EAL, "invalid coremask\n");
 			return -1;
@@ -782,6 +819,7 @@ eal_parse_common_option(int opt, const char *optarg,
 		break;
 	/* corelist */
 	case 'l':
+		param_corelist = 1;
 		if (eal_parse_corelist(optarg) < 0) {
 			RTE_LOG(ERR, EAL, "invalid core list\n");
 			return -1;
@@ -890,12 +928,25 @@ eal_parse_common_option(int opt, const char *optarg,
 		break;
 	}
 	case OPT_LCORES_NUM:
+		param_coremap = 1;
 		if (eal_parse_lcores(optarg) < 0) {
 			RTE_LOG(ERR, EAL, "invalid parameter for --"
 				OPT_LCORES "\n");
 			return -1;
 		}
 		break;
+	case OPT_AVAIL_CORES_NUM:
+		if (param_coremask || param_corelist || param_coremap) {
+			RTE_LOG(ERR, EAL, "should put --" OPT_AVAIL_CORES
+				" before -c, -l and --" OPT_LCORES "\n");
+			return -1;
+		}
+		if (eal_parse_avail_cores() < 0) {
+			RTE_LOG(ERR, EAL, "failed to use --"
+				OPT_AVAIL_CORES "\n");
+			return -1;
+		}
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -990,6 +1041,7 @@ eal_common_usage(void)
 	       "                      ',' is used for single number separator.\n"
 	       "                      '( )' can be omitted for single element group,\n"
 	       "                      '@' can be omitted if cpus and lcores have the same value\n"
+	       "  --"OPT_AVAIL_CORES"       Use pthread_getaffinity_np() to detect cores to be used\n"
 	       "  --"OPT_MASTER_LCORE" ID   Core ID that is used as master\n"
 	       "  -n CHANNELS         Number of memory channels\n"
 	       "  -m MB               Memory to allocate (see also --"OPT_SOCKET_MEM")\n"
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..b2ddea3 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#define OPT_AVAIL_CORES       "avail-cores"
+	OPT_AVAIL_CORES_NUM,
 	OPT_LONG_MAX_NUM
 };
 
-- 
2.1.4

[PATCH] eal: make hugetlb initialization more robust

[Jianfeng Tan]

This patch adds an option, --huge-trybest, to use a recover mechanism to
the case that there are not so many hugepages (declared in sysfs), which
can be used. It relys on a mem access to fault-in hugepages, and if fails
with SIGBUS, recover to previously saved stack environment with
siglongjmp().

Test example:
  a. cgcreate -g hugetlb:/test-subgroup
  b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
  c. cgexec -g hugetlb:test-subgroup \
	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
 lib/librte_eal/common/eal_common_options.c |  4 ++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 +
 lib/librte_eal/linuxapp/eal/eal.c          |  1 +
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 95 +++++++++++++++++++++++++++---
 5 files changed, 95 insertions(+), 8 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 29942ea..8ff6a2e 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+	{OPT_HUGE_TRYBEST,      0, NULL, OPT_HUGE_TRYBEST_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -896,6 +897,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_HUGE_TRYBEST_NUM:
+		internal_config.huge_trybest = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..90a3533 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -64,6 +64,7 @@ struct internal_config {
 	volatile unsigned force_nchannel; /**< force number of channels */
 	volatile unsigned force_nrank;    /**< force number of ranks */
 	volatile unsigned no_hugetlbfs;   /**< true to disable hugetlbfs */
+	volatile unsigned huge_trybest;   /**< try best to allocate hugepages */
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned xen_dom0_support; /**< support app running on Xen Dom0*/
 	volatile unsigned no_pci;         /**< true to disable PCI */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..02397c5 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#define OPT_HUGE_TRYBEST      "huge-trybest"
+	OPT_HUGE_TRYBEST_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index ceac435..3e23877 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -343,6 +343,7 @@ eal_usage(const char *prgname)
 	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
 	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
 	       "  --"OPT_XEN_DOM0"          Support running on Xen dom0 without hugetlbfs\n"
+	       "  --"OPT_HUGE_TRYBEST"      Try best to accommodate hugepages\n"
 	       "\n");
 	/* Allow the application to print its usage message too if hook is set */
 	if ( rte_application_usage_hook ) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 5b9132c..1766d7f 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,12 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf jmpenv;
+
+static void sigbus_handler(int signo __rte_unused)
+{
+	siglongjmp(jmpenv, 1);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
@@ -396,7 +404,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		if (fd < 0) {
 			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		/* map the segment, and populate page tables,
@@ -407,7 +415,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
@@ -417,12 +425,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			hugepg_tbl[i].final_va = virtaddr;
 		}
 
+		if (orig && internal_config.huge_trybest) {
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (sigsetjmp(jmpenv, 0)) {
+				RTE_LOG(ERR, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				unlink(hugepg_tbl[i].filepath);
+				return i;
+			}
+			*(int *)virtaddr = 0;
+		}
+
+
 		/* set shared flock on the file. */
 		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
 			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -430,7 +459,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1036,6 +1065,33 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static struct sigaction action_old;
+static int need_recover;
+
+static void
+register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = sigbus_handler;
+
+	need_recover = !sigaction(SIGBUS, &action, &action_old);
+}
+
+static void
+recover_sigbus(void)
+{
+	if (need_recover) {
+		sigaction(SIGBUS, &action_old, NULL);
+		need_recover = 0;
+	}
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1122,8 +1178,12 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	if (internal_config.huge_trybest)
+		register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		int pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1137,10 +1197,24 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
-			goto fail;
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+			if (internal_config.huge_trybest) {
+				int pages = pages_old - pages_new;
+
+				internal_config.memory -=
+					hpi->hugepage_sz * pages;
+				nr_hugepages -= pages;
+				hpi->num_pages[0] = pages_new;
+				if (pages_new == 0)
+					continue;
+			} else
+				goto fail;
 		}
 
 		/* find physical addresses and sockets for each hugepage */
@@ -1187,6 +1261,9 @@ rte_eal_hugepage_init(void)
 #endif
 	}
 
+	if (internal_config.huge_trybest)
+		recover_sigbus();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
@@ -1373,6 +1450,8 @@ rte_eal_hugepage_init(void)
 	return 0;
 
 fail:
+	if (internal_config.huge_trybest)
+		recover_sigbus();
 	free(tmp_hp);
 	return -1;
 }
-- 
2.1.4

[PATCH v2] eal: make hugetlb initialization more robust

[Jianfeng Tan]

This patch adds an option, --huge-trybest, to use a recover mechanism to
the case that there are not so many hugepages (declared in sysfs), which
can be used. It relys on a mem access to fault-in hugepages, and if fails
with SIGBUS, recover to previously saved stack environment with
siglongjmp().

Test example:
  a. cgcreate -g hugetlb:/test-subgroup
  b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
  c. cgexec -g hugetlb:test-subgroup \
	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
v2:
 - Address the compiling error by move setjmp into a wrap method.

 lib/librte_eal/common/eal_common_options.c |   4 ++
 lib/librte_eal/common/eal_internal_cfg.h   |   1 +
 lib/librte_eal/common/eal_options.h        |   2 +
 lib/librte_eal/linuxapp/eal/eal.c          |   1 +
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 104 ++++++++++++++++++++++++++---
 5 files changed, 104 insertions(+), 8 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 29942ea..8ff6a2e 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+	{OPT_HUGE_TRYBEST,      0, NULL, OPT_HUGE_TRYBEST_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -896,6 +897,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_HUGE_TRYBEST_NUM:
+		internal_config.huge_trybest = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..90a3533 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -64,6 +64,7 @@ struct internal_config {
 	volatile unsigned force_nchannel; /**< force number of channels */
 	volatile unsigned force_nrank;    /**< force number of ranks */
 	volatile unsigned no_hugetlbfs;   /**< true to disable hugetlbfs */
+	volatile unsigned huge_trybest;   /**< try best to allocate hugepages */
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned xen_dom0_support; /**< support app running on Xen Dom0*/
 	volatile unsigned no_pci;         /**< true to disable PCI */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..02397c5 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#define OPT_HUGE_TRYBEST      "huge-trybest"
+	OPT_HUGE_TRYBEST_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index ceac435..3e23877 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -343,6 +343,7 @@ eal_usage(const char *prgname)
 	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
 	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
 	       "  --"OPT_XEN_DOM0"          Support running on Xen dom0 without hugetlbfs\n"
+	       "  --"OPT_HUGE_TRYBEST"      Try best to accommodate hugepages\n"
 	       "\n");
 	/* Allow the application to print its usage message too if hook is set */
 	if ( rte_application_usage_hook ) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 5b9132c..e4e1f3b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,21 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf jmpenv;
+
+static void sigbus_handler(int signo __rte_unused)
+{
+	siglongjmp(jmpenv, 1);
+}
+
+/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
+ * non-static local variable in the stack frame calling setjmp might be
+ * clobbered by a call to longjmp.
+ */
+static int wrap_setjmp(void)
+{
+	return setjmp(jmpenv);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
@@ -396,7 +413,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		if (fd < 0) {
 			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		/* map the segment, and populate page tables,
@@ -407,7 +424,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
@@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			hugepg_tbl[i].final_va = virtaddr;
 		}
 
+		if (orig && internal_config.huge_trybest) {
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (wrap_setjmp()) {
+				RTE_LOG(ERR, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				unlink(hugepg_tbl[i].filepath);
+				return i;
+			}
+			*(int *)virtaddr = 0;
+		}
+
+
 		/* set shared flock on the file. */
 		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
 			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -430,7 +468,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1036,6 +1074,33 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static struct sigaction action_old;
+static int need_recover;
+
+static void
+register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = sigbus_handler;
+
+	need_recover = !sigaction(SIGBUS, &action, &action_old);
+}
+
+static void
+recover_sigbus(void)
+{
+	if (need_recover) {
+		sigaction(SIGBUS, &action_old, NULL);
+		need_recover = 0;
+	}
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1122,8 +1187,12 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	if (internal_config.huge_trybest)
+		register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		int pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1137,10 +1206,24 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
-			goto fail;
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+			if (internal_config.huge_trybest) {
+				int pages = pages_old - pages_new;
+
+				internal_config.memory -=
+					hpi->hugepage_sz * pages;
+				nr_hugepages -= pages;
+				hpi->num_pages[0] = pages_new;
+				if (pages_new == 0)
+					continue;
+			} else
+				goto fail;
 		}
 
 		/* find physical addresses and sockets for each hugepage */
@@ -1187,6 +1270,9 @@ rte_eal_hugepage_init(void)
 #endif
 	}
 
+	if (internal_config.huge_trybest)
+		recover_sigbus();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
@@ -1373,6 +1459,8 @@ rte_eal_hugepage_init(void)
 	return 0;
 
 fail:
+	if (internal_config.huge_trybest)
+		recover_sigbus();
 	free(tmp_hp);
 	return -1;
 }
-- 
2.1.4

Re: [PATCH v2] eal: make hugetlb initialization more robust

[Tan, Jianfeng]

On 3/8/2016 9:42 AM, Jianfeng Tan wrote:
> This patch adds an option, --huge-trybest, to use a recover mechanism to
> the case that there are not so many hugepages (declared in sysfs), which
> can be used. It relys on a mem access to fault-in hugepages, and if fails
> with SIGBUS, recover to previously saved stack environment with
> siglongjmp().
>
> Test example:
>    a. cgcreate -g hugetlb:/test-subgroup
>    b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>    c. cgexec -g hugetlb:test-subgroup \
> 	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest
>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>

Sorry, forgot to add ack from Neil.
Acked-by: Neil Horman <nhorman@tuxdriver.com>

> ---
> v2:
>   - Address the compiling error by move setjmp into a wrap method.
>
>   lib/librte_eal/common/eal_common_options.c |   4 ++
>   lib/librte_eal/common/eal_internal_cfg.h   |   1 +
>   lib/librte_eal/common/eal_options.h        |   2 +
>   lib/librte_eal/linuxapp/eal/eal.c          |   1 +
>   lib/librte_eal/linuxapp/eal/eal_memory.c   | 104 ++++++++++++++++++++++++++---
>   5 files changed, 104 insertions(+), 8 deletions(-)
>
> diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
> index 29942ea..8ff6a2e 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -95,6 +95,7 @@ eal_long_options[] = {
>   	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
>   	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
>   	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
> +	{OPT_HUGE_TRYBEST,      0, NULL, OPT_HUGE_TRYBEST_NUM     },
>   	{0,                     0, NULL, 0                        }
>   };
>   
> @@ -896,6 +897,9 @@ eal_parse_common_option(int opt, const char *optarg,
>   			return -1;
>   		}
>   		break;
> +	case OPT_HUGE_TRYBEST_NUM:
> +		internal_config.huge_trybest = 1;
> +		break;
>   
>   	/* don't know what to do, leave this to caller */
>   	default:
> diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
> index 5f1367e..90a3533 100644
> --- a/lib/librte_eal/common/eal_internal_cfg.h
> +++ b/lib/librte_eal/common/eal_internal_cfg.h
> @@ -64,6 +64,7 @@ struct internal_config {
>   	volatile unsigned force_nchannel; /**< force number of channels */
>   	volatile unsigned force_nrank;    /**< force number of ranks */
>   	volatile unsigned no_hugetlbfs;   /**< true to disable hugetlbfs */
> +	volatile unsigned huge_trybest;   /**< try best to allocate hugepages */
>   	unsigned hugepage_unlink;         /**< true to unlink backing files */
>   	volatile unsigned xen_dom0_support; /**< support app running on Xen Dom0*/
>   	volatile unsigned no_pci;         /**< true to disable PCI */
> diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
> index a881c62..02397c5 100644
> --- a/lib/librte_eal/common/eal_options.h
> +++ b/lib/librte_eal/common/eal_options.h
> @@ -83,6 +83,8 @@ enum {
>   	OPT_VMWARE_TSC_MAP_NUM,
>   #define OPT_XEN_DOM0          "xen-dom0"
>   	OPT_XEN_DOM0_NUM,
> +#define OPT_HUGE_TRYBEST      "huge-trybest"
> +	OPT_HUGE_TRYBEST_NUM,
>   	OPT_LONG_MAX_NUM
>   };
>   
> diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
> index ceac435..3e23877 100644
> --- a/lib/librte_eal/linuxapp/eal/eal.c
> +++ b/lib/librte_eal/linuxapp/eal/eal.c
> @@ -343,6 +343,7 @@ eal_usage(const char *prgname)
>   	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
>   	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
>   	       "  --"OPT_XEN_DOM0"          Support running on Xen dom0 without hugetlbfs\n"
> +	       "  --"OPT_HUGE_TRYBEST"      Try best to accommodate hugepages\n"
>   	       "\n");
>   	/* Allow the application to print its usage message too if hook is set */
>   	if ( rte_application_usage_hook ) {
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 5b9132c..e4e1f3b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -80,6 +80,8 @@
>   #include <errno.h>
>   #include <sys/ioctl.h>
>   #include <sys/time.h>
> +#include <signal.h>
> +#include <setjmp.h>
>   
>   #include <rte_log.h>
>   #include <rte_memory.h>
> @@ -309,6 +311,21 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
>   	return addr;
>   }
>   
> +static sigjmp_buf jmpenv;
> +
> +static void sigbus_handler(int signo __rte_unused)
> +{
> +	siglongjmp(jmpenv, 1);
> +}
> +
> +/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
> + * non-static local variable in the stack frame calling setjmp might be
> + * clobbered by a call to longjmp.
> + */
> +static int wrap_setjmp(void)
> +{
> +	return setjmp(jmpenv);
> +}
>   /*
>    * Mmap all hugepages of hugepage table: it first open a file in
>    * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
> @@ -396,7 +413,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   		if (fd < 0) {
>   			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
>   					strerror(errno));
> -			return -1;
> +			return i;
>   		}
>   
>   		/* map the segment, and populate page tables,
> @@ -407,7 +424,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
>   					strerror(errno));
>   			close(fd);
> -			return -1;
> +			return i;
>   		}
>   
>   		if (orig) {
> @@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   			hugepg_tbl[i].final_va = virtaddr;
>   		}
>   
> +		if (orig && internal_config.huge_trybest) {
> +			/* In linux, hugetlb limitations, like cgroup, are
> +			 * enforced at fault time instead of mmap(), even
> +			 * with the option of MAP_POPULATE. Kernel will send
> +			 * a SIGBUS signal. To avoid to be killed, save stack
> +			 * environment here, if SIGBUS happens, we can jump
> +			 * back here.
> +			 */
> +			if (wrap_setjmp()) {
> +				RTE_LOG(ERR, EAL, "SIGBUS: Cannot mmap more "
> +					"hugepages of size %u MB\n",
> +					(unsigned)(hugepage_sz / 0x100000));
> +				munmap(virtaddr, hugepage_sz);
> +				close(fd);
> +				unlink(hugepg_tbl[i].filepath);
> +				return i;
> +			}
> +			*(int *)virtaddr = 0;
> +		}
> +
> +
>   		/* set shared flock on the file. */
>   		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
>   			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
>   				__func__, strerror(errno));
>   			close(fd);
> -			return -1;
> +			return i;
>   		}
>   
>   		close(fd);
> @@ -430,7 +468,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   		vma_addr = (char *)vma_addr + hugepage_sz;
>   		vma_len -= hugepage_sz;
>   	}
> -	return 0;
> +	return i;
>   }
>   
>   #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
> @@ -1036,6 +1074,33 @@ calc_num_pages_per_socket(uint64_t * memory,
>   	return total_num_pages;
>   }
>   
> +static struct sigaction action_old;
> +static int need_recover;
> +
> +static void
> +register_sigbus(void)
> +{
> +	sigset_t mask;
> +	struct sigaction action;
> +
> +	sigemptyset(&mask);
> +	sigaddset(&mask, SIGBUS);
> +	action.sa_flags = 0;
> +	action.sa_mask = mask;
> +	action.sa_handler = sigbus_handler;
> +
> +	need_recover = !sigaction(SIGBUS, &action, &action_old);
> +}
> +
> +static void
> +recover_sigbus(void)
> +{
> +	if (need_recover) {
> +		sigaction(SIGBUS, &action_old, NULL);
> +		need_recover = 0;
> +	}
> +}
> +
>   /*
>    * Prepare physical memory mapping: fill configuration structure with
>    * these infos, return 0 on success.
> @@ -1122,8 +1187,12 @@ rte_eal_hugepage_init(void)
>   
>   	hp_offset = 0; /* where we start the current page size entries */
>   
> +	if (internal_config.huge_trybest)
> +		register_sigbus();
> +
>   	/* map all hugepages and sort them */
>   	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
> +		int pages_old, pages_new;
>   		struct hugepage_info *hpi;
>   
>   		/*
> @@ -1137,10 +1206,24 @@ rte_eal_hugepage_init(void)
>   			continue;
>   
>   		/* map all hugepages available */
> -		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
> -			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
> -					(unsigned)(hpi->hugepage_sz / 0x100000));
> -			goto fail;
> +		pages_old = hpi->num_pages[0];
> +		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
> +		if (pages_new < pages_old) {
> +			RTE_LOG(DEBUG, EAL,
> +				"%d not %d hugepages of size %u MB allocated\n",
> +				pages_new, pages_old,
> +				(unsigned)(hpi->hugepage_sz / 0x100000));
> +			if (internal_config.huge_trybest) {
> +				int pages = pages_old - pages_new;
> +
> +				internal_config.memory -=
> +					hpi->hugepage_sz * pages;
> +				nr_hugepages -= pages;
> +				hpi->num_pages[0] = pages_new;
> +				if (pages_new == 0)
> +					continue;
> +			} else
> +				goto fail;
>   		}
>   
>   		/* find physical addresses and sockets for each hugepage */
> @@ -1187,6 +1270,9 @@ rte_eal_hugepage_init(void)
>   #endif
>   	}
>   
> +	if (internal_config.huge_trybest)
> +		recover_sigbus();
> +
>   #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
>   	nr_hugefiles = 0;
>   	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
> @@ -1373,6 +1459,8 @@ rte_eal_hugepage_init(void)
>   	return 0;
>   
>   fail:
> +	if (internal_config.huge_trybest)
> +		recover_sigbus();
>   	free(tmp_hp);
>   	return -1;
>   }

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Panu Matilainen]

On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
> This patch adds option, --avail-cores, to use lcores which are available
> by calling pthread_getaffinity_np() to narrow down detected cores before
> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>
> Test example:
> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> 		--avail-cores -m 1024
>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> Acked-by: Neil Horman <nhorman@tuxdriver.com>

Hmm, to me this sounds like something that should be done always so 
there's no need for an option. Or if there's a chance it might do the 
wrong thing in some rare circumstance then perhaps there should be a 
disabler option instead?

Or am I just missing something?

	- Panu -

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

Hi Panu,

On 3/8/2016 4:54 PM, Panu Matilainen wrote:
> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>> This patch adds option, --avail-cores, to use lcores which are available
>> by calling pthread_getaffinity_np() to narrow down detected cores before
>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>
>> Test example:
>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>         --avail-cores -m 1024
>>
>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>
> Hmm, to me this sounds like something that should be done always so 
> there's no need for an option. Or if there's a chance it might do the 
> wrong thing in some rare circumstance then perhaps there should be a 
> disabler option instead?

Thanks for comments.

Yes, there's a use case that we cannot handle.

If we make it as default, DPDK applications may fail to start, when user 
specifies a core in isolcpus and its parent process (say bash) has a 
cpuset affinity that excludes isolcpus. Originally, DPDK applications 
just blindly do pthread_setaffinity_np() and it always succeeds because 
it always has root privilege to change any cpu affinity.

Now, if we do the checking in rte_eal_cpu_init(), those lcores will be 
flagged as undetected (in my older implementation) and leads to failure. 
To make it correct, we would always add "taskset mask" (or other ways) 
before DPDK application cmd lines.

How do you think?

Thanks,
Jianfeng

>
> Or am I just missing something?
>
>     - Panu -
>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Panu Matilainen]

On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
> Hi Panu,
>
> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>> This patch adds option, --avail-cores, to use lcores which are available
>>> by calling pthread_getaffinity_np() to narrow down detected cores before
>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>
>>> Test example:
>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>         --avail-cores -m 1024
>>>
>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>
>> Hmm, to me this sounds like something that should be done always so
>> there's no need for an option. Or if there's a chance it might do the
>> wrong thing in some rare circumstance then perhaps there should be a
>> disabler option instead?
>
> Thanks for comments.
>
> Yes, there's a use case that we cannot handle.
>
> If we make it as default, DPDK applications may fail to start, when user
> specifies a core in isolcpus and its parent process (say bash) has a
> cpuset affinity that excludes isolcpus. Originally, DPDK applications
> just blindly do pthread_setaffinity_np() and it always succeeds because
> it always has root privilege to change any cpu affinity.
>
> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
> flagged as undetected (in my older implementation) and leads to failure.
> To make it correct, we would always add "taskset mask" (or other ways)
> before DPDK application cmd lines.
>
> How do you think?

I still think it sounds like something that should be done by default 
and maybe be overridable with some flag, rather than the other way 
around. Another alternative might be detecting the cores always but if 
running as root, override but with a warning.

But I dont know, just wondering. To look at it from another angle: why 
would somebody use this new --avail-cores option and in what situation, 
if things "just work" otherwise anyway?

	- Panu -

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

On 3/9/2016 9:05 PM, Panu Matilainen wrote:
> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
>> Hi Panu,
>>
>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>> This patch adds option, --avail-cores, to use lcores which are 
>>>> available
>>>> by calling pthread_getaffinity_np() to narrow down detected cores 
>>>> before
>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>
>>>> Test example:
>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>         --avail-cores -m 1024
>>>>
>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>
>>> Hmm, to me this sounds like something that should be done always so
>>> there's no need for an option. Or if there's a chance it might do the
>>> wrong thing in some rare circumstance then perhaps there should be a
>>> disabler option instead?
>>
>> Thanks for comments.
>>
>> Yes, there's a use case that we cannot handle.
>>
>> If we make it as default, DPDK applications may fail to start, when user
>> specifies a core in isolcpus and its parent process (say bash) has a
>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>> just blindly do pthread_setaffinity_np() and it always succeeds because
>> it always has root privilege to change any cpu affinity.
>>
>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>> flagged as undetected (in my older implementation) and leads to failure.
>> To make it correct, we would always add "taskset mask" (or other ways)
>> before DPDK application cmd lines.
>>
>> How do you think?
>
> I still think it sounds like something that should be done by default 
> and maybe be overridable with some flag, rather than the other way 
> around. Another alternative might be detecting the cores always but if 
> running as root, override but with a warning.

For your second solution, only root can setaffinity to isolcpus?
Your first solution seems like a promising way for me.

>
> But I dont know, just wondering. To look at it from another angle: why 
> would somebody use this new --avail-cores option and in what 
> situation, if things "just work" otherwise anyway?

For DPDK applications, the most common case to initialize DPDK is like 
this: "$dpdk-app [options for DPDK] -- [options for app]", so users need 
to specify which cores to run and how much hugepages are used. Suppose 
we need this dpdk-app to run in a container, users already give those 
information when they build up the cgroup for it to run inside, this 
option or this patch is to make DPDK more smart to discover how much 
resource will be used. Make sense?

Thanks,
Jianfeng


>
>     - Panu -
>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Ananyev, Konstantin]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng
> Sent: Wednesday, March 09, 2016 1:53 PM
> To: Panu Matilainen; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
> 
> 
> 
> On 3/9/2016 9:05 PM, Panu Matilainen wrote:
> > On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
> >> Hi Panu,
> >>
> >> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
> >>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
> >>>> This patch adds option, --avail-cores, to use lcores which are
> >>>> available
> >>>> by calling pthread_getaffinity_np() to narrow down detected cores
> >>>> before
> >>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
> >>>>
> >>>> Test example:
> >>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> >>>>         --avail-cores -m 1024
> >>>>
> >>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> >>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> >>>
> >>> Hmm, to me this sounds like something that should be done always so
> >>> there's no need for an option. Or if there's a chance it might do the
> >>> wrong thing in some rare circumstance then perhaps there should be a
> >>> disabler option instead?
> >>
> >> Thanks for comments.
> >>
> >> Yes, there's a use case that we cannot handle.
> >>
> >> If we make it as default, DPDK applications may fail to start, when user
> >> specifies a core in isolcpus and its parent process (say bash) has a
> >> cpuset affinity that excludes isolcpus. Originally, DPDK applications
> >> just blindly do pthread_setaffinity_np() and it always succeeds because
> >> it always has root privilege to change any cpu affinity.
> >>
> >> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
> >> flagged as undetected (in my older implementation) and leads to failure.
> >> To make it correct, we would always add "taskset mask" (or other ways)
> >> before DPDK application cmd lines.
> >>
> >> How do you think?
> >
> > I still think it sounds like something that should be done by default
> > and maybe be overridable with some flag, rather than the other way
> > around. Another alternative might be detecting the cores always but if
> > running as root, override but with a warning.
> 
> For your second solution, only root can setaffinity to isolcpus?
> Your first solution seems like a promising way for me.
> 
> >
> > But I dont know, just wondering. To look at it from another angle: why
> > would somebody use this new --avail-cores option and in what
> > situation, if things "just work" otherwise anyway?
> 
> For DPDK applications, the most common case to initialize DPDK is like
> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
> to specify which cores to run and how much hugepages are used. Suppose
> we need this dpdk-app to run in a container, users already give those
> information when they build up the cgroup for it to run inside, this
> option or this patch is to make DPDK more smart to discover how much
> resource will be used. Make sense?

But then, all we need might be just a script that would extract this information from the system
and form a proper cmdline parameter for DPDK? 
Konstantin

> 
> Thanks,
> Jianfeng
> 
> 
> >
> >     - Panu -
> >

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

On 3/9/2016 10:01 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng
>> Sent: Wednesday, March 09, 2016 1:53 PM
>> To: Panu Matilainen; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
>>
>>
>>
>> On 3/9/2016 9:05 PM, Panu Matilainen wrote:
>>> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
>>>> Hi Panu,
>>>>
>>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>>>> This patch adds option, --avail-cores, to use lcores which are
>>>>>> available
>>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
>>>>>> before
>>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>>>
>>>>>> Test example:
>>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>>>          --avail-cores -m 1024
>>>>>>
>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>>> Hmm, to me this sounds like something that should be done always so
>>>>> there's no need for an option. Or if there's a chance it might do the
>>>>> wrong thing in some rare circumstance then perhaps there should be a
>>>>> disabler option instead?
>>>> Thanks for comments.
>>>>
>>>> Yes, there's a use case that we cannot handle.
>>>>
>>>> If we make it as default, DPDK applications may fail to start, when user
>>>> specifies a core in isolcpus and its parent process (say bash) has a
>>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>>>> just blindly do pthread_setaffinity_np() and it always succeeds because
>>>> it always has root privilege to change any cpu affinity.
>>>>
>>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>>>> flagged as undetected (in my older implementation) and leads to failure.
>>>> To make it correct, we would always add "taskset mask" (or other ways)
>>>> before DPDK application cmd lines.
>>>>
>>>> How do you think?
>>> I still think it sounds like something that should be done by default
>>> and maybe be overridable with some flag, rather than the other way
>>> around. Another alternative might be detecting the cores always but if
>>> running as root, override but with a warning.
>> For your second solution, only root can setaffinity to isolcpus?
>> Your first solution seems like a promising way for me.
>>
>>> But I dont know, just wondering. To look at it from another angle: why
>>> would somebody use this new --avail-cores option and in what
>>> situation, if things "just work" otherwise anyway?
>> For DPDK applications, the most common case to initialize DPDK is like
>> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
>> to specify which cores to run and how much hugepages are used. Suppose
>> we need this dpdk-app to run in a container, users already give those
>> information when they build up the cgroup for it to run inside, this
>> option or this patch is to make DPDK more smart to discover how much
>> resource will be used. Make sense?
> But then, all we need might be just a script that would extract this information from the system
> and form a proper cmdline parameter for DPDK?

Yes, a script will work. Or to construct (argc, argv) to call 
rte_eal_init() in the application. But as Neil Horman once suggested, a 
simple pthread_getaffinity_np() will get all things done. So if it worth 
a patch here?

Thanks,
Jianfeng

> Konstantin
>
>> Thanks,
>> Jianfeng
>>
>>
>>>      - Panu -
>>>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Ananyev, Konstantin]

> -----Original Message-----
> From: Tan, Jianfeng
> Sent: Wednesday, March 09, 2016 2:17 PM
> To: Ananyev, Konstantin; Panu Matilainen; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
> 
> 
> 
> On 3/9/2016 10:01 PM, Ananyev, Konstantin wrote:
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng
> >> Sent: Wednesday, March 09, 2016 1:53 PM
> >> To: Panu Matilainen; dev@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
> >>
> >>
> >>
> >> On 3/9/2016 9:05 PM, Panu Matilainen wrote:
> >>> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
> >>>> Hi Panu,
> >>>>
> >>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
> >>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
> >>>>>> This patch adds option, --avail-cores, to use lcores which are
> >>>>>> available
> >>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
> >>>>>> before
> >>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
> >>>>>>
> >>>>>> Test example:
> >>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> >>>>>>          --avail-cores -m 1024
> >>>>>>
> >>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> >>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> >>>>> Hmm, to me this sounds like something that should be done always so
> >>>>> there's no need for an option. Or if there's a chance it might do the
> >>>>> wrong thing in some rare circumstance then perhaps there should be a
> >>>>> disabler option instead?
> >>>> Thanks for comments.
> >>>>
> >>>> Yes, there's a use case that we cannot handle.
> >>>>
> >>>> If we make it as default, DPDK applications may fail to start, when user
> >>>> specifies a core in isolcpus and its parent process (say bash) has a
> >>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
> >>>> just blindly do pthread_setaffinity_np() and it always succeeds because
> >>>> it always has root privilege to change any cpu affinity.
> >>>>
> >>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
> >>>> flagged as undetected (in my older implementation) and leads to failure.
> >>>> To make it correct, we would always add "taskset mask" (or other ways)
> >>>> before DPDK application cmd lines.
> >>>>
> >>>> How do you think?
> >>> I still think it sounds like something that should be done by default
> >>> and maybe be overridable with some flag, rather than the other way
> >>> around. Another alternative might be detecting the cores always but if
> >>> running as root, override but with a warning.
> >> For your second solution, only root can setaffinity to isolcpus?
> >> Your first solution seems like a promising way for me.
> >>
> >>> But I dont know, just wondering. To look at it from another angle: why
> >>> would somebody use this new --avail-cores option and in what
> >>> situation, if things "just work" otherwise anyway?
> >> For DPDK applications, the most common case to initialize DPDK is like
> >> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
> >> to specify which cores to run and how much hugepages are used. Suppose
> >> we need this dpdk-app to run in a container, users already give those
> >> information when they build up the cgroup for it to run inside, this
> >> option or this patch is to make DPDK more smart to discover how much
> >> resource will be used. Make sense?
> > But then, all we need might be just a script that would extract this information from the system
> > and form a proper cmdline parameter for DPDK?
> 
> Yes, a script will work. Or to construct (argc, argv) to call
> rte_eal_init() in the application. But as Neil Horman once suggested, a
> simple pthread_getaffinity_np() will get all things done. So if it worth
> a patch here?

Don't know...
Personally I would prefer not to put extra logic inside EAL.
For me - there are too many different options already.
From other side looking at the patch itself:
You are updating lcore_count and lcore_config[],based on physical cpu availability,
but these days it is not always one-to-one mapping between EAL lcore and physical cpu. 
Shouldn't that be taken into account?
Konstantin
 

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

Hi Konstantin,

On 3/9/2016 10:44 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: Tan, Jianfeng
>> Sent: Wednesday, March 09, 2016 2:17 PM
>> To: Ananyev, Konstantin; Panu Matilainen; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
>>
>>
>>
>> On 3/9/2016 10:01 PM, Ananyev, Konstantin wrote:
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng
>>>> Sent: Wednesday, March 09, 2016 1:53 PM
>>>> To: Panu Matilainen; dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
>>>>
>>>>
>>>>
>>>> On 3/9/2016 9:05 PM, Panu Matilainen wrote:
>>>>> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
>>>>>> Hi Panu,
>>>>>>
>>>>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>>>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>>>>>> This patch adds option, --avail-cores, to use lcores which are
>>>>>>>> available
>>>>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
>>>>>>>> before
>>>>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>>>>>
>>>>>>>> Test example:
>>>>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>>>>>           --avail-cores -m 1024
>>>>>>>>
>>>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>>>>> Hmm, to me this sounds like something that should be done always so
>>>>>>> there's no need for an option. Or if there's a chance it might do the
>>>>>>> wrong thing in some rare circumstance then perhaps there should be a
>>>>>>> disabler option instead?
>>>>>> Thanks for comments.
>>>>>>
>>>>>> Yes, there's a use case that we cannot handle.
>>>>>>
>>>>>> If we make it as default, DPDK applications may fail to start, when user
>>>>>> specifies a core in isolcpus and its parent process (say bash) has a
>>>>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>>>>>> just blindly do pthread_setaffinity_np() and it always succeeds because
>>>>>> it always has root privilege to change any cpu affinity.
>>>>>>
>>>>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>>>>>> flagged as undetected (in my older implementation) and leads to failure.
>>>>>> To make it correct, we would always add "taskset mask" (or other ways)
>>>>>> before DPDK application cmd lines.
>>>>>>
>>>>>> How do you think?
>>>>> I still think it sounds like something that should be done by default
>>>>> and maybe be overridable with some flag, rather than the other way
>>>>> around. Another alternative might be detecting the cores always but if
>>>>> running as root, override but with a warning.
>>>> For your second solution, only root can setaffinity to isolcpus?
>>>> Your first solution seems like a promising way for me.
>>>>
>>>>> But I dont know, just wondering. To look at it from another angle: why
>>>>> would somebody use this new --avail-cores option and in what
>>>>> situation, if things "just work" otherwise anyway?
>>>> For DPDK applications, the most common case to initialize DPDK is like
>>>> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
>>>> to specify which cores to run and how much hugepages are used. Suppose
>>>> we need this dpdk-app to run in a container, users already give those
>>>> information when they build up the cgroup for it to run inside, this
>>>> option or this patch is to make DPDK more smart to discover how much
>>>> resource will be used. Make sense?
>>> But then, all we need might be just a script that would extract this information from the system
>>> and form a proper cmdline parameter for DPDK?
>> Yes, a script will work. Or to construct (argc, argv) to call
>> rte_eal_init() in the application. But as Neil Horman once suggested, a
>> simple pthread_getaffinity_np() will get all things done. So if it worth
>> a patch here?
> Don't know...
> Personally I would prefer not to put extra logic inside EAL.
> For me - there are too many different options already.

Then how about make it default in rte_eal_cpu_init()? And it is already 
known it will bring trouble to those use isolcpus users, they need to 
add "taskset [mask]" before starting a DPDK app.

>  From other side looking at the patch itself:
> You are updating lcore_count and lcore_config[],based on physical cpu availability,
> but these days it is not always one-to-one mapping between EAL lcore and physical cpu.
> Shouldn't that be taken into account?

I have not see the problem so far, because this work is done before 
parsing coremask (-c), corelist (-l), and coremap (--lcores). If a core 
is disabled here, it's like it is not detected in rte_eal_cpu_init(). Or 
could you please give more hints?

Thanks,
Jianfeng

> Konstantin
>   
>
>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Ananyev, Konstantin]

Hi Jianfeng,

> -----Original Message-----
> From: Tan, Jianfeng
> Sent: Wednesday, March 09, 2016 2:56 PM
> To: Ananyev, Konstantin; Panu Matilainen; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
> 
> Hi Konstantin,
> 
> On 3/9/2016 10:44 PM, Ananyev, Konstantin wrote:
> >
> >> -----Original Message-----
> >> From: Tan, Jianfeng
> >> Sent: Wednesday, March 09, 2016 2:17 PM
> >> To: Ananyev, Konstantin; Panu Matilainen; dev@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
> >>
> >>
> >>
> >> On 3/9/2016 10:01 PM, Ananyev, Konstantin wrote:
> >>>> -----Original Message-----
> >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng
> >>>> Sent: Wednesday, March 09, 2016 1:53 PM
> >>>> To: Panu Matilainen; dev@dpdk.org
> >>>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
> >>>>
> >>>>
> >>>>
> >>>> On 3/9/2016 9:05 PM, Panu Matilainen wrote:
> >>>>> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
> >>>>>> Hi Panu,
> >>>>>>
> >>>>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
> >>>>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
> >>>>>>>> This patch adds option, --avail-cores, to use lcores which are
> >>>>>>>> available
> >>>>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
> >>>>>>>> before
> >>>>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
> >>>>>>>>
> >>>>>>>> Test example:
> >>>>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> >>>>>>>>           --avail-cores -m 1024
> >>>>>>>>
> >>>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> >>>>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> >>>>>>> Hmm, to me this sounds like something that should be done always so
> >>>>>>> there's no need for an option. Or if there's a chance it might do the
> >>>>>>> wrong thing in some rare circumstance then perhaps there should be a
> >>>>>>> disabler option instead?
> >>>>>> Thanks for comments.
> >>>>>>
> >>>>>> Yes, there's a use case that we cannot handle.
> >>>>>>
> >>>>>> If we make it as default, DPDK applications may fail to start, when user
> >>>>>> specifies a core in isolcpus and its parent process (say bash) has a
> >>>>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
> >>>>>> just blindly do pthread_setaffinity_np() and it always succeeds because
> >>>>>> it always has root privilege to change any cpu affinity.
> >>>>>>
> >>>>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
> >>>>>> flagged as undetected (in my older implementation) and leads to failure.
> >>>>>> To make it correct, we would always add "taskset mask" (or other ways)
> >>>>>> before DPDK application cmd lines.
> >>>>>>
> >>>>>> How do you think?
> >>>>> I still think it sounds like something that should be done by default
> >>>>> and maybe be overridable with some flag, rather than the other way
> >>>>> around. Another alternative might be detecting the cores always but if
> >>>>> running as root, override but with a warning.
> >>>> For your second solution, only root can setaffinity to isolcpus?
> >>>> Your first solution seems like a promising way for me.
> >>>>
> >>>>> But I dont know, just wondering. To look at it from another angle: why
> >>>>> would somebody use this new --avail-cores option and in what
> >>>>> situation, if things "just work" otherwise anyway?
> >>>> For DPDK applications, the most common case to initialize DPDK is like
> >>>> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
> >>>> to specify which cores to run and how much hugepages are used. Suppose
> >>>> we need this dpdk-app to run in a container, users already give those
> >>>> information when they build up the cgroup for it to run inside, this
> >>>> option or this patch is to make DPDK more smart to discover how much
> >>>> resource will be used. Make sense?
> >>> But then, all we need might be just a script that would extract this information from the system
> >>> and form a proper cmdline parameter for DPDK?
> >> Yes, a script will work. Or to construct (argc, argv) to call
> >> rte_eal_init() in the application. But as Neil Horman once suggested, a
> >> simple pthread_getaffinity_np() will get all things done. So if it worth
> >> a patch here?
> > Don't know...
> > Personally I would prefer not to put extra logic inside EAL.
> > For me - there are too many different options already.
> 
> Then how about make it default in rte_eal_cpu_init()? And it is already
> known it will bring trouble to those use isolcpus users, they need to
> add "taskset [mask]" before starting a DPDK app.

As I said - provide a script?
Same might be for amount of hugepage memory available to the user? 

> 
> >  From other side looking at the patch itself:
> > You are updating lcore_count and lcore_config[],based on physical cpu availability,
> > but these days it is not always one-to-one mapping between EAL lcore and physical cpu.
> > Shouldn't that be taken into account?
> 
> I have not see the problem so far, because this work is done before
> parsing coremask (-c), corelist (-l), and coremap (--lcores). If a core
> is disabled here, it's like it is not detected in rte_eal_cpu_init(). Or
> could you please give more hints?

I didn't test try changes, so probably I am missing something.
Let say iuser allowed to use only cpus 0-3.
If he would type with:
 --avail-cores  --lcores='(1-7)@2',
then only lcores 1-3 would be started.
Again if user would specify '2@(1-7)' it would also be undetected
that cpus 4-7 are note available to the user. 
Is that so?

Konstantin

> 
> Thanks,
> Jianfeng
> 
> > Konstantin
> >
> >
> >

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

Hi Konstantin,

On 3/9/2016 11:17 PM, Ananyev, Konstantin wrote:
> Hi Jianfeng,
>
>> -----Original Message-----
>> From: Tan, Jianfeng
>> Sent: Wednesday, March 09, 2016 2:56 PM
>> To: Ananyev, Konstantin; Panu Matilainen; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
>>
>> Hi Konstantin,
>>
>> On 3/9/2016 10:44 PM, Ananyev, Konstantin wrote:
>>>> -----Original Message-----
>>>> From: Tan, Jianfeng
>>>> Sent: Wednesday, March 09, 2016 2:17 PM
>>>> To: Ananyev, Konstantin; Panu Matilainen; dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
>>>>
>>>>
>>>>
>>>> On 3/9/2016 10:01 PM, Ananyev, Konstantin wrote:
>>>>>> -----Original Message-----
>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng
>>>>>> Sent: Wednesday, March 09, 2016 1:53 PM
>>>>>> To: Panu Matilainen; dev@dpdk.org
>>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: add option --avail-cores to detect lcores
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 3/9/2016 9:05 PM, Panu Matilainen wrote:
>>>>>>> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
>>>>>>>> Hi Panu,
>>>>>>>>
>>>>>>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>>>>>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>>>>>>>> This patch adds option, --avail-cores, to use lcores which are
>>>>>>>>>> available
>>>>>>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
>>>>>>>>>> before
>>>>>>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>>>>>>>
>>>>>>>>>> Test example:
>>>>>>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>>>>>>>            --avail-cores -m 1024
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>>>>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>>>>>>> Hmm, to me this sounds like something that should be done always so
>>>>>>>>> there's no need for an option. Or if there's a chance it might do the
>>>>>>>>> wrong thing in some rare circumstance then perhaps there should be a
>>>>>>>>> disabler option instead?
>>>>>>>> Thanks for comments.
>>>>>>>>
>>>>>>>> Yes, there's a use case that we cannot handle.
>>>>>>>>
>>>>>>>> If we make it as default, DPDK applications may fail to start, when user
>>>>>>>> specifies a core in isolcpus and its parent process (say bash) has a
>>>>>>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>>>>>>>> just blindly do pthread_setaffinity_np() and it always succeeds because
>>>>>>>> it always has root privilege to change any cpu affinity.
>>>>>>>>
>>>>>>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>>>>>>>> flagged as undetected (in my older implementation) and leads to failure.
>>>>>>>> To make it correct, we would always add "taskset mask" (or other ways)
>>>>>>>> before DPDK application cmd lines.
>>>>>>>>
>>>>>>>> How do you think?
>>>>>>> I still think it sounds like something that should be done by default
>>>>>>> and maybe be overridable with some flag, rather than the other way
>>>>>>> around. Another alternative might be detecting the cores always but if
>>>>>>> running as root, override but with a warning.
>>>>>> For your second solution, only root can setaffinity to isolcpus?
>>>>>> Your first solution seems like a promising way for me.
>>>>>>
>>>>>>> But I dont know, just wondering. To look at it from another angle: why
>>>>>>> would somebody use this new --avail-cores option and in what
>>>>>>> situation, if things "just work" otherwise anyway?
>>>>>> For DPDK applications, the most common case to initialize DPDK is like
>>>>>> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
>>>>>> to specify which cores to run and how much hugepages are used. Suppose
>>>>>> we need this dpdk-app to run in a container, users already give those
>>>>>> information when they build up the cgroup for it to run inside, this
>>>>>> option or this patch is to make DPDK more smart to discover how much
>>>>>> resource will be used. Make sense?
>>>>> But then, all we need might be just a script that would extract this information from the system
>>>>> and form a proper cmdline parameter for DPDK?
>>>> Yes, a script will work. Or to construct (argc, argv) to call
>>>> rte_eal_init() in the application. But as Neil Horman once suggested, a
>>>> simple pthread_getaffinity_np() will get all things done. So if it worth
>>>> a patch here?
>>> Don't know...
>>> Personally I would prefer not to put extra logic inside EAL.
>>> For me - there are too many different options already.
>> Then how about make it default in rte_eal_cpu_init()? And it is already
>> known it will bring trouble to those use isolcpus users, they need to
>> add "taskset [mask]" before starting a DPDK app.
> As I said - provide a script?

Yes. But what I want to say is this script is hard to be right, if there 
are different kinds of limitations. (Barely happen though :-) )

> Same might be for amount of hugepage memory available to the user?

Ditto. Limitations like hugetlbfs quota, cgroup hugetlb, some are used 
by app themself (more like an artificial argument) ...
>
>>>   From other side looking at the patch itself:
>>> You are updating lcore_count and lcore_config[],based on physical cpu availability,
>>> but these days it is not always one-to-one mapping between EAL lcore and physical cpu.
>>> Shouldn't that be taken into account?
>> I have not see the problem so far, because this work is done before
>> parsing coremask (-c), corelist (-l), and coremap (--lcores). If a core
>> is disabled here, it's like it is not detected in rte_eal_cpu_init(). Or
>> could you please give more hints?
> I didn't test try changes, so probably I am missing something.
> Let say iuser allowed to use only cpus 0-3.
> If he would type with:
>   --avail-cores  --lcores='(1-7)@2',
> then only lcores 1-3 would be started.
> Again if user would specify '2@(1-7)' it would also be undetected
> that cpus 4-7 are note available to the user.
> Is that so?

After reading the code:
For case --lcores='(1-7)@2', lcores 1-7 would be started, and bind to 
pcore 2.
For case --lcores='2@(1-7)', this will fail with "core 4 unavailable".

It's because:
a.  although 1:1 mapping is built-up and flagged as detected if pcore is 
found in sysfs. (ROLE_RTE, cpuset, detected is true)
b. in the beginning of eal_parse_lcores(), "reset lcore config". 
(ROLE_OFF, cpuset is empty, detected is still true)
c. pcore cpuset will be checked by convert_to_cpuset using the previous 
"detected" value.

I have tested it with the patch. Result aligns above analysis.
For case --lcores='(1-7)@2': sudo taskset 0xf 
./examples/helloworld/build/helloworld --avail-cores --lcores='(1-7)@2'
...
hello from core 2
hello from core 3
hello from core 4
hello from core 5
hello from core 6
hello from core 7
hello from core 1

For case --lcores='2@(1-7)': sudo taskset 0xf 
./examples/helloworld/build/helloworld --avail-cores --lcores='2@(1-7)'
...
EAL: core 4 unavailable
EAL: invalid parameter for --lcores
...

One thing may worth mention: shall "detected" be maintained in struct 
lcore_config? Maybe we need to maintain an data structure for pcores?

Thanks,
Jianfeng

>
> Konstantin
>
>> Thanks,
>> Jianfeng
>>
>>> Konstantin
>>>
>>>
>>>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Ananyev, Konstantin]

> >>>>>>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
> >>>>>>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
> >>>>>>>>>> This patch adds option, --avail-cores, to use lcores which are
> >>>>>>>>>> available
> >>>>>>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
> >>>>>>>>>> before
> >>>>>>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
> >>>>>>>>>>
> >>>>>>>>>> Test example:
> >>>>>>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> >>>>>>>>>>            --avail-cores -m 1024
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> >>>>>>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> >>>>>>>>> Hmm, to me this sounds like something that should be done always so
> >>>>>>>>> there's no need for an option. Or if there's a chance it might do the
> >>>>>>>>> wrong thing in some rare circumstance then perhaps there should be a
> >>>>>>>>> disabler option instead?
> >>>>>>>> Thanks for comments.
> >>>>>>>>
> >>>>>>>> Yes, there's a use case that we cannot handle.
> >>>>>>>>
> >>>>>>>> If we make it as default, DPDK applications may fail to start, when user
> >>>>>>>> specifies a core in isolcpus and its parent process (say bash) has a
> >>>>>>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
> >>>>>>>> just blindly do pthread_setaffinity_np() and it always succeeds because
> >>>>>>>> it always has root privilege to change any cpu affinity.
> >>>>>>>>
> >>>>>>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
> >>>>>>>> flagged as undetected (in my older implementation) and leads to failure.
> >>>>>>>> To make it correct, we would always add "taskset mask" (or other ways)
> >>>>>>>> before DPDK application cmd lines.
> >>>>>>>>
> >>>>>>>> How do you think?
> >>>>>>> I still think it sounds like something that should be done by default
> >>>>>>> and maybe be overridable with some flag, rather than the other way
> >>>>>>> around. Another alternative might be detecting the cores always but if
> >>>>>>> running as root, override but with a warning.
> >>>>>> For your second solution, only root can setaffinity to isolcpus?
> >>>>>> Your first solution seems like a promising way for me.
> >>>>>>
> >>>>>>> But I dont know, just wondering. To look at it from another angle: why
> >>>>>>> would somebody use this new --avail-cores option and in what
> >>>>>>> situation, if things "just work" otherwise anyway?
> >>>>>> For DPDK applications, the most common case to initialize DPDK is like
> >>>>>> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
> >>>>>> to specify which cores to run and how much hugepages are used. Suppose
> >>>>>> we need this dpdk-app to run in a container, users already give those
> >>>>>> information when they build up the cgroup for it to run inside, this
> >>>>>> option or this patch is to make DPDK more smart to discover how much
> >>>>>> resource will be used. Make sense?
> >>>>> But then, all we need might be just a script that would extract this information from the system
> >>>>> and form a proper cmdline parameter for DPDK?
> >>>> Yes, a script will work. Or to construct (argc, argv) to call
> >>>> rte_eal_init() in the application. But as Neil Horman once suggested, a
> >>>> simple pthread_getaffinity_np() will get all things done. So if it worth
> >>>> a patch here?
> >>> Don't know...
> >>> Personally I would prefer not to put extra logic inside EAL.
> >>> For me - there are too many different options already.
> >> Then how about make it default in rte_eal_cpu_init()? And it is already
> >> known it will bring trouble to those use isolcpus users, they need to
> >> add "taskset [mask]" before starting a DPDK app.
> > As I said - provide a script?
> 
> Yes. But what I want to say is this script is hard to be right, if there
> are different kinds of limitations. (Barely happen though :-) )

My thought was to keep dpdk code untouched - i.e. let it still blindly set_pthread_affinity()
based on the input parameters, and in addition provide a script for those who want to run
in '--avail-cores' mode. 
So it could do 'taskset -p $$' and then either form -c parameter list  for the app,
or check existing -c/-l/--lcores parameter and complain if not allowed pcpu detected.
But ok, might be it is easier and more convenient to have this logic inside EAL,
then in a separate script.

> 
> > Same might be for amount of hugepage memory available to the user?
> 
> Ditto. Limitations like hugetlbfs quota, cgroup hugetlb, some are used
> by app themself (more like an artificial argument) ...
> >
> >>>   From other side looking at the patch itself:
> >>> You are updating lcore_count and lcore_config[],based on physical cpu availability,
> >>> but these days it is not always one-to-one mapping between EAL lcore and physical cpu.
> >>> Shouldn't that be taken into account?
> >> I have not see the problem so far, because this work is done before
> >> parsing coremask (-c), corelist (-l), and coremap (--lcores). If a core
> >> is disabled here, it's like it is not detected in rte_eal_cpu_init(). Or
> >> could you please give more hints?
> > I didn't test try changes, so probably I am missing something.
> > Let say iuser allowed to use only cpus 0-3.
> > If he would type with:
> >   --avail-cores  --lcores='(1-7)@2',
> > then only lcores 1-3 would be started.
> > Again if user would specify '2@(1-7)' it would also be undetected
> > that cpus 4-7 are note available to the user.
> > Is that so?
> 
> After reading the code:
> For case --lcores='(1-7)@2', lcores 1-7 would be started, and bind to
> pcore 2.
> For case --lcores='2@(1-7)', this will fail with "core 4 unavailable".
> 
> It's because:
> a.  although 1:1 mapping is built-up and flagged as detected if pcore is
> found in sysfs. (ROLE_RTE, cpuset, detected is true)
> b. in the beginning of eal_parse_lcores(), "reset lcore config".
> (ROLE_OFF, cpuset is empty, detected is still true)
> c. pcore cpuset will be checked by convert_to_cpuset using the previous
> "detected" value.

Ok, my bad then - I misunderstood the code.
Thanks for explanation.
So if I get it right now - first inside lib/librte_eal/common/eal_common_lcore.c
Both lcore_count and lcore_config relate to the pcpus.
Then later, at lib/librte_eal/common/eal_common_options.c
they are overwritten related to lcores information.
Except lcore_config[].detected, which seems kept intact.
Is that correct? 

> 
> I have tested it with the patch. Result aligns above analysis.
> For case --lcores='(1-7)@2': sudo taskset 0xf
> ./examples/helloworld/build/helloworld --avail-cores --lcores='(1-7)@2'
> ...
> hello from core 2
> hello from core 3
> hello from core 4
> hello from core 5
> hello from core 6
> hello from core 7
> hello from core 1
> 
> For case --lcores='2@(1-7)': sudo taskset 0xf
> ./examples/helloworld/build/helloworld --avail-cores --lcores='2@(1-7)'
> ...
> EAL: core 4 unavailable
> EAL: invalid parameter for --lcores
> ...
> 
> One thing may worth mention: shall "detected" be maintained in struct
> lcore_config? Maybe we need to maintain an data structure for pcores?

Yes, it might be good to split pcpu and lcores information somehow,
as it is a bit confusing right now.
But I suppose this is a subject for another patch/discussion.
Konstantin

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

On 3/10/2016 3:33 AM, Ananyev, Konstantin wrote:
>
>>>>>>>>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>>>>>>>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>>>>>>>>>> This patch adds option, --avail-cores, to use lcores which are
>>>>>>>>>>>> available
>>>>>>>>>>>> by calling pthread_getaffinity_np() to narrow down detected cores
>>>>>>>>>>>> before
>>>>>>>>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>>>>>>>>>
>>>>>>>>>>>> Test example:
>>>>>>>>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>>>>>>>>>             --avail-cores -m 1024
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>>>>>>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>>>>>>>>> Hmm, to me this sounds like something that should be done always so
>>>>>>>>>>> there's no need for an option. Or if there's a chance it might do the
>>>>>>>>>>> wrong thing in some rare circumstance then perhaps there should be a
>>>>>>>>>>> disabler option instead?
>>>>>>>>>> Thanks for comments.
>>>>>>>>>>
>>>>>>>>>> Yes, there's a use case that we cannot handle.
>>>>>>>>>>
>>>>>>>>>> If we make it as default, DPDK applications may fail to start, when user
>>>>>>>>>> specifies a core in isolcpus and its parent process (say bash) has a
>>>>>>>>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>>>>>>>>>> just blindly do pthread_setaffinity_np() and it always succeeds because
>>>>>>>>>> it always has root privilege to change any cpu affinity.
>>>>>>>>>>
>>>>>>>>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>>>>>>>>>> flagged as undetected (in my older implementation) and leads to failure.
>>>>>>>>>> To make it correct, we would always add "taskset mask" (or other ways)
>>>>>>>>>> before DPDK application cmd lines.
>>>>>>>>>>
>>>>>>>>>> How do you think?
>>>>>>>>> I still think it sounds like something that should be done by default
>>>>>>>>> and maybe be overridable with some flag, rather than the other way
>>>>>>>>> around. Another alternative might be detecting the cores always but if
>>>>>>>>> running as root, override but with a warning.
>>>>>>>> For your second solution, only root can setaffinity to isolcpus?
>>>>>>>> Your first solution seems like a promising way for me.
>>>>>>>>
>>>>>>>>> But I dont know, just wondering. To look at it from another angle: why
>>>>>>>>> would somebody use this new --avail-cores option and in what
>>>>>>>>> situation, if things "just work" otherwise anyway?
>>>>>>>> For DPDK applications, the most common case to initialize DPDK is like
>>>>>>>> this: "$dpdk-app [options for DPDK] -- [options for app]", so users need
>>>>>>>> to specify which cores to run and how much hugepages are used. Suppose
>>>>>>>> we need this dpdk-app to run in a container, users already give those
>>>>>>>> information when they build up the cgroup for it to run inside, this
>>>>>>>> option or this patch is to make DPDK more smart to discover how much
>>>>>>>> resource will be used. Make sense?
>>>>>>> But then, all we need might be just a script that would extract this information from the system
>>>>>>> and form a proper cmdline parameter for DPDK?
>>>>>> Yes, a script will work. Or to construct (argc, argv) to call
>>>>>> rte_eal_init() in the application. But as Neil Horman once suggested, a
>>>>>> simple pthread_getaffinity_np() will get all things done. So if it worth
>>>>>> a patch here?
>>>>> Don't know...
>>>>> Personally I would prefer not to put extra logic inside EAL.
>>>>> For me - there are too many different options already.
>>>> Then how about make it default in rte_eal_cpu_init()? And it is already
>>>> known it will bring trouble to those use isolcpus users, they need to
>>>> add "taskset [mask]" before starting a DPDK app.
>>> As I said - provide a script?
>> Yes. But what I want to say is this script is hard to be right, if there
>> are different kinds of limitations. (Barely happen though :-) )
> My thought was to keep dpdk code untouched - i.e. let it still blindly set_pthread_affinity()
> based on the input parameters, and in addition provide a script for those who want to run
> in '--avail-cores' mode.
> So it could do 'taskset -p $$' and then either form -c parameter list  for the app,
> or check existing -c/-l/--lcores parameter and complain if not allowed pcpu detected.
> But ok, might be it is easier and more convenient to have this logic inside EAL,
> then in a separate script.
>
>>> Same might be for amount of hugepage memory available to the user?
>> Ditto. Limitations like hugetlbfs quota, cgroup hugetlb, some are used
>> by app themself (more like an artificial argument) ...
>>>>>    From other side looking at the patch itself:
>>>>> You are updating lcore_count and lcore_config[],based on physical cpu availability,
>>>>> but these days it is not always one-to-one mapping between EAL lcore and physical cpu.
>>>>> Shouldn't that be taken into account?
>>>> I have not see the problem so far, because this work is done before
>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores). If a core
>>>> is disabled here, it's like it is not detected in rte_eal_cpu_init(). Or
>>>> could you please give more hints?
>>> I didn't test try changes, so probably I am missing something.
>>> Let say iuser allowed to use only cpus 0-3.
>>> If he would type with:
>>>    --avail-cores  --lcores='(1-7)@2',
>>> then only lcores 1-3 would be started.
>>> Again if user would specify '2@(1-7)' it would also be undetected
>>> that cpus 4-7 are note available to the user.
>>> Is that so?
>> After reading the code:
>> For case --lcores='(1-7)@2', lcores 1-7 would be started, and bind to
>> pcore 2.
>> For case --lcores='2@(1-7)', this will fail with "core 4 unavailable".
>>
>> It's because:
>> a.  although 1:1 mapping is built-up and flagged as detected if pcore is
>> found in sysfs. (ROLE_RTE, cpuset, detected is true)
>> b. in the beginning of eal_parse_lcores(), "reset lcore config".
>> (ROLE_OFF, cpuset is empty, detected is still true)
>> c. pcore cpuset will be checked by convert_to_cpuset using the previous
>> "detected" value.
> Ok, my bad then - I misunderstood the code.
> Thanks for explanation.
> So if I get it right now - first inside lib/librte_eal/common/eal_common_lcore.c
> Both lcore_count and lcore_config relate to the pcpus.
> Then later, at lib/librte_eal/common/eal_common_options.c
> they are overwritten related to lcores information.
> Except lcore_config[].detected, which seems kept intact.
> Is that correct?

Yes, exactly. And really appreciate that you raise up this question for 
discussion.

>
>> I have tested it with the patch. Result aligns above analysis.
>> For case --lcores='(1-7)@2': sudo taskset 0xf
>> ./examples/helloworld/build/helloworld --avail-cores --lcores='(1-7)@2'
>> ...
>> hello from core 2
>> hello from core 3
>> hello from core 4
>> hello from core 5
>> hello from core 6
>> hello from core 7
>> hello from core 1
>>
>> For case --lcores='2@(1-7)': sudo taskset 0xf
>> ./examples/helloworld/build/helloworld --avail-cores --lcores='2@(1-7)'
>> ...
>> EAL: core 4 unavailable
>> EAL: invalid parameter for --lcores
>> ...
>>
>> One thing may worth mention: shall "detected" be maintained in struct
>> lcore_config? Maybe we need to maintain an data structure for pcores?
> Yes, it might be good to split pcpu and lcores information somehow,
> as it is a bit confusing right now.
> But I suppose this is a subject for another patch/discussion.

Yes, just another topic.

Thanks,
Jianfeng

> Konstantin
>
>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

Hi,

Since some guys are asking about the status of this patch, I'd like to 
ping if anyone still has concerns.
Current conclusion is: with option --avail-cores.

Thanks,
Jianfeng

On 3/4/2016 6:05 PM, Jianfeng Tan wrote:
> This patch adds option, --avail-cores, to use lcores which are available
> by calling pthread_getaffinity_np() to narrow down detected cores before
> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>
> Test example:
> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> 		--avail-cores -m 1024
>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> ---
>   lib/librte_eal/common/eal_common_options.c | 52 ++++++++++++++++++++++++++++++
>   lib/librte_eal/common/eal_options.h        |  2 ++
>   2 files changed, 54 insertions(+)
>
> diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
> index 29942ea..dc4882d 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -95,6 +95,7 @@ eal_long_options[] = {
>   	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
>   	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
>   	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
> +	{OPT_AVAIL_CORES,       0, NULL, OPT_AVAIL_CORES_NUM      },
>   	{0,                     0, NULL, 0                        }
>   };
>   
> @@ -681,6 +682,37 @@ err:
>   }
>   
>   static int
> +eal_parse_avail_cores(void)
> +{
> +	int i, count;
> +	pthread_t tid;
> +	rte_cpuset_t cpuset;
> +	struct rte_config *cfg = rte_eal_get_configuration();
> +
> +	tid = pthread_self();
> +	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cpuset) != 0)
> +		return -1;
> +
> +	for (i = 0, count = 0; i < RTE_MAX_LCORE; i++) {
> +		if (lcore_config[i].detected && !CPU_ISSET(i, &cpuset)) {
> +			RTE_LOG(DEBUG, EAL, "Flag lcore %u as undetected\n", i);
> +			lcore_config[i].detected = 0;
> +			lcore_config[i].core_index = -1;
> +			cfg->lcore_role[i] = ROLE_OFF;
> +			count++;
> +		}
> +	}
> +	cfg->lcore_count -= count;
> +	if (cfg->lcore_count == 0) {
> +		RTE_LOG(ERR, EAL, "No lcores available\n");
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +
> +static int
>   eal_parse_syslog(const char *facility, struct internal_config *conf)
>   {
>   	int i;
> @@ -754,6 +786,10 @@ eal_parse_proc_type(const char *arg)
>   	return RTE_PROC_INVALID;
>   }
>   
> +static int param_coremask;
> +static int param_corelist;
> +static int param_coremap;
> +
>   int
>   eal_parse_common_option(int opt, const char *optarg,
>   			struct internal_config *conf)
> @@ -775,6 +811,7 @@ eal_parse_common_option(int opt, const char *optarg,
>   		break;
>   	/* coremask */
>   	case 'c':
> +		param_coremask = 1;
>   		if (eal_parse_coremask(optarg) < 0) {
>   			RTE_LOG(ERR, EAL, "invalid coremask\n");
>   			return -1;
> @@ -782,6 +819,7 @@ eal_parse_common_option(int opt, const char *optarg,
>   		break;
>   	/* corelist */
>   	case 'l':
> +		param_corelist = 1;
>   		if (eal_parse_corelist(optarg) < 0) {
>   			RTE_LOG(ERR, EAL, "invalid core list\n");
>   			return -1;
> @@ -890,12 +928,25 @@ eal_parse_common_option(int opt, const char *optarg,
>   		break;
>   	}
>   	case OPT_LCORES_NUM:
> +		param_coremap = 1;
>   		if (eal_parse_lcores(optarg) < 0) {
>   			RTE_LOG(ERR, EAL, "invalid parameter for --"
>   				OPT_LCORES "\n");
>   			return -1;
>   		}
>   		break;
> +	case OPT_AVAIL_CORES_NUM:
> +		if (param_coremask || param_corelist || param_coremap) {
> +			RTE_LOG(ERR, EAL, "should put --" OPT_AVAIL_CORES
> +				" before -c, -l and --" OPT_LCORES "\n");
> +			return -1;
> +		}
> +		if (eal_parse_avail_cores() < 0) {
> +			RTE_LOG(ERR, EAL, "failed to use --"
> +				OPT_AVAIL_CORES "\n");
> +			return -1;
> +		}
> +		break;
>   
>   	/* don't know what to do, leave this to caller */
>   	default:
> @@ -990,6 +1041,7 @@ eal_common_usage(void)
>   	       "                      ',' is used for single number separator.\n"
>   	       "                      '( )' can be omitted for single element group,\n"
>   	       "                      '@' can be omitted if cpus and lcores have the same value\n"
> +	       "  --"OPT_AVAIL_CORES"       Use pthread_getaffinity_np() to detect cores to be used\n"
>   	       "  --"OPT_MASTER_LCORE" ID   Core ID that is used as master\n"
>   	       "  -n CHANNELS         Number of memory channels\n"
>   	       "  -m MB               Memory to allocate (see also --"OPT_SOCKET_MEM")\n"
> diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
> index a881c62..b2ddea3 100644
> --- a/lib/librte_eal/common/eal_options.h
> +++ b/lib/librte_eal/common/eal_options.h
> @@ -83,6 +83,8 @@ enum {
>   	OPT_VMWARE_TSC_MAP_NUM,
>   #define OPT_XEN_DOM0          "xen-dom0"
>   	OPT_XEN_DOM0_NUM,
> +#define OPT_AVAIL_CORES       "avail-cores"
> +	OPT_AVAIL_CORES_NUM,
>   	OPT_LONG_MAX_NUM
>   };
>   

Re: [PATCH v2] eal: make hugetlb initialization more robust

[Sergio Gonzalez Monroy]

On 08/03/2016 01:42, Jianfeng Tan wrote:
> This patch adds an option, --huge-trybest, to use a recover mechanism to
> the case that there are not so many hugepages (declared in sysfs), which
> can be used. It relys on a mem access to fault-in hugepages, and if fails
> with SIGBUS, recover to previously saved stack environment with
> siglongjmp().
>
> Test example:
>    a. cgcreate -g hugetlb:/test-subgroup
>    b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>    c. cgexec -g hugetlb:test-subgroup \
> 	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest

I think you should mention in the commit message that this option also 
covers the case
of hugetlbfs mount with quota.

>   
> +static sigjmp_buf jmpenv;
> +
> +static void sigbus_handler(int signo __rte_unused)
> +{
> +	siglongjmp(jmpenv, 1);
> +}
> +
> +/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
> + * non-static local variable in the stack frame calling setjmp might be
> + * clobbered by a call to longjmp.
> + */
> +static int wrap_setjmp(void)
> +{
> +	return setjmp(jmpenv);
> +}

Use sigsetjmp instead of setjmp and restore the signal masks.

>   /*
>    * Mmap all hugepages of hugepage table: it first open a file in
>    * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
> @@ -396,7 +413,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   		if (fd < 0) {
>   			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
>   					strerror(errno));
> -			return -1;
> +			return i;

When using --try-best, we could get an error and still work as expected.
It can be confusing for users to see an error when it is expected behavior.

Any thoughts?

>   		}
>   
>   		/* map the segment, and populate page tables,
> @@ -407,7 +424,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
>   					strerror(errno));
>   			close(fd);
> -			return -1;
> +			return i;
>   		}
>   

Same comment as above

>   		/* set shared flock on the file. */
>   		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
>   			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
>   				__func__, strerror(errno));
>   			close(fd);
> -			return -1;
> +			return i;

Same comment as above

> @@ -1137,10 +1206,24 @@ rte_eal_hugepage_init(void)
>   			continue;
>   
>   		/* map all hugepages available */
> -		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
> -			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
> -					(unsigned)(hpi->hugepage_sz / 0x100000));
> -			goto fail;
> +		pages_old = hpi->num_pages[0];
> +		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
> +		if (pages_new < pages_old) {
> +			RTE_LOG(DEBUG, EAL,
> +				"%d not %d hugepages of size %u MB allocated\n",
> +				pages_new, pages_old,
> +				(unsigned)(hpi->hugepage_sz / 0x100000));
> +			if (internal_config.huge_trybest) {
> +				int pages = pages_old - pages_new;
> +
> +				internal_config.memory -=
> +					hpi->hugepage_sz * pages;
> +				nr_hugepages -= pages;
> +				hpi->num_pages[0] = pages_new;
> +				if (pages_new == 0)
> +					continue;
> +			} else
> +				goto fail;
>   		}

There is another call to map_all_hugepages that you are not updating the 
check of the return value.

Sergio

Re: [PATCH v2] eal: make hugetlb initialization more robust

[Tan, Jianfeng]

Hi Sergio,


On 5/4/2016 7:07 PM, Sergio Gonzalez Monroy wrote:
> On 08/03/2016 01:42, Jianfeng Tan wrote:
>> This patch adds an option, --huge-trybest, to use a recover mechanism to
>> the case that there are not so many hugepages (declared in sysfs), which
>> can be used. It relys on a mem access to fault-in hugepages, and if 
>> fails
>> with SIGBUS, recover to previously saved stack environment with
>> siglongjmp().
>>
>> Test example:
>>    a. cgcreate -g hugetlb:/test-subgroup
>>    b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>>    c. cgexec -g hugetlb:test-subgroup \
>>       ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest
>
> I think you should mention in the commit message that this option also 
> covers the case
> of hugetlbfs mount with quota.

Yes, I should do that.

>
>>   +static sigjmp_buf jmpenv;
>> +
>> +static void sigbus_handler(int signo __rte_unused)
>> +{
>> +    siglongjmp(jmpenv, 1);
>> +}
>> +
>> +/* Put setjmp into a wrap method to avoid compiling error. Any 
>> non-volatile,
>> + * non-static local variable in the stack frame calling setjmp might be
>> + * clobbered by a call to longjmp.
>> + */
>> +static int wrap_setjmp(void)
>> +{
>> +    return setjmp(jmpenv);
>> +}
>
> Use sigsetjmp instead of setjmp and restore the signal masks.

The difference lies in whether signal mask will be saved for further 
restore. And you are right we should keep either sigsetjmp(xxx, 
1)/siglongjmp(xxx, 1) or setjmp()/longjmp. Nice catch! I'll go with the 
former.

>
>>   /*
>>    * Mmap all hugepages of hugepage table: it first open a file in
>>    * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
>> @@ -396,7 +413,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>>           if (fd < 0) {
>>               RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
>>                       strerror(errno));
>> -            return -1;
>> +            return i;
>
> When using --try-best, we could get an error and still work as expected.
> It can be confusing for users to see an error when it is expected 
> behavior.
>
> Any thoughts?

Shall we remove those RTE_LOG complaints, because the failure here does 
not mean we cannot satisfy the requirements of applications?

...
> There is another call to map_all_hugepages that you are not updating 
> the check of the return value.

You are right, the second map_all_hugepages's return value check should 
be changed to compare with the total hugepages owned by the hpi. I'll 
fix this.

Thanks,
Jianfeng

>
> Sergio
>

Re: [PATCH v2] eal: make hugetlb initialization more robust

[Sergio Gonzalez Monroy]

On 08/03/2016 01:42, Jianfeng Tan wrote:
> This patch adds an option, --huge-trybest, to use a recover mechanism to
> the case that there are not so many hugepages (declared in sysfs), which
> can be used. It relys on a mem access to fault-in hugepages, and if fails
> with SIGBUS, recover to previously saved stack environment with
> siglongjmp().
>
> Test example:
>    a. cgcreate -g hugetlb:/test-subgroup
>    b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>    c. cgexec -g hugetlb:test-subgroup \
> 	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest

I think you should mention in the commit message that this option also 
covers the case
of hugetlbfs mount with quota.

>
> +static sigjmp_buf jmpenv;
> +
> +static void sigbus_handler(int signo __rte_unused)
> +{
> +	siglongjmp(jmpenv, 1);
> +}
> +
> +/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
> + * non-static local variable in the stack frame calling setjmp might be
> + * clobbered by a call to longjmp.
> + */
> +static int wrap_setjmp(void)
> +{
> +	return setjmp(jmpenv);
> +}

Use sigsetjmp instead of setjmp and restore the signal masks.

>   /*
>    * Mmap all hugepages of hugepage table: it first open a file in
>    * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
> @@ -396,7 +413,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   		if (fd < 0) {
>   			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
>   					strerror(errno));
> -			return -1;
> +			return i;

When using --try-best, we could get an error and still work as expected.
It can be confusing for users to see an error when it is expected behavior.

Any thoughts?

>   		}
>
>   		/* map the segment, and populate page tables,
> @@ -407,7 +424,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>   			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
>   					strerror(errno));
>   			close(fd);
> -			return -1;
> +			return i;
>   		}
>

Same comment as above

>   		/* set shared flock on the file. */
>   		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
>   			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
>   				__func__, strerror(errno));
>   			close(fd);
> -			return -1;
> +			return i;

Same comment as above

> @@ -1137,10 +1206,24 @@ rte_eal_hugepage_init(void)
>   			continue;
>
>   		/* map all hugepages available */
> -		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
> -			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
> -					(unsigned)(hpi->hugepage_sz / 0x100000));
> -			goto fail;
> +		pages_old = hpi->num_pages[0];
> +		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
> +		if (pages_new < pages_old) {
> +			RTE_LOG(DEBUG, EAL,
> +				"%d not %d hugepages of size %u MB allocated\n",
> +				pages_new, pages_old,
> +				(unsigned)(hpi->hugepage_sz / 0x100000));
> +			if (internal_config.huge_trybest) {
> +				int pages = pages_old - pages_new;
> +
> +				internal_config.memory -=
> +					hpi->hugepage_sz * pages;
> +				nr_hugepages -= pages;
> +				hpi->num_pages[0] = pages_new;
> +				if (pages_new == 0)
> +					continue;
> +			} else
> +				goto fail;
>   		}

There is another call to map_all_hugepages that you are not updating the 
check of the return value.

Sergio

[PATCH v3] eal: make hugetlb initialization more robust

[Jianfeng Tan]

This patch adds an option, --huge-trybest, to use a recover mechanism to
the case that there are not so many hugepages (declared in sysfs), which
can be used. It relys on a mem access to fault-in hugepages, and if fails
with SIGBUS, recover to previously saved stack environment with
siglongjmp().

Besides, this solution fixes an issue when hugetlbfs is specified with an
option of size. Currently DPDK does not respect the quota of a hugetblfs
mount. It fails to init the EAL because it tries to map the number of free
hugepages in the system rather than using the number specified in the quota
for that mount.

It's still an open issue with CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS. Under
this case (such as IVSHMEM target), having hugetlbfs mounts with quota will
fail to remap hugepages as it relies on having mapped all free hugepages
in the system.

Test example:
  a. cgcreate -g hugetlb:/test-subgroup
  b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
  c. cgexec -g hugetlb:test-subgroup \
	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v3:
 - Reword commit message to include it fixes the hugetlbfs quota issue.
 - setjmp -> sigsetjmp.
 - Fix RTE_LOG complaint from ERR to DEBUG as it does not mean init error
   so far.
 - Fix the second map_all_hugepages's return value check.
v2:
 - Address the compiling error by move setjmp into a wrap method.

 lib/librte_eal/common/eal_common_options.c |   4 +
 lib/librte_eal/common/eal_internal_cfg.h   |   1 +
 lib/librte_eal/common/eal_options.h        |   2 +
 lib/librte_eal/linuxapp/eal/eal.c          |   1 +
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 115 +++++++++++++++++++++++++----
 5 files changed, 110 insertions(+), 13 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 3efc90f..e9a111d 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+	{OPT_HUGE_TRYBEST,      0, NULL, OPT_HUGE_TRYBEST_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -899,6 +900,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_HUGE_TRYBEST_NUM:
+		internal_config.huge_trybest = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..90a3533 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -64,6 +64,7 @@ struct internal_config {
 	volatile unsigned force_nchannel; /**< force number of channels */
 	volatile unsigned force_nrank;    /**< force number of ranks */
 	volatile unsigned no_hugetlbfs;   /**< true to disable hugetlbfs */
+	volatile unsigned huge_trybest;   /**< try best to allocate hugepages */
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned xen_dom0_support; /**< support app running on Xen Dom0*/
 	volatile unsigned no_pci;         /**< true to disable PCI */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..02397c5 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#define OPT_HUGE_TRYBEST      "huge-trybest"
+	OPT_HUGE_TRYBEST_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8aafd51..eeb1d4e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -343,6 +343,7 @@ eal_usage(const char *prgname)
 	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
 	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
 	       "  --"OPT_XEN_DOM0"          Support running on Xen dom0 without hugetlbfs\n"
+	       "  --"OPT_HUGE_TRYBEST"      Try best to accommodate hugepages\n"
 	       "\n");
 	/* Allow the application to print its usage message too if hook is set */
 	if ( rte_application_usage_hook ) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 5b9132c..cb0df76 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,21 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf jmpenv;
+
+static void sigbus_handler(int signo __rte_unused)
+{
+	siglongjmp(jmpenv, 1);
+}
+
+/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
+ * non-static local variable in the stack frame calling sigsetjmp might be
+ * clobbered by a call to longjmp.
+ */
+static int wrap_sigsetjmp(void)
+{
+	return sigsetjmp(jmpenv, 1);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
@@ -394,9 +411,9 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		/* try to create hugepage file */
 		fd = open(hugepg_tbl[i].filepath, O_CREAT | O_RDWR, 0755);
 		if (fd < 0) {
-			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		/* map the segment, and populate page tables,
@@ -404,10 +421,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
 				MAP_SHARED | MAP_POPULATE, fd, 0);
 		if (virtaddr == MAP_FAILED) {
-			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
@@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			hugepg_tbl[i].final_va = virtaddr;
 		}
 
+		if (orig && internal_config.huge_trybest) {
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (wrap_sigsetjmp()) {
+				RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				unlink(hugepg_tbl[i].filepath);
+				return i;
+			}
+			*(int *)virtaddr = 0;
+		}
+
+
 		/* set shared flock on the file. */
 		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
-			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
+			RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -430,7 +468,7 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1036,6 +1074,33 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static struct sigaction action_old;
+static int need_recover;
+
+static void
+register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = sigbus_handler;
+
+	need_recover = !sigaction(SIGBUS, &action, &action_old);
+}
+
+static void
+recover_sigbus(void)
+{
+	if (need_recover) {
+		sigaction(SIGBUS, &action_old, NULL);
+		need_recover = 0;
+	}
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1122,8 +1187,12 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	if (internal_config.huge_trybest)
+		register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		int pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1137,10 +1206,24 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
-			goto fail;
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+			if (internal_config.huge_trybest) {
+				int pages = pages_old - pages_new;
+
+				internal_config.memory -=
+					hpi->hugepage_sz * pages;
+				nr_hugepages -= pages;
+				hpi->num_pages[0] = pages_new;
+				if (pages_new == 0)
+					continue;
+			} else
+				goto fail;
 		}
 
 		/* find physical addresses and sockets for each hugepage */
@@ -1172,8 +1255,9 @@ rte_eal_hugepage_init(void)
 		hp_offset += new_pages_count[i];
 #else
 		/* remap all hugepages */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to remap %u MB pages\n",
+		if ((uint32_t)map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) !=
+		    hpi->num_pages[0]) {
+			RTE_LOG(ERR, EAL, "Failed to remap %u MB pages\n",
 					(unsigned)(hpi->hugepage_sz / 0x100000));
 			goto fail;
 		}
@@ -1187,6 +1271,9 @@ rte_eal_hugepage_init(void)
 #endif
 	}
 
+	if (internal_config.huge_trybest)
+		recover_sigbus();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
@@ -1373,6 +1460,8 @@ rte_eal_hugepage_init(void)
 	return 0;
 
 fail:
+	if (internal_config.huge_trybest)
+		recover_sigbus();
 	free(tmp_hp);
 	return -1;
 }
-- 
2.1.4

Re: [PATCH v3] eal: make hugetlb initialization more robust

[Sergio Gonzalez Monroy]

Hi Jianfeng,

On 09/05/2016 11:48, Jianfeng Tan wrote:

>   		/* find physical addresses and sockets for each hugepage */
> @@ -1172,8 +1255,9 @@ rte_eal_hugepage_init(void)
>   		hp_offset += new_pages_count[i];
>   #else
>   		/* remap all hugepages */
> -		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) < 0){
> -			RTE_LOG(DEBUG, EAL, "Failed to remap %u MB pages\n",
> +		if ((uint32_t)map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) !=
> +		    hpi->num_pages[0]) {

It probably makes more sense to have map_all_hugepages return uint32_t 
instead.

Sergio

Re: [PATCH v3] eal: make hugetlb initialization more robust

[Tan, Jianfeng]

Hi Sergio,

> -----Original Message-----
> From: Gonzalez Monroy, Sergio
> Sent: Tuesday, May 10, 2016 4:55 PM
> To: Tan, Jianfeng; dev@dpdk.org
> Cc: david.marchand@6wind.com; nhorman@tuxdriver.com
> Subject: Re: [PATCH v3] eal: make hugetlb initialization more robust
> 
> 
> Hi Jianfeng,
> 
> On 09/05/2016 11:48, Jianfeng Tan wrote:
> 
> >   		/* find physical addresses and sockets for each hugepage */
> > @@ -1172,8 +1255,9 @@ rte_eal_hugepage_init(void)
> >   		hp_offset += new_pages_count[i];
> >   #else
> >   		/* remap all hugepages */
> > -		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) < 0){
> > -			RTE_LOG(DEBUG, EAL, "Failed to remap %u MB
> pages\n",
> > +		if ((uint32_t)map_all_hugepages(&tmp_hp[hp_offset], hpi,
> 0) !=
> > +		    hpi->num_pages[0]) {
> 
> It probably makes more sense to have map_all_hugepages return uint32_t
> instead.

Yes, I agree. I was wrongly expecting there's a freebsd version map_all_hugepages with the same function type.

I'll fix this in next version.

Thanks,
Jianfeng

> 
> Sergio

[PATCH v4] eal: make hugetlb initialization more robust

[Jianfeng Tan]

This patch adds an option, --huge-trybest, to use a recover mechanism to
the case that there are not so many hugepages (declared in sysfs), which
can be used. It relys on a mem access to fault-in hugepages, and if fails
with SIGBUS, recover to previously saved stack environment with
siglongjmp().

Besides, this solution fixes an issue when hugetlbfs is specified with an
option of size. Currently DPDK does not respect the quota of a hugetblfs
mount. It fails to init the EAL because it tries to map the number of free
hugepages in the system rather than using the number specified in the quota
for that mount.

It's still an open issue with CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS. Under
this case (such as IVSHMEM target), having hugetlbfs mounts with quota will
fail to remap hugepages as it relies on having mapped all free hugepages
in the system.

Test example:
  a. cgcreate -g hugetlb:/test-subgroup
  b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
  c. cgexec -g hugetlb:test-subgroup \
	  ./examples/helloworld/build/helloworld -c 0x2 -n 4 --huge-trybest

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v4:
 - Change map_all_hugepages to return unsigned instead of int.
v3:
 - Reword commit message to include it fixes the hugetlbfs quota issue.
 - setjmp -> sigsetjmp.
 - Fix RTE_LOG complaint from ERR to DEBUG as it does not mean init error
   so far.
 - Fix the second map_all_hugepages's return value check.
v2:
 - Address the compiling error by move setjmp into a wrap method.

 lib/librte_eal/common/eal_common_options.c |   4 +
 lib/librte_eal/common/eal_internal_cfg.h   |   1 +
 lib/librte_eal/common/eal_options.h        |   2 +
 lib/librte_eal/linuxapp/eal/eal.c          |   1 +
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 118 +++++++++++++++++++++++++----
 5 files changed, 112 insertions(+), 14 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 3efc90f..e9a111d 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -95,6 +95,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_XEN_DOM0,          0, NULL, OPT_XEN_DOM0_NUM         },
+	{OPT_HUGE_TRYBEST,      0, NULL, OPT_HUGE_TRYBEST_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -899,6 +900,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_HUGE_TRYBEST_NUM:
+		internal_config.huge_trybest = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index 5f1367e..90a3533 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -64,6 +64,7 @@ struct internal_config {
 	volatile unsigned force_nchannel; /**< force number of channels */
 	volatile unsigned force_nrank;    /**< force number of ranks */
 	volatile unsigned no_hugetlbfs;   /**< true to disable hugetlbfs */
+	volatile unsigned huge_trybest;   /**< try best to allocate hugepages */
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned xen_dom0_support; /**< support app running on Xen Dom0*/
 	volatile unsigned no_pci;         /**< true to disable PCI */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index a881c62..02397c5 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -83,6 +83,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_XEN_DOM0          "xen-dom0"
 	OPT_XEN_DOM0_NUM,
+#define OPT_HUGE_TRYBEST      "huge-trybest"
+	OPT_HUGE_TRYBEST_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8aafd51..eeb1d4e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -343,6 +343,7 @@ eal_usage(const char *prgname)
 	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
 	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
 	       "  --"OPT_XEN_DOM0"          Support running on Xen dom0 without hugetlbfs\n"
+	       "  --"OPT_HUGE_TRYBEST"      Try best to accommodate hugepages\n"
 	       "\n");
 	/* Allow the application to print its usage message too if hook is set */
 	if ( rte_application_usage_hook ) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 5b9132c..8c77010 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,21 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf jmpenv;
+
+static void sigbus_handler(int signo __rte_unused)
+{
+	siglongjmp(jmpenv, 1);
+}
+
+/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
+ * non-static local variable in the stack frame calling sigsetjmp might be
+ * clobbered by a call to longjmp.
+ */
+static int wrap_sigsetjmp(void)
+{
+	return sigsetjmp(jmpenv, 1);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
@@ -316,7 +333,7 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
  * in hugepg_tbl[i].final_va. The second mapping (when orig is 0) tries to
  * map continguous physical blocks in contiguous virtual blocks.
  */
-static int
+static unsigned
 map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		struct hugepage_info *hpi, int orig)
 {
@@ -394,9 +411,9 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		/* try to create hugepage file */
 		fd = open(hugepg_tbl[i].filepath, O_CREAT | O_RDWR, 0755);
 		if (fd < 0) {
-			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		/* map the segment, and populate page tables,
@@ -404,10 +421,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
 				MAP_SHARED | MAP_POPULATE, fd, 0);
 		if (virtaddr == MAP_FAILED) {
-			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
@@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			hugepg_tbl[i].final_va = virtaddr;
 		}
 
+		if (orig && internal_config.huge_trybest) {
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (wrap_sigsetjmp()) {
+				RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				unlink(hugepg_tbl[i].filepath);
+				return i;
+			}
+			*(int *)virtaddr = 0;
+		}
+
+
 		/* set shared flock on the file. */
 		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
-			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
+			RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -430,7 +468,8 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1036,6 +1075,33 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static struct sigaction action_old;
+static int need_recover;
+
+static void
+register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = sigbus_handler;
+
+	need_recover = !sigaction(SIGBUS, &action, &action_old);
+}
+
+static void
+recover_sigbus(void)
+{
+	if (need_recover) {
+		sigaction(SIGBUS, &action_old, NULL);
+		need_recover = 0;
+	}
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1122,8 +1188,12 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	if (internal_config.huge_trybest)
+		register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		unsigned pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1137,10 +1207,24 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
-			goto fail;
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+			if (internal_config.huge_trybest) {
+				int pages = pages_old - pages_new;
+
+				internal_config.memory -=
+					hpi->hugepage_sz * pages;
+				nr_hugepages -= pages;
+				hpi->num_pages[0] = pages_new;
+				if (pages_new == 0)
+					continue;
+			} else
+				goto fail;
 		}
 
 		/* find physical addresses and sockets for each hugepage */
@@ -1172,8 +1256,9 @@ rte_eal_hugepage_init(void)
 		hp_offset += new_pages_count[i];
 #else
 		/* remap all hugepages */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to remap %u MB pages\n",
+		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) !=
+		    hpi->num_pages[0]) {
+			RTE_LOG(ERR, EAL, "Failed to remap %u MB pages\n",
 					(unsigned)(hpi->hugepage_sz / 0x100000));
 			goto fail;
 		}
@@ -1187,6 +1272,9 @@ rte_eal_hugepage_init(void)
 #endif
 	}
 
+	if (internal_config.huge_trybest)
+		recover_sigbus();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
@@ -1373,6 +1461,8 @@ rte_eal_hugepage_init(void)
 	return 0;
 
 fail:
+	if (internal_config.huge_trybest)
+		recover_sigbus();
 	free(tmp_hp);
 	return -1;
 }
-- 
2.1.4

Re: [PATCH v4] eal: make hugetlb initialization more robust

[David Marchand]

Hello Jianfeng,

On Thu, May 12, 2016 at 2:44 AM, Jianfeng Tan <jianfeng.tan@intel.com> wrote:
> This patch adds an option, --huge-trybest, to use a recover mechanism to
> the case that there are not so many hugepages (declared in sysfs), which
> can be used. It relys on a mem access to fault-in hugepages, and if fails
> with SIGBUS, recover to previously saved stack environment with
> siglongjmp().
>
> Besides, this solution fixes an issue when hugetlbfs is specified with an
> option of size. Currently DPDK does not respect the quota of a hugetblfs
> mount. It fails to init the EAL because it tries to map the number of free
> hugepages in the system rather than using the number specified in the quota
> for that mount.
>
> It's still an open issue with CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS. Under
> this case (such as IVSHMEM target), having hugetlbfs mounts with quota will
> fail to remap hugepages as it relies on having mapped all free hugepages
> in the system.

For such a case case, maybe having some warning log message when it
fails would help the user.
+ a known issue in the release notes ?


> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index 5b9132c..8c77010 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>                         hugepg_tbl[i].final_va = virtaddr;
>                 }
>
> +               if (orig && internal_config.huge_trybest) {
> +                       /* In linux, hugetlb limitations, like cgroup, are
> +                        * enforced at fault time instead of mmap(), even
> +                        * with the option of MAP_POPULATE. Kernel will send
> +                        * a SIGBUS signal. To avoid to be killed, save stack
> +                        * environment here, if SIGBUS happens, we can jump
> +                        * back here.
> +                        */
> +                       if (wrap_sigsetjmp()) {
> +                               RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
> +                                       "hugepages of size %u MB\n",
> +                                       (unsigned)(hugepage_sz / 0x100000));
> +                               munmap(virtaddr, hugepage_sz);
> +                               close(fd);
> +                               unlink(hugepg_tbl[i].filepath);
> +                               return i;
> +                       }
> +                       *(int *)virtaddr = 0;
> +               }
> +
> +
>                 /* set shared flock on the file. */
>                 if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
> -                       RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
> +                       RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s \n",
>                                 __func__, strerror(errno));
>                         close(fd);
> -                       return -1;
> +                       return i;
>                 }
>
>                 close(fd);

Maybe I missed something, but we are writing into some hugepage before
the flock has been called.
Are we sure there is nobody else using this hugepage ?

Especially, can't this cause trouble to a primary process running if
we start the exact same primary process ?


-- 
David Marchand

Re: [PATCH v4] eal: make hugetlb initialization more robust

[Thomas Monjalon]

2016-05-12 00:44, Jianfeng Tan:
> This patch adds an option, --huge-trybest, to use a recover mechanism to
> the case that there are not so many hugepages (declared in sysfs), which
> can be used. It relys on a mem access to fault-in hugepages, and if fails

relys -> relies

> with SIGBUS, recover to previously saved stack environment with
> siglongjmp().
> 
> Besides, this solution fixes an issue when hugetlbfs is specified with an
> option of size. Currently DPDK does not respect the quota of a hugetblfs
> mount. It fails to init the EAL because it tries to map the number of free
> hugepages in the system rather than using the number specified in the quota
> for that mount.

It looks to be a bug. Why adding an option?
What is the benefit of the old behaviour, not using --try-best?

> +static sigjmp_buf jmpenv;
> +
> +static void sigbus_handler(int signo __rte_unused)
> +{
> +	siglongjmp(jmpenv, 1);
> +}
> +
> +/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
> + * non-static local variable in the stack frame calling sigsetjmp might be
> + * clobbered by a call to longjmp.
> + */
> +static int wrap_sigsetjmp(void)
> +{
> +	return sigsetjmp(jmpenv, 1);
> +}

Please add the word "huge" to these variables and functions.

> +static struct sigaction action_old;
> +static int need_recover;
> +
> +static void
> +register_sigbus(void)
> +{
> +	sigset_t mask;
> +	struct sigaction action;
> +
> +	sigemptyset(&mask);
> +	sigaddset(&mask, SIGBUS);
> +	action.sa_flags = 0;
> +	action.sa_mask = mask;
> +	action.sa_handler = sigbus_handler;
> +
> +	need_recover = !sigaction(SIGBUS, &action, &action_old);
> +}
> +
> +static void
> +recover_sigbus(void)
> +{
> +	if (need_recover) {
> +		sigaction(SIGBUS, &action_old, NULL);
> +		need_recover = 0;
> +	}
> +}

Idem, Please add the word "huge".

Re: [PATCH v4] eal: make hugetlb initialization more robust

[Sergio Gonzalez Monroy]

On 17/05/2016 17:39, David Marchand wrote:
> Hello Jianfeng,
>
> On Thu, May 12, 2016 at 2:44 AM, Jianfeng Tan <jianfeng.tan@intel.com> wrote:
>> This patch adds an option, --huge-trybest, to use a recover mechanism to
>> the case that there are not so many hugepages (declared in sysfs), which
>> can be used. It relys on a mem access to fault-in hugepages, and if fails
>> with SIGBUS, recover to previously saved stack environment with
>> siglongjmp().
>>
>> Besides, this solution fixes an issue when hugetlbfs is specified with an
>> option of size. Currently DPDK does not respect the quota of a hugetblfs
>> mount. It fails to init the EAL because it tries to map the number of free
>> hugepages in the system rather than using the number specified in the quota
>> for that mount.
>>
>> It's still an open issue with CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS. Under
>> this case (such as IVSHMEM target), having hugetlbfs mounts with quota will
>> fail to remap hugepages as it relies on having mapped all free hugepages
>> in the system.
> For such a case case, maybe having some warning log message when it
> fails would help the user.
> + a known issue in the release notes ?
>
>
>> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
>> index 5b9132c..8c77010 100644
>> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
>> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
>> @@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>>                          hugepg_tbl[i].final_va = virtaddr;
>>                  }
>>
>> +               if (orig && internal_config.huge_trybest) {
>> +                       /* In linux, hugetlb limitations, like cgroup, are
>> +                        * enforced at fault time instead of mmap(), even
>> +                        * with the option of MAP_POPULATE. Kernel will send
>> +                        * a SIGBUS signal. To avoid to be killed, save stack
>> +                        * environment here, if SIGBUS happens, we can jump
>> +                        * back here.
>> +                        */
>> +                       if (wrap_sigsetjmp()) {
>> +                               RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
>> +                                       "hugepages of size %u MB\n",
>> +                                       (unsigned)(hugepage_sz / 0x100000));
>> +                               munmap(virtaddr, hugepage_sz);
>> +                               close(fd);
>> +                               unlink(hugepg_tbl[i].filepath);
>> +                               return i;
>> +                       }
>> +                       *(int *)virtaddr = 0;
>> +               }
>> +
>> +
>>                  /* set shared flock on the file. */
>>                  if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
>> -                       RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
>> +                       RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s \n",
>>                                  __func__, strerror(errno));
>>                          close(fd);
>> -                       return -1;
>> +                       return i;
>>                  }
>>
>>                  close(fd);
> Maybe I missed something, but we are writing into some hugepage before
> the flock has been called.
> Are we sure there is nobody else using this hugepage ?
>
> Especially, can't this cause trouble to a primary process running if
> we start the exact same primary process ?
>

We lock the hugepage directory during eal_hugepage_info_init(), and we 
do not unlock
until we have finished eal_memory_init.

I think that takes care of that case.

Sergio

Re: [PATCH v4] eal: make hugetlb initialization more robust

[Sergio Gonzalez Monroy]

On 17/05/2016 17:40, Thomas Monjalon wrote:
> 2016-05-12 00:44, Jianfeng Tan:
>> This patch adds an option, --huge-trybest, to use a recover mechanism to
>> the case that there are not so many hugepages (declared in sysfs), which
>> can be used. It relys on a mem access to fault-in hugepages, and if fails
> relys -> relies
>
>> with SIGBUS, recover to previously saved stack environment with
>> siglongjmp().
>>
>> Besides, this solution fixes an issue when hugetlbfs is specified with an
>> option of size. Currently DPDK does not respect the quota of a hugetblfs
>> mount. It fails to init the EAL because it tries to map the number of free
>> hugepages in the system rather than using the number specified in the quota
>> for that mount.
> It looks to be a bug. Why adding an option?
> What is the benefit of the old behaviour, not using --try-best?

I do not see any benefit to the old behavior.
Given that we need the signal handling for the cgroup use case, I would 
be inclined to use
this method as the default instead of trying to figure out how many 
hugepages we have free, etc.

Thoughts?

Sergio

>> +static sigjmp_buf jmpenv;
>> +
>> +static void sigbus_handler(int signo __rte_unused)
>> +{
>> +	siglongjmp(jmpenv, 1);
>> +}
>> +
>> +/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
>> + * non-static local variable in the stack frame calling sigsetjmp might be
>> + * clobbered by a call to longjmp.
>> + */
>> +static int wrap_sigsetjmp(void)
>> +{
>> +	return sigsetjmp(jmpenv, 1);
>> +}
> Please add the word "huge" to these variables and functions.
>
>> +static struct sigaction action_old;
>> +static int need_recover;
>> +
>> +static void
>> +register_sigbus(void)
>> +{
>> +	sigset_t mask;
>> +	struct sigaction action;
>> +
>> +	sigemptyset(&mask);
>> +	sigaddset(&mask, SIGBUS);
>> +	action.sa_flags = 0;
>> +	action.sa_mask = mask;
>> +	action.sa_handler = sigbus_handler;
>> +
>> +	need_recover = !sigaction(SIGBUS, &action, &action_old);
>> +}
>> +
>> +static void
>> +recover_sigbus(void)
>> +{
>> +	if (need_recover) {
>> +		sigaction(SIGBUS, &action_old, NULL);
>> +		need_recover = 0;
>> +	}
>> +}
> Idem, Please add the word "huge".
>

Re: [PATCH v4] eal: make hugetlb initialization more robust

[David Marchand]

Hello Sergio,

On Wed, May 18, 2016 at 9:56 AM, Sergio Gonzalez Monroy
<sergio.gonzalez.monroy@intel.com> wrote:
> On 17/05/2016 17:39, David Marchand wrote:
>>> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c
>>> b/lib/librte_eal/linuxapp/eal/eal_memory.c
>>> index 5b9132c..8c77010 100644
>>> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
>>> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
>>> @@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>>>                          hugepg_tbl[i].final_va = virtaddr;
>>>                  }
>>>
>>> +               if (orig && internal_config.huge_trybest) {
>>> +                       /* In linux, hugetlb limitations, like cgroup,
>>> are
>>> +                        * enforced at fault time instead of mmap(), even
>>> +                        * with the option of MAP_POPULATE. Kernel will
>>> send
>>> +                        * a SIGBUS signal. To avoid to be killed, save
>>> stack
>>> +                        * environment here, if SIGBUS happens, we can
>>> jump
>>> +                        * back here.
>>> +                        */
>>> +                       if (wrap_sigsetjmp()) {
>>> +                               RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap
>>> more "
>>> +                                       "hugepages of size %u MB\n",
>>> +                                       (unsigned)(hugepage_sz /
>>> 0x100000));
>>> +                               munmap(virtaddr, hugepage_sz);
>>> +                               close(fd);
>>> +                               unlink(hugepg_tbl[i].filepath);
>>> +                               return i;
>>> +                       }
>>> +                       *(int *)virtaddr = 0;
>>> +               }
>>> +
>>> +
>>>                  /* set shared flock on the file. */
>>>                  if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
>>> -                       RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s
>>> \n",
>>> +                       RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s
>>> \n",
>>>                                  __func__, strerror(errno));
>>>                          close(fd);
>>> -                       return -1;
>>> +                       return i;
>>>                  }
>>>
>>>                  close(fd);
>>
>> Maybe I missed something, but we are writing into some hugepage before
>> the flock has been called.
>> Are we sure there is nobody else using this hugepage ?
>>
>> Especially, can't this cause trouble to a primary process running if
>> we start the exact same primary process ?
>>
>
> We lock the hugepage directory during eal_hugepage_info_init(), and we do
> not unlock
> until we have finished eal_memory_init.
>
> I think that takes care of that case.

Yes, thanks.

-- 
David Marchand

Re: [PATCH v4] eal: make hugetlb initialization more robust

[David Marchand]

On Wed, May 18, 2016 at 10:06 AM, Sergio Gonzalez Monroy
<sergio.gonzalez.monroy@intel.com> wrote:
> On 17/05/2016 17:40, Thomas Monjalon wrote:
>>
>> 2016-05-12 00:44, Jianfeng Tan:
>>>
>>> This patch adds an option, --huge-trybest, to use a recover mechanism to
>>> the case that there are not so many hugepages (declared in sysfs), which
>>> can be used. It relys on a mem access to fault-in hugepages, and if fails
>>
>> relys -> relies
>>
>>> with SIGBUS, recover to previously saved stack environment with
>>> siglongjmp().
>>>
>>> Besides, this solution fixes an issue when hugetlbfs is specified with an
>>> option of size. Currently DPDK does not respect the quota of a hugetblfs
>>> mount. It fails to init the EAL because it tries to map the number of
>>> free
>>> hugepages in the system rather than using the number specified in the
>>> quota
>>> for that mount.
>>
>> It looks to be a bug. Why adding an option?
>> What is the benefit of the old behaviour, not using --try-best?
>
>
> I do not see any benefit to the old behavior.
> Given that we need the signal handling for the cgroup use case, I would be
> inclined to use
> this method as the default instead of trying to figure out how many
> hugepages we have free, etc.

+1


-- 
David Marchand

Re: [PATCH] eal: add option --avail-cores to detect lcores

[David Marchand]

Hello Jianfeng,

On Wed, Mar 9, 2016 at 2:05 PM, Panu Matilainen <pmatilai@redhat.com> wrote:
> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
>>
>> Hi Panu,
>>
>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>>
>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>>
>>>> This patch adds option, --avail-cores, to use lcores which are available
>>>> by calling pthread_getaffinity_np() to narrow down detected cores before
>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>
>>>> Test example:
>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>         --avail-cores -m 1024
>>>>
>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>
>>>
>>> Hmm, to me this sounds like something that should be done always so
>>> there's no need for an option. Or if there's a chance it might do the
>>> wrong thing in some rare circumstance then perhaps there should be a
>>> disabler option instead?
>>
>>
>> Thanks for comments.
>>
>> Yes, there's a use case that we cannot handle.
>>
>> If we make it as default, DPDK applications may fail to start, when user
>> specifies a core in isolcpus and its parent process (say bash) has a
>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>> just blindly do pthread_setaffinity_np() and it always succeeds because
>> it always has root privilege to change any cpu affinity.
>>
>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>> flagged as undetected (in my older implementation) and leads to failure.
>> To make it correct, we would always add "taskset mask" (or other ways)
>> before DPDK application cmd lines.
>>
>> How do you think?
>
>
> I still think it sounds like something that should be done by default and
> maybe be overridable with some flag, rather than the other way around.
> Another alternative might be detecting the cores always but if running as
> root, override but with a warning.
>
> But I dont know, just wondering. To look at it from another angle: why would
> somebody use this new --avail-cores option and in what situation, if things
> "just work" otherwise anyway?

+1 and I don't even see why we should have an option to disable this,
since taskset would do the job.

Looking at your special case, if the user did set an isolcpus option
for another use, with no -c/-l, I understand the dpdk application
won't care too much about it.
So, this seems like somehow rude to the rest of the system and unwanted.

We can still help the user starting its application as root (without
taskset) by adding a warning message if a requested cpu (-c / -l ..)
is not part of the available cpus.


-- 
David Marchand

Re: [PATCH v4] eal: make hugetlb initialization more robust

[Tan, Jianfeng]

Hi David,


On 5/18/2016 12:39 AM, David Marchand wrote:
> Hello Jianfeng,
>
> On Thu, May 12, 2016 at 2:44 AM, Jianfeng Tan <jianfeng.tan@intel.com> wrote:
>> This patch adds an option, --huge-trybest, to use a recover mechanism to
>> the case that there are not so many hugepages (declared in sysfs), which
>> can be used. It relys on a mem access to fault-in hugepages, and if fails
>> with SIGBUS, recover to previously saved stack environment with
>> siglongjmp().
>>
>> Besides, this solution fixes an issue when hugetlbfs is specified with an
>> option of size. Currently DPDK does not respect the quota of a hugetblfs
>> mount. It fails to init the EAL because it tries to map the number of free
>> hugepages in the system rather than using the number specified in the quota
>> for that mount.
>>
>> It's still an open issue with CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS. Under
>> this case (such as IVSHMEM target), having hugetlbfs mounts with quota will
>> fail to remap hugepages as it relies on having mapped all free hugepages
>> in the system.
>
>
>
>> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
>> index 5b9132c..8c77010 100644
>> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
>> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
>> @@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
>>                          hugepg_tbl[i].final_va = virtaddr;
>>                  }
>>
>> +               if (orig && internal_config.huge_trybest) {
>> +                       /* In linux, hugetlb limitations, like cgroup, are
>> +                        * enforced at fault time instead of mmap(), even
>> +                        * with the option of MAP_POPULATE. Kernel will send
>> +                        * a SIGBUS signal. To avoid to be killed, save stack
>> +                        * environment here, if SIGBUS happens, we can jump
>> +                        * back here.
>> +                        */
>> +                       if (wrap_sigsetjmp()) {
>> +                               RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
>> +                                       "hugepages of size %u MB\n",
>> +                                       (unsigned)(hugepage_sz / 0x100000));
> For such a case case, maybe having some warning log message when it
> fails would help the user.
> + a known issue in the release notes ?

Do you mean when sigbus is triggered, like here, warn the user that "it 
fails to hold all free hugepages as sysfs shows", and
#ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
/*we need to return error from rte_eal_init_memory */
#endif

Thanks,
Jianfeng

Re: [PATCH v4] eal: make hugetlb initialization more robust

[Tan, Jianfeng]

Hi Thomas & Sergio,


On 5/18/2016 4:06 PM, Sergio Gonzalez Monroy wrote:
> On 17/05/2016 17:40, Thomas Monjalon wrote:
>> 2016-05-12 00:44, Jianfeng Tan:
>>> This patch adds an option, --huge-trybest, to use a recover 
>>> mechanism to
>>> the case that there are not so many hugepages (declared in sysfs), 
>>> which
>>> can be used. It relys on a mem access to fault-in hugepages, and if 
>>> fails
>> relys -> relies
>>
>>> with SIGBUS, recover to previously saved stack environment with
>>> siglongjmp().
>>>
>>> Besides, this solution fixes an issue when hugetlbfs is specified 
>>> with an
>>> option of size. Currently DPDK does not respect the quota of a 
>>> hugetblfs
>>> mount. It fails to init the EAL because it tries to map the number 
>>> of free
>>> hugepages in the system rather than using the number specified in 
>>> the quota
>>> for that mount.
>> It looks to be a bug. Why adding an option?
>> What is the benefit of the old behaviour, not using --try-best?
>
> I do not see any benefit to the old behavior.
> Given that we need the signal handling for the cgroup use case, I 
> would be inclined to use
> this method as the default instead of trying to figure out how many 
> hugepages we have free, etc.
>
> Thoughts?

I tend to use this method as the default too, with some warning logs as 
suggested by David, and return error from rte_eal_memory() when sigbus 
is triggered under the case of RTE_EAL_SINGLE_FILE_SEGMENTS.

Thomas, all other trivial issues will be fixed in next version. Thank you!

Thanks,
Jianfeng

>
> Sergio
>
>>> +static sigjmp_buf jmpenv;
>>> +
>>> +static void sigbus_handler(int signo __rte_unused)
>>> +{
>>> +    siglongjmp(jmpenv, 1);
>>> +}
>>> +
>>> +/* Put setjmp into a wrap method to avoid compiling error. Any 
>>> non-volatile,
>>> + * non-static local variable in the stack frame calling sigsetjmp 
>>> might be
>>> + * clobbered by a call to longjmp.
>>> + */
>>> +static int wrap_sigsetjmp(void)
>>> +{
>>> +    return sigsetjmp(jmpenv, 1);
>>> +}
>> Please add the word "huge" to these variables and functions.
>>
>>> +static struct sigaction action_old;
>>> +static int need_recover;
>>> +
>>> +static void
>>> +register_sigbus(void)
>>> +{
>>> +    sigset_t mask;
>>> +    struct sigaction action;
>>> +
>>> +    sigemptyset(&mask);
>>> +    sigaddset(&mask, SIGBUS);
>>> +    action.sa_flags = 0;
>>> +    action.sa_mask = mask;
>>> +    action.sa_handler = sigbus_handler;
>>> +
>>> +    need_recover = !sigaction(SIGBUS, &action, &action_old);
>>> +}
>>> +
>>> +static void
>>> +recover_sigbus(void)
>>> +{
>>> +    if (need_recover) {
>>> +        sigaction(SIGBUS, &action_old, NULL);
>>> +        need_recover = 0;
>>> +    }
>>> +}
>> Idem, Please add the word "huge".
>>
>

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

Hi David,


On 5/18/2016 8:46 PM, David Marchand wrote:
> Hello Jianfeng,
>
> On Wed, Mar 9, 2016 at 2:05 PM, Panu Matilainen <pmatilai@redhat.com> wrote:
>> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
>>> Hi Panu,
>>>
>>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
>>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
>>>>> This patch adds option, --avail-cores, to use lcores which are available
>>>>> by calling pthread_getaffinity_np() to narrow down detected cores before
>>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
>>>>>
>>>>> Test example:
>>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
>>>>>          --avail-cores -m 1024
>>>>>
>>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
>>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
>>>>
>>>> Hmm, to me this sounds like something that should be done always so
>>>> there's no need for an option. Or if there's a chance it might do the
>>>> wrong thing in some rare circumstance then perhaps there should be a
>>>> disabler option instead?
>>>
>>> Thanks for comments.
>>>
>>> Yes, there's a use case that we cannot handle.
>>>
>>> If we make it as default, DPDK applications may fail to start, when user
>>> specifies a core in isolcpus and its parent process (say bash) has a
>>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
>>> just blindly do pthread_setaffinity_np() and it always succeeds because
>>> it always has root privilege to change any cpu affinity.
>>>
>>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
>>> flagged as undetected (in my older implementation) and leads to failure.
>>> To make it correct, we would always add "taskset mask" (or other ways)
>>> before DPDK application cmd lines.
>>>
>>> How do you think?
>>
>> I still think it sounds like something that should be done by default and
>> maybe be overridable with some flag, rather than the other way around.
>> Another alternative might be detecting the cores always but if running as
>> root, override but with a warning.
>>
>> But I dont know, just wondering. To look at it from another angle: why would
>> somebody use this new --avail-cores option and in what situation, if things
>> "just work" otherwise anyway?
> +1 and I don't even see why we should have an option to disable this,
> since taskset would do the job.
>
> Looking at your special case, if the user did set an isolcpus option
> for another use, with no -c/-l, I understand the dpdk application
> won't care too much about it.
> So, this seems like somehow rude to the rest of the system and unwanted.

The case you mentioned above is not the case I mean. But you make your 
point about this one.
The case I originally mean: user sets an isolcpus option for DPDK 
applications. Originally, DPDK apps would be started without any 
problem. But for now, fail to start them because the required cores are 
excluded before -c/-l. As per your comments following, we can add a 
warning message (or should we quit on this situation?). But it indeed 
has an effect on old users (they should changed to use "taskset 
./dpdk_app ..."). Do you think it's a problem?

Thanks,
Jianfeng


>
> We can still help the user starting its application as root (without
> taskset) by adding a warning message if a requested cpu (-c / -l ..)
> is not part of the available cpus.
>
>

[PATCH v5] eal: fix allocating all free hugepages

[Jianfeng Tan]

EAL memory init allocates all free hugepages of the whole system,
which seen from sysfs, even when applications do not ask so many.
When there is a limitation on how many hugepages an application can
use (such as cgroup.hugetlb), or hugetlbfs is specified with an
option of size (exceeding the quota of the fs), it just fails to
start even there are enough hugepages allocated.

To fix above issue, this patch:
 - Changes the logic to continue memory init to see if hugetlb
   requirement of application can be addressed by already allocated
   hugepages.
 - To make sure each hugepage is allocated successfully, we add a
   recover mechanism, which relies on a mem access to fault-in
   hugepages, and if it fails with SIGBUS, recover to previously
   saved stack environment with siglongjmp().

For the case of CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS (enabled by
default when compiling IVSHMEM target), it's indispensable to
mapp all free hugepages in the system. Under this case, it fails
to start when allocating fails.

Test example:
  a. cgcreate -g hugetlb:/test-subgroup
  b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
  c. cgexec -g hugetlb:test-subgroup \
          ./examples/helloworld/build/helloworld -c 0x2 -n 4

       
Fixes: af75078fece ("first public release")

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v5:
 - Make this method as default instead of using an option.
 - When SIGBUS is triggered in the case of RTE_EAL_SINGLE_FILE_SEGMENTS,
   just return error.
 - Add prefix "huge_" to newly added function and static variables.
 - Move the internal_config.memory assignment after the page allocations.
v4:
 - Change map_all_hugepages to return unsigned instead of int.
v3:
 - Reword commit message to include it fixes the hugetlbfs quota issue.
 - setjmp -> sigsetjmp.
 - Fix RTE_LOG complaint from ERR to DEBUG as it does not mean init error
   so far.
 - Fix the second map_all_hugepages's return value check.
v2:
 - Address the compiling error by move setjmp into a wrap method.

 lib/librte_eal/linuxapp/eal/eal.c        |  20 -----
 lib/librte_eal/linuxapp/eal/eal_memory.c | 138 ++++++++++++++++++++++++++++---
 2 files changed, 125 insertions(+), 33 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8aafd51..4a8dfbd 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -465,24 +465,6 @@ eal_parse_vfio_intr(const char *mode)
 	return -1;
 }
 
-static inline size_t
-eal_get_hugepage_mem_size(void)
-{
-	uint64_t size = 0;
-	unsigned i, j;
-
-	for (i = 0; i < internal_config.num_hugepage_sizes; i++) {
-		struct hugepage_info *hpi = &internal_config.hugepage_info[i];
-		if (hpi->hugedir != NULL) {
-			for (j = 0; j < RTE_MAX_NUMA_NODES; j++) {
-				size += hpi->hugepage_sz * hpi->num_pages[j];
-			}
-		}
-	}
-
-	return (size < SIZE_MAX) ? (size_t)(size) : SIZE_MAX;
-}
-
 /* Parse the arguments for --log-level only */
 static void
 eal_log_level_parse(int argc, char **argv)
@@ -766,8 +748,6 @@ rte_eal_init(int argc, char **argv)
 	if (internal_config.memory == 0 && internal_config.force_sockets == 0) {
 		if (internal_config.no_hugetlbfs)
 			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
-		else
-			internal_config.memory = eal_get_hugepage_mem_size();
 	}
 
 	if (internal_config.vmware_tsc_map == 1) {
diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 5b9132c..dc6f49b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,21 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf huge_jmpenv;
+
+static void huge_sigbus_handler(int signo __rte_unused)
+{
+	siglongjmp(huge_jmpenv, 1);
+}
+
+/* Put setjmp into a wrap method to avoid compiling error. Any non-volatile,
+ * non-static local variable in the stack frame calling sigsetjmp might be
+ * clobbered by a call to longjmp.
+ */
+static int huge_wrap_sigsetjmp(void)
+{
+	return sigsetjmp(huge_jmpenv, 1);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the
@@ -316,7 +333,7 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
  * in hugepg_tbl[i].final_va. The second mapping (when orig is 0) tries to
  * map continguous physical blocks in contiguous virtual blocks.
  */
-static int
+static unsigned
 map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		struct hugepage_info *hpi, int orig)
 {
@@ -394,9 +411,9 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		/* try to create hugepage file */
 		fd = open(hugepg_tbl[i].filepath, O_CREAT | O_RDWR, 0755);
 		if (fd < 0) {
-			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		/* map the segment, and populate page tables,
@@ -404,10 +421,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
 				MAP_SHARED | MAP_POPULATE, fd, 0);
 		if (virtaddr == MAP_FAILED) {
-			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
@@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			hugepg_tbl[i].final_va = virtaddr;
 		}
 
+		if (orig) {
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (huge_wrap_sigsetjmp()) {
+				RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				unlink(hugepg_tbl[i].filepath);
+				return i;
+			}
+			*(int *)virtaddr = 0;
+		}
+
+
 		/* set shared flock on the file. */
 		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
-			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
+			RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -430,7 +468,8 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1036,6 +1075,51 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static inline size_t
+eal_get_hugepage_mem_size(void)
+{
+	uint64_t size = 0;
+	unsigned i, j;
+
+	for (i = 0; i < internal_config.num_hugepage_sizes; i++) {
+		struct hugepage_info *hpi = &internal_config.hugepage_info[i];
+		if (hpi->hugedir != NULL) {
+			for (j = 0; j < RTE_MAX_NUMA_NODES; j++) {
+				size += hpi->hugepage_sz * hpi->num_pages[j];
+			}
+		}
+	}
+
+	return (size < SIZE_MAX) ? (size_t)(size) : SIZE_MAX;
+}
+
+static struct sigaction huge_action_old;
+static int huge_need_recover;
+
+static void
+huge_register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = huge_sigbus_handler;
+
+	huge_need_recover = !sigaction(SIGBUS, &action, &huge_action_old);
+}
+
+static void
+huge_recover_sigbus(void)
+{
+	if (huge_need_recover) {
+		sigaction(SIGBUS, &huge_action_old, NULL);
+		huge_need_recover = 0;
+	}
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1122,8 +1206,11 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	huge_register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		unsigned pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1137,10 +1224,28 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+#ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
+			RTE_LOG(ERR, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
 			goto fail;
+#else
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+
+			int pages = pages_old - pages_new;
+
+			nr_hugepages -= pages;
+			hpi->num_pages[0] = pages_new;
+			if (pages_new == 0)
+				continue;
+#endif
 		}
 
 		/* find physical addresses and sockets for each hugepage */
@@ -1172,8 +1277,9 @@ rte_eal_hugepage_init(void)
 		hp_offset += new_pages_count[i];
 #else
 		/* remap all hugepages */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to remap %u MB pages\n",
+		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) !=
+		    hpi->num_pages[0]) {
+			RTE_LOG(ERR, EAL, "Failed to remap %u MB pages\n",
 					(unsigned)(hpi->hugepage_sz / 0x100000));
 			goto fail;
 		}
@@ -1187,6 +1293,11 @@ rte_eal_hugepage_init(void)
 #endif
 	}
 
+	huge_recover_sigbus();
+
+	if (internal_config.memory == 0 && internal_config.force_sockets == 0)
+		internal_config.memory = eal_get_hugepage_mem_size();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) {
@@ -1373,6 +1484,7 @@ rte_eal_hugepage_init(void)
 	return 0;
 
 fail:
+	huge_recover_sigbus();
 	free(tmp_hp);
 	return -1;
 }
-- 
2.1.4

Re: [PATCH v5] eal: fix allocating all free hugepages

[Pei, Yulong]

Tested-by: Yulong Pei <Yulong.pei@intel.com>

1. Run dpdk app with multiple mount points, it works as expected.
2. Create new cgroup with limited hugepages like the following, and Run dpdk app with the newly created cgroup, it works as expected.

#cgcreate -g hugetlb:/test-subgroup
# cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
# cgexec -g hugetlb:test-subgroup ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 -- -i

Best Regards
Yulong Pei

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jianfeng Tan
Sent: Tuesday, May 31, 2016 11:37 AM
To: dev@dpdk.org
Cc: Gonzalez Monroy, Sergio <sergio.gonzalez.monroy@intel.com>; nhorman@tuxdriver.com; david.marchand@6wind.com; thomas.monjalon@6wind.com; Tan, Jianfeng <jianfeng.tan@intel.com>
Subject: [dpdk-dev] [PATCH v5] eal: fix allocating all free hugepages

EAL memory init allocates all free hugepages of the whole system, which seen from sysfs, even when applications do not ask so many.
When there is a limitation on how many hugepages an application can use (such as cgroup.hugetlb), or hugetlbfs is specified with an option of size (exceeding the quota of the fs), it just fails to start even there are enough hugepages allocated.

To fix above issue, this patch:
 - Changes the logic to continue memory init to see if hugetlb
   requirement of application can be addressed by already allocated
   hugepages.
 - To make sure each hugepage is allocated successfully, we add a
   recover mechanism, which relies on a mem access to fault-in
   hugepages, and if it fails with SIGBUS, recover to previously
   saved stack environment with siglongjmp().

For the case of CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS (enabled by default when compiling IVSHMEM target), it's indispensable to mapp all free hugepages in the system. Under this case, it fails to start when allocating fails.

Test example:
  a. cgcreate -g hugetlb:/test-subgroup
  b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
  c. cgexec -g hugetlb:test-subgroup \
          ./examples/helloworld/build/helloworld -c 0x2 -n 4

       
Fixes: af75078fece ("first public release")

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v5:
 - Make this method as default instead of using an option.
 - When SIGBUS is triggered in the case of RTE_EAL_SINGLE_FILE_SEGMENTS,
   just return error.
 - Add prefix "huge_" to newly added function and static variables.
 - Move the internal_config.memory assignment after the page allocations.
v4:
 - Change map_all_hugepages to return unsigned instead of int.
v3:
 - Reword commit message to include it fixes the hugetlbfs quota issue.
 - setjmp -> sigsetjmp.
 - Fix RTE_LOG complaint from ERR to DEBUG as it does not mean init error
   so far.
 - Fix the second map_all_hugepages's return value check.
v2:
 - Address the compiling error by move setjmp into a wrap method.

 lib/librte_eal/linuxapp/eal/eal.c        |  20 -----
 lib/librte_eal/linuxapp/eal/eal_memory.c | 138 ++++++++++++++++++++++++++++---
 2 files changed, 125 insertions(+), 33 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8aafd51..4a8dfbd 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -465,24 +465,6 @@ eal_parse_vfio_intr(const char *mode)
 	return -1;
 }
 
-static inline size_t
-eal_get_hugepage_mem_size(void)
-{
-	uint64_t size = 0;
-	unsigned i, j;
-
-	for (i = 0; i < internal_config.num_hugepage_sizes; i++) {
-		struct hugepage_info *hpi = &internal_config.hugepage_info[i];
-		if (hpi->hugedir != NULL) {
-			for (j = 0; j < RTE_MAX_NUMA_NODES; j++) {
-				size += hpi->hugepage_sz * hpi->num_pages[j];
-			}
-		}
-	}
-
-	return (size < SIZE_MAX) ? (size_t)(size) : SIZE_MAX;
-}
-
 /* Parse the arguments for --log-level only */  static void  eal_log_level_parse(int argc, char **argv) @@ -766,8 +748,6 @@ rte_eal_init(int argc, char **argv)
 	if (internal_config.memory == 0 && internal_config.force_sockets == 0) {
 		if (internal_config.no_hugetlbfs)
 			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
-		else
-			internal_config.memory = eal_get_hugepage_mem_size();
 	}
 
 	if (internal_config.vmware_tsc_map == 1) { diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
index 5b9132c..dc6f49b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_memory.c
+++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
@@ -80,6 +80,8 @@
 #include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/time.h>
+#include <signal.h>
+#include <setjmp.h>
 
 #include <rte_log.h>
 #include <rte_memory.h>
@@ -309,6 +311,21 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
 	return addr;
 }
 
+static sigjmp_buf huge_jmpenv;
+
+static void huge_sigbus_handler(int signo __rte_unused) {
+	siglongjmp(huge_jmpenv, 1);
+}
+
+/* Put setjmp into a wrap method to avoid compiling error. Any 
+non-volatile,
+ * non-static local variable in the stack frame calling sigsetjmp might 
+be
+ * clobbered by a call to longjmp.
+ */
+static int huge_wrap_sigsetjmp(void)
+{
+	return sigsetjmp(huge_jmpenv, 1);
+}
 /*
  * Mmap all hugepages of hugepage table: it first open a file in
  * hugetlbfs, then mmap() hugepage_sz data in it. If orig is set, the @@ -316,7 +333,7 @@ get_virtual_area(size_t *size, size_t hugepage_sz)
  * in hugepg_tbl[i].final_va. The second mapping (when orig is 0) tries to
  * map continguous physical blocks in contiguous virtual blocks.
  */
-static int
+static unsigned
 map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		struct hugepage_info *hpi, int orig)
 {
@@ -394,9 +411,9 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		/* try to create hugepage file */
 		fd = open(hugepg_tbl[i].filepath, O_CREAT | O_RDWR, 0755);
 		if (fd < 0) {
-			RTE_LOG(ERR, EAL, "%s(): open failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): open failed: %s\n", __func__,
 					strerror(errno));
-			return -1;
+			return i;
 		}
 
 		/* map the segment, and populate page tables, @@ -404,10 +421,10 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE,
 				MAP_SHARED | MAP_POPULATE, fd, 0);
 		if (virtaddr == MAP_FAILED) {
-			RTE_LOG(ERR, EAL, "%s(): mmap failed: %s\n", __func__,
+			RTE_LOG(DEBUG, EAL, "%s(): mmap failed: %s\n", __func__,
 					strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		if (orig) {
@@ -417,12 +434,33 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 			hugepg_tbl[i].final_va = virtaddr;
 		}
 
+		if (orig) {
+			/* In linux, hugetlb limitations, like cgroup, are
+			 * enforced at fault time instead of mmap(), even
+			 * with the option of MAP_POPULATE. Kernel will send
+			 * a SIGBUS signal. To avoid to be killed, save stack
+			 * environment here, if SIGBUS happens, we can jump
+			 * back here.
+			 */
+			if (huge_wrap_sigsetjmp()) {
+				RTE_LOG(DEBUG, EAL, "SIGBUS: Cannot mmap more "
+					"hugepages of size %u MB\n",
+					(unsigned)(hugepage_sz / 0x100000));
+				munmap(virtaddr, hugepage_sz);
+				close(fd);
+				unlink(hugepg_tbl[i].filepath);
+				return i;
+			}
+			*(int *)virtaddr = 0;
+		}
+
+
 		/* set shared flock on the file. */
 		if (flock(fd, LOCK_SH | LOCK_NB) == -1) {
-			RTE_LOG(ERR, EAL, "%s(): Locking file failed:%s \n",
+			RTE_LOG(DEBUG, EAL, "%s(): Locking file failed:%s \n",
 				__func__, strerror(errno));
 			close(fd);
-			return -1;
+			return i;
 		}
 
 		close(fd);
@@ -430,7 +468,8 @@ map_all_hugepages(struct hugepage_file *hugepg_tbl,
 		vma_addr = (char *)vma_addr + hugepage_sz;
 		vma_len -= hugepage_sz;
 	}
-	return 0;
+
+	return i;
 }
 
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
@@ -1036,6 +1075,51 @@ calc_num_pages_per_socket(uint64_t * memory,
 	return total_num_pages;
 }
 
+static inline size_t
+eal_get_hugepage_mem_size(void)
+{
+	uint64_t size = 0;
+	unsigned i, j;
+
+	for (i = 0; i < internal_config.num_hugepage_sizes; i++) {
+		struct hugepage_info *hpi = &internal_config.hugepage_info[i];
+		if (hpi->hugedir != NULL) {
+			for (j = 0; j < RTE_MAX_NUMA_NODES; j++) {
+				size += hpi->hugepage_sz * hpi->num_pages[j];
+			}
+		}
+	}
+
+	return (size < SIZE_MAX) ? (size_t)(size) : SIZE_MAX; }
+
+static struct sigaction huge_action_old; static int huge_need_recover;
+
+static void
+huge_register_sigbus(void)
+{
+	sigset_t mask;
+	struct sigaction action;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGBUS);
+	action.sa_flags = 0;
+	action.sa_mask = mask;
+	action.sa_handler = huge_sigbus_handler;
+
+	huge_need_recover = !sigaction(SIGBUS, &action, &huge_action_old); }
+
+static void
+huge_recover_sigbus(void)
+{
+	if (huge_need_recover) {
+		sigaction(SIGBUS, &huge_action_old, NULL);
+		huge_need_recover = 0;
+	}
+}
+
 /*
  * Prepare physical memory mapping: fill configuration structure with
  * these infos, return 0 on success.
@@ -1122,8 +1206,11 @@ rte_eal_hugepage_init(void)
 
 	hp_offset = 0; /* where we start the current page size entries */
 
+	huge_register_sigbus();
+
 	/* map all hugepages and sort them */
 	for (i = 0; i < (int)internal_config.num_hugepage_sizes; i ++){
+		unsigned pages_old, pages_new;
 		struct hugepage_info *hpi;
 
 		/*
@@ -1137,10 +1224,28 @@ rte_eal_hugepage_init(void)
 			continue;
 
 		/* map all hugepages available */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 1) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to mmap %u MB hugepages\n",
-					(unsigned)(hpi->hugepage_sz / 0x100000));
+		pages_old = hpi->num_pages[0];
+		pages_new = map_all_hugepages(&tmp_hp[hp_offset], hpi, 1);
+		if (pages_new < pages_old) {
+#ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
+			RTE_LOG(ERR, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
 			goto fail;
+#else
+			RTE_LOG(DEBUG, EAL,
+				"%d not %d hugepages of size %u MB allocated\n",
+				pages_new, pages_old,
+				(unsigned)(hpi->hugepage_sz / 0x100000));
+
+			int pages = pages_old - pages_new;
+
+			nr_hugepages -= pages;
+			hpi->num_pages[0] = pages_new;
+			if (pages_new == 0)
+				continue;
+#endif
 		}
 
 		/* find physical addresses and sockets for each hugepage */ @@ -1172,8 +1277,9 @@ rte_eal_hugepage_init(void)
 		hp_offset += new_pages_count[i];
 #else
 		/* remap all hugepages */
-		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) < 0){
-			RTE_LOG(DEBUG, EAL, "Failed to remap %u MB pages\n",
+		if (map_all_hugepages(&tmp_hp[hp_offset], hpi, 0) !=
+		    hpi->num_pages[0]) {
+			RTE_LOG(ERR, EAL, "Failed to remap %u MB pages\n",
 					(unsigned)(hpi->hugepage_sz / 0x100000));
 			goto fail;
 		}
@@ -1187,6 +1293,11 @@ rte_eal_hugepage_init(void)  #endif
 	}
 
+	huge_recover_sigbus();
+
+	if (internal_config.memory == 0 && internal_config.force_sockets == 0)
+		internal_config.memory = eal_get_hugepage_mem_size();
+
 #ifdef RTE_EAL_SINGLE_FILE_SEGMENTS
 	nr_hugefiles = 0;
 	for (i = 0; i < (int) internal_config.num_hugepage_sizes; i++) { @@ -1373,6 +1484,7 @@ rte_eal_hugepage_init(void)
 	return 0;
 
 fail:
+	huge_recover_sigbus();
 	free(tmp_hp);
 	return -1;
 }
--
2.1.4

Re: [PATCH v5] eal: fix allocating all free hugepages

[Sergio Gonzalez Monroy]

On 31/05/2016 04:37, Jianfeng Tan wrote:
> EAL memory init allocates all free hugepages of the whole system,
> which seen from sysfs, even when applications do not ask so many.
> When there is a limitation on how many hugepages an application can
> use (such as cgroup.hugetlb), or hugetlbfs is specified with an
> option of size (exceeding the quota of the fs), it just fails to
> start even there are enough hugepages allocated.
>
> To fix above issue, this patch:
>   - Changes the logic to continue memory init to see if hugetlb
>     requirement of application can be addressed by already allocated
>     hugepages.
>   - To make sure each hugepage is allocated successfully, we add a
>     recover mechanism, which relies on a mem access to fault-in
>     hugepages, and if it fails with SIGBUS, recover to previously
>     saved stack environment with siglongjmp().
>
> For the case of CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS (enabled by
> default when compiling IVSHMEM target), it's indispensable to
> mapp all free hugepages in the system. Under this case, it fails
> to start when allocating fails.
>
> Test example:
>    a. cgcreate -g hugetlb:/test-subgroup
>    b. cgset -r hugetlb.1GB.limit_in_bytes=2147483648 test-subgroup
>    c. cgexec -g hugetlb:test-subgroup \
>            ./examples/helloworld/build/helloworld -c 0x2 -n 4
>
>         
> Fixes: af75078fece ("first public release")
>
> Signed-off-by: Jianfeng Tan<jianfeng.tan@intel.com>
> Acked-by: Neil Horman<nhorman@tuxdriver.com>
> ---
> v5:
>   - Make this method as default instead of using an option.
>   - When SIGBUS is triggered in the case of RTE_EAL_SINGLE_FILE_SEGMENTS,
>     just return error.
>   - Add prefix "huge_" to newly added function and static variables.
>   - Move the internal_config.memory assignment after the page allocations.
> v4:
>   - Change map_all_hugepages to return unsigned instead of int.
> v3:
>   - Reword commit message to include it fixes the hugetlbfs quota issue.
>   - setjmp -> sigsetjmp.
>   - Fix RTE_LOG complaint from ERR to DEBUG as it does not mean init error
>     so far.
>   - Fix the second map_all_hugepages's return value check.
> v2:
>   - Address the compiling error by move setjmp into a wrap method.
>
>   lib/librte_eal/linuxapp/eal/eal.c        |  20 -----
>   lib/librte_eal/linuxapp/eal/eal_memory.c | 138 ++++++++++++++++++++++++++++---
>   2 files changed, 125 insertions(+), 33 deletions(-)
>

Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

Re: [PATCH v5] eal: fix allocating all free hugepages

[Thomas Monjalon]

> > EAL memory init allocates all free hugepages of the whole system,
> > which seen from sysfs, even when applications do not ask so many.
> > When there is a limitation on how many hugepages an application can
> > use (such as cgroup.hugetlb), or hugetlbfs is specified with an
> > option of size (exceeding the quota of the fs), it just fails to
> > start even there are enough hugepages allocated.
> >
> > To fix above issue, this patch:
> >   - Changes the logic to continue memory init to see if hugetlb
> >     requirement of application can be addressed by already allocated
> >     hugepages.
> >   - To make sure each hugepage is allocated successfully, we add a
> >     recover mechanism, which relies on a mem access to fault-in
> >     hugepages, and if it fails with SIGBUS, recover to previously
> >     saved stack environment with siglongjmp().
> >
> > For the case of CONFIG_RTE_EAL_SINGLE_FILE_SEGMENTS (enabled by
> > default when compiling IVSHMEM target), it's indispensable to
> > mapp all free hugepages in the system. Under this case, it fails
> > to start when allocating fails.
[...]
> > Signed-off-by: Jianfeng Tan<jianfeng.tan@intel.com>
> > Acked-by: Neil Horman<nhorman@tuxdriver.com>
> 
> Acked-by: Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>

Applied, thanks

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Thomas Monjalon]

2016-05-19 10:25, Tan, Jianfeng:
> On 5/18/2016 8:46 PM, David Marchand wrote:
> > On Wed, Mar 9, 2016 at 2:05 PM, Panu Matilainen <pmatilai@redhat.com> wrote:
> >> On 03/08/2016 07:38 PM, Tan, Jianfeng wrote:
> >>> On 3/8/2016 4:54 PM, Panu Matilainen wrote:
> >>>> On 03/04/2016 12:05 PM, Jianfeng Tan wrote:
> >>>>> This patch adds option, --avail-cores, to use lcores which are available
> >>>>> by calling pthread_getaffinity_np() to narrow down detected cores before
> >>>>> parsing coremask (-c), corelist (-l), and coremap (--lcores).
> >>>>>
> >>>>> Test example:
> >>>>> $ taskset 0xc0000 ./examples/helloworld/build/helloworld \
> >>>>>          --avail-cores -m 1024
> >>>>>
> >>>>> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> >>>>> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> >>>>
> >>>> Hmm, to me this sounds like something that should be done always so
> >>>> there's no need for an option. Or if there's a chance it might do the
> >>>> wrong thing in some rare circumstance then perhaps there should be a
> >>>> disabler option instead?
> >>>
> >>> Thanks for comments.
> >>>
> >>> Yes, there's a use case that we cannot handle.
> >>>
> >>> If we make it as default, DPDK applications may fail to start, when user
> >>> specifies a core in isolcpus and its parent process (say bash) has a
> >>> cpuset affinity that excludes isolcpus. Originally, DPDK applications
> >>> just blindly do pthread_setaffinity_np() and it always succeeds because
> >>> it always has root privilege to change any cpu affinity.
> >>>
> >>> Now, if we do the checking in rte_eal_cpu_init(), those lcores will be
> >>> flagged as undetected (in my older implementation) and leads to failure.
> >>> To make it correct, we would always add "taskset mask" (or other ways)
> >>> before DPDK application cmd lines.
> >>>
> >>> How do you think?
> >>
> >> I still think it sounds like something that should be done by default and
> >> maybe be overridable with some flag, rather than the other way around.
> >> Another alternative might be detecting the cores always but if running as
> >> root, override but with a warning.
> >>
> >> But I dont know, just wondering. To look at it from another angle: why would
> >> somebody use this new --avail-cores option and in what situation, if things
> >> "just work" otherwise anyway?
> > +1 and I don't even see why we should have an option to disable this,
> > since taskset would do the job.
> >
> > Looking at your special case, if the user did set an isolcpus option
> > for another use, with no -c/-l, I understand the dpdk application
> > won't care too much about it.
> > So, this seems like somehow rude to the rest of the system and unwanted.
> 
> The case you mentioned above is not the case I mean. But you make your 
> point about this one.
> The case I originally mean: user sets an isolcpus option for DPDK 
> applications. Originally, DPDK apps would be started without any 
> problem. But for now, fail to start them because the required cores are 
> excluded before -c/-l. As per your comments following, we can add a 
> warning message (or should we quit on this situation?). But it indeed 
> has an effect on old users (they should changed to use "taskset 
> ./dpdk_app ..."). Do you think it's a problem?

There is no activity on this patch.
Jianfeng, do not hesitate to ping if needed.
Should we class this patch as "changes requested"?

Re: [PATCH] eal: add option --avail-cores to detect lcores

[Tan, Jianfeng]

Hi Thomas,

> > >
> > > Looking at your special case, if the user did set an isolcpus option
> > > for another use, with no -c/-l, I understand the dpdk application
> > > won't care too much about it.
> > > So, this seems like somehow rude to the rest of the system and
> unwanted.
> >
> > The case you mentioned above is not the case I mean. But you make your
> > point about this one.
> > The case I originally mean: user sets an isolcpus option for DPDK
> > applications. Originally, DPDK apps would be started without any
> > problem. But for now, fail to start them because the required cores are
> > excluded before -c/-l. As per your comments following, we can add a
> > warning message (or should we quit on this situation?). But it indeed
> > has an effect on old users (they should changed to use "taskset
> > ./dpdk_app ..."). Do you think it's a problem?
> 
> There is no activity on this patch.
> Jianfeng, do not hesitate to ping if needed.
> Should we class this patch as "changes requested"?

Yes, according to latest comments, it should be classified as "changes requested" (I've done that).

I'll resent a new version.

Thanks,
Jianfeng

[PATCH v2] eal: restrict cores detection

[Jianfeng Tan]

This patch uses pthread_getaffinity_np() to narrow down detected
cores before parsing coremask (-c), corelist (-l), and coremap
(--lcores).

The purpose of this patch is to leave out these core related options
when DPDK applications are deployed under container env, so that
users only specify core restriction as starting the instance.

Note: previously, some users are using isolated CPUs, which could
be excluded by default. Please add commands like taskset to use
those cores.

Test example:
$ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v2:
  - Make it as default instead of adding the new options.
 lib/librte_eal/common/eal_common_lcore.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
index 2cd4132..62e4f67 100644
--- a/lib/librte_eal/common/eal_common_lcore.c
+++ b/lib/librte_eal/common/eal_common_lcore.c
@@ -57,6 +57,14 @@ rte_eal_cpu_init(void)
 	struct rte_config *config = rte_eal_get_configuration();
 	unsigned lcore_id;
 	unsigned count = 0;
+	rte_cpuset_t cs;
+	pthread_t tid = pthread_self();
+
+	/* Add below method to obtain core restrictions, like ulimit,
+	 * cgroup.cpuset, etc. Will not use those cores, which are rebuffed.
+	 */
+	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cs) < 0)
+		CPU_ZERO(&cs);
 
 	/*
 	 * Parse the maximum set of logical cores, detect the subset of running
@@ -70,7 +78,8 @@ rte_eal_cpu_init(void)
 
 		/* in 1:1 mapping, record related cpu detected state */
 		lcore_config[lcore_id].detected = eal_cpu_detected(lcore_id);
-		if (lcore_config[lcore_id].detected == 0) {
+		if (lcore_config[lcore_id].detected == 0 ||
+		    !CPU_ISSET(lcore_id, &cs)) {
 			config->lcore_role[lcore_id] = ROLE_OFF;
 			lcore_config[lcore_id].core_index = -1;
 			continue;
-- 
2.7.4

Re: [PATCH v2] eal: restrict cores detection

[Stephen Hemminger]

On Wed, 31 Aug 2016 03:07:10 +0000
Jianfeng Tan <jianfeng.tan@intel.com> wrote:

> This patch uses pthread_getaffinity_np() to narrow down detected
> cores before parsing coremask (-c), corelist (-l), and coremap
> (--lcores).
> 
> The purpose of this patch is to leave out these core related options
> when DPDK applications are deployed under container env, so that
> users only specify core restriction as starting the instance.
> 
> Note: previously, some users are using isolated CPUs, which could
> be excluded by default. Please add commands like taskset to use
> those cores.
> 
> Test example:
> $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024
> 
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> Acked-by: Neil Horman <nhorman@tuxdriver.com>
> ---
> v2:
>   - Make it as default instead of adding the new options.
>  lib/librte_eal/common/eal_common_lcore.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
> index 2cd4132..62e4f67 100644
> --- a/lib/librte_eal/common/eal_common_lcore.c
> +++ b/lib/librte_eal/common/eal_common_lcore.c
> @@ -57,6 +57,14 @@ rte_eal_cpu_init(void)
>  	struct rte_config *config = rte_eal_get_configuration();
>  	unsigned lcore_id;
>  	unsigned count = 0;
> +	rte_cpuset_t cs;
> +	pthread_t tid = pthread_self();
> +
> +	/* Add below method to obtain core restrictions, like ulimit,
> +	 * cgroup.cpuset, etc. Will not use those cores, which are rebuffed.
> +	 */
> +	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cs) < 0)
> +		CPU_ZERO(&cs);
>  

This patch makes sense but the comment is hard to read because of wording
and grammar.

If you choose variable names better then there really is no need for
a comment in many cases. Code is often easier to read/write than comments
for non-native English speakers.

Remove the comment and rename 'cs' as 'affinity_set' or something equally
as descriptive.

Re: [PATCH v2] eal: restrict cores detection

[Tan, Jianfeng]

Hi Stephen,

> -----Original Message-----
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Wednesday, August 31, 2016 11:31 PM
> To: Tan, Jianfeng
> Cc: dev@dpdk.org; david.marchand@6wind.com; pmatilai@redhat.com;
> thomas.monjalon@6wind.com
> Subject: Re: [dpdk-dev] [PATCH v2] eal: restrict cores detection
> 
> On Wed, 31 Aug 2016 03:07:10 +0000
> Jianfeng Tan <jianfeng.tan@intel.com> wrote:
> 
> > This patch uses pthread_getaffinity_np() to narrow down detected
> > cores before parsing coremask (-c), corelist (-l), and coremap
> > (--lcores).
> >
> > The purpose of this patch is to leave out these core related options
> > when DPDK applications are deployed under container env, so that
> > users only specify core restriction as starting the instance.
> >
> > Note: previously, some users are using isolated CPUs, which could
> > be excluded by default. Please add commands like taskset to use
> > those cores.
> >
> > Test example:
> > $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024
> >
> > Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> > Acked-by: Neil Horman <nhorman@tuxdriver.com>
> > ---
> > v2:
> >   - Make it as default instead of adding the new options.
> >  lib/librte_eal/common/eal_common_lcore.c | 11 ++++++++++-
> >  1 file changed, 10 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/librte_eal/common/eal_common_lcore.c
> b/lib/librte_eal/common/eal_common_lcore.c
> > index 2cd4132..62e4f67 100644
> > --- a/lib/librte_eal/common/eal_common_lcore.c
> > +++ b/lib/librte_eal/common/eal_common_lcore.c
> > @@ -57,6 +57,14 @@ rte_eal_cpu_init(void)
> >  	struct rte_config *config = rte_eal_get_configuration();
> >  	unsigned lcore_id;
> >  	unsigned count = 0;
> > +	rte_cpuset_t cs;
> > +	pthread_t tid = pthread_self();
> > +
> > +	/* Add below method to obtain core restrictions, like ulimit,
> > +	 * cgroup.cpuset, etc. Will not use those cores, which are rebuffed.
> > +	 */
> > +	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t), &cs) < 0)
> > +		CPU_ZERO(&cs);
> >
> 
> This patch makes sense but the comment is hard to read because of wording
> and grammar.
> 
> If you choose variable names better then there really is no need for
> a comment in many cases. Code is often easier to read/write than comments
> for non-native English speakers.
> 
> Remove the comment and rename 'cs' as 'affinity_set' or something equally
> as descriptive.

Great suggestion. I'll resend one as you suggest.

Thanks,
Jianfeng

[PATCH v3] eal: restrict cores detection

[Jianfeng Tan]

This patch uses pthread_getaffinity_np() to narrow down detected
cores before parsing coremask (-c), corelist (-l), and coremap
(--lcores).

The purpose of this patch is to leave out these core related options
when DPDK applications are deployed under container env, so that
users only specify core restriction as starting the instance.

Note: previously, some users are using isolated CPUs, which could
be excluded by default. Please add commands like taskset to use
those cores.

Test example:
$ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v3:
  - Choose a more descriptive variable name, and remove comments
    as suggested by Stephen Hemminger.
v2:
  - Make it as default instead of adding the new options.
 lib/librte_eal/common/eal_common_lcore.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_lcore.c b/lib/librte_eal/common/eal_common_lcore.c
index 2cd4132..71c575c 100644
--- a/lib/librte_eal/common/eal_common_lcore.c
+++ b/lib/librte_eal/common/eal_common_lcore.c
@@ -57,6 +57,12 @@ rte_eal_cpu_init(void)
 	struct rte_config *config = rte_eal_get_configuration();
 	unsigned lcore_id;
 	unsigned count = 0;
+	rte_cpuset_t affinity_set;
+	pthread_t tid = pthread_self();
+
+	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t),
+				   &affinity_set) < 0)
+		CPU_ZERO(&affinity_set);
 
 	/*
 	 * Parse the maximum set of logical cores, detect the subset of running
@@ -70,7 +76,8 @@ rte_eal_cpu_init(void)
 
 		/* in 1:1 mapping, record related cpu detected state */
 		lcore_config[lcore_id].detected = eal_cpu_detected(lcore_id);
-		if (lcore_config[lcore_id].detected == 0) {
+		if (lcore_config[lcore_id].detected == 0 ||
+		    !CPU_ISSET(lcore_id, &affinity_set)) {
 			config->lcore_role[lcore_id] = ROLE_OFF;
 			lcore_config[lcore_id].core_index = -1;
 			continue;
-- 
2.7.4

Re: [PATCH v3] eal: restrict cores detection

[Bruce Richardson]

On Thu, Sep 01, 2016 at 01:31:47AM +0000, Jianfeng Tan wrote:
> This patch uses pthread_getaffinity_np() to narrow down detected
> cores before parsing coremask (-c), corelist (-l), and coremap
> (--lcores).
> 
> The purpose of this patch is to leave out these core related options
> when DPDK applications are deployed under container env, so that
> users only specify core restriction as starting the instance.
> 
> Note: previously, some users are using isolated CPUs, which could
> be excluded by default. Please add commands like taskset to use
> those cores.
> 
> Test example:
> $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024
> 

So, to be clear, does this patch mean that DPDK cannot use isolated cores
any more unless you explicitly run the app using taskset?
Is so, NAK, since isolating cores has been part of standard DPDK setup since
the first versions, and I don't believe that we should break that behaviour.

/Bruce

Re: [PATCH v3] eal: restrict cores detection

[Thomas Monjalon]

2016-09-01 01:31, Jianfeng Tan:
> This patch uses pthread_getaffinity_np() to narrow down detected
> cores before parsing coremask (-c), corelist (-l), and coremap
> (--lcores).
> 
> The purpose of this patch is to leave out these core related options
> when DPDK applications are deployed under container env, so that
> users only specify core restriction as starting the instance.
[...]
> --- a/lib/librte_eal/common/eal_common_lcore.c
> +++ b/lib/librte_eal/common/eal_common_lcore.c
> @@ -57,6 +57,12 @@ rte_eal_cpu_init(void)
>  	struct rte_config *config = rte_eal_get_configuration();
>  	unsigned lcore_id;
>  	unsigned count = 0;
> +	rte_cpuset_t affinity_set;
> +	pthread_t tid = pthread_self();
> +

A comment is needed here to explain which errors we are checking.

> +	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t),
> +				   &affinity_set) < 0)
> +		CPU_ZERO(&affinity_set);
>  
>  	/*
>  	 * Parse the maximum set of logical cores, detect the subset of running
> @@ -70,7 +76,8 @@ rte_eal_cpu_init(void)
>  
>  		/* in 1:1 mapping, record related cpu detected state */
>  		lcore_config[lcore_id].detected = eal_cpu_detected(lcore_id);
> -		if (lcore_config[lcore_id].detected == 0) {
> +		if (lcore_config[lcore_id].detected == 0 ||
> +		    !CPU_ISSET(lcore_id, &affinity_set)) {
>  			config->lcore_role[lcore_id] = ROLE_OFF;
>  			lcore_config[lcore_id].core_index = -1;
>  			continue;
> 

Re: [PATCH v3] eal: restrict cores detection

[Thomas Monjalon]

2016-09-02 17:53, Bruce Richardson:
> On Thu, Sep 01, 2016 at 01:31:47AM +0000, Jianfeng Tan wrote:

It would help the discussion to have a problem statement here.

> > This patch uses pthread_getaffinity_np() to narrow down detected
> > cores before parsing coremask (-c), corelist (-l), and coremap
> > (--lcores).
> > 
> > The purpose of this patch is to leave out these core related options
> > when DPDK applications are deployed under container env, so that
> > users only specify core restriction as starting the instance.
> > 
> > Note: previously, some users are using isolated CPUs, which could
> > be excluded by default. Please add commands like taskset to use
> > those cores.
> > 
> > Test example:
> > $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024
> > 
> 
> So, to be clear, does this patch mean that DPDK cannot use isolated cores
> any more unless you explicitly run the app using taskset?
> Is so, NAK, since isolating cores has been part of standard DPDK setup since
> the first versions, and I don't believe that we should break that behaviour.

So how could we help the container use-case?
Any suggestions?

[PATCH v4] eal: restrict cores auto detection

[Jianfeng Tan]

This patch uses pthread_getaffinity_np() to narrow down used
cores when none of below options is specified:
  * coremask (-c)
  * corelist (-l)
  * and coremap (--lcores)

The purpose of this patch is to leave out these core related options
when DPDK applications are deployed under container env, so that
users do not need decide the core related parameters when developing
applications. Instead, when applications are deployed in containers,
use cpu-set to constrain which cores can be used inside this container
instance. And DPDK application inside containers just rely on this
auto detect mechanism to start polling threads.

Note: previously, some users are using isolated CPUs, which could
be excluded by default. Please add commands like taskset to use
those cores.

Test example:
$ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
Acked-by: Neil Horman <nhorman@tuxdriver.com>
---
v4:
  - Address Bruce's comment: only enable this auto detection
    mechanism when none of core options is specified.
  - More detailed use case on how it helps in containers.
v3:
  - Choose a more descriptive variable name, and remove comments
    as suggested by Stephen Hemminger.
v2:
  - Make it as default instead of adding the new options.
---
 lib/librte_eal/common/eal_common_options.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 6ca8af1..d192de1 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -126,6 +126,7 @@ static const char dpdk_solib_path[] __attribute__((used)) =
 
 static int master_lcore_parsed;
 static int mem_parsed;
+static int core_specified;
 
 void
 eal_reset_internal_config(struct internal_config *internal_cfg)
@@ -797,6 +798,7 @@ eal_parse_common_option(int opt, const char *optarg,
 			RTE_LOG(ERR, EAL, "invalid coremask\n");
 			return -1;
 		}
+		core_specified = 1;
 		break;
 	/* corelist */
 	case 'l':
@@ -804,6 +806,7 @@ eal_parse_common_option(int opt, const char *optarg,
 			RTE_LOG(ERR, EAL, "invalid core list\n");
 			return -1;
 		}
+		core_specified = 1;
 		break;
 	/* size of memory */
 	case 'm':
@@ -912,6 +915,7 @@ eal_parse_common_option(int opt, const char *optarg,
 				OPT_LCORES "\n");
 			return -1;
 		}
+		core_specified = 1;
 		break;
 
 	/* don't know what to do, leave this to caller */
@@ -923,12 +927,38 @@ eal_parse_common_option(int opt, const char *optarg,
 	return 0;
 }
 
+static void
+eal_auto_detect_cores(struct rte_config *cfg)
+{
+	unsigned int lcore_id;
+	unsigned int removed = 0;
+	rte_cpuset_t affinity_set;
+	pthread_t tid = pthread_self();
+
+	if (pthread_getaffinity_np(tid, sizeof(rte_cpuset_t),
+				&affinity_set) < 0)
+		CPU_ZERO(&affinity_set);
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		if (cfg->lcore_role[lcore_id] == ROLE_RTE &&
+		    !CPU_ISSET(lcore_id, &affinity_set)) {
+			cfg->lcore_role[lcore_id] = ROLE_OFF;
+			removed++;
+		}
+	}
+
+	cfg->lcore_count -= removed;
+}
+
 int
 eal_adjust_config(struct internal_config *internal_cfg)
 {
 	int i;
 	struct rte_config *cfg = rte_eal_get_configuration();
 
+	if (!core_specified)
+		eal_auto_detect_cores(cfg);
+
 	if (internal_config.process_type == RTE_PROC_AUTO)
 		internal_config.process_type = eal_proc_type_detect();
 
-- 
2.7.4

Re: [PATCH v4] eal: restrict cores auto detection

[Thomas Monjalon]

2016-12-02 17:48, Jianfeng Tan:
> This patch uses pthread_getaffinity_np() to narrow down used
> cores when none of below options is specified:
>   * coremask (-c)
>   * corelist (-l)
>   * and coremap (--lcores)
> 
> The purpose of this patch is to leave out these core related options
> when DPDK applications are deployed under container env, so that
> users do not need decide the core related parameters when developing
> applications. Instead, when applications are deployed in containers,
> use cpu-set to constrain which cores can be used inside this container
> instance. And DPDK application inside containers just rely on this
> auto detect mechanism to start polling threads.
> 
> Note: previously, some users are using isolated CPUs, which could
> be excluded by default. Please add commands like taskset to use
> those cores.
> 
> Test example:
> $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024

Bruce, what do you think of this version?
It requires taskset only if -c, -l and --lcores are not used.

>  static int master_lcore_parsed;
>  static int mem_parsed;
> +static int core_specified;

I think it's better to keep the word "parsed" as others.

Re: [PATCH v4] eal: restrict cores auto detection

[Bruce Richardson]

On Thu, Dec 08, 2016 at 07:19:41PM +0100, Thomas Monjalon wrote:
> 2016-12-02 17:48, Jianfeng Tan:
> > This patch uses pthread_getaffinity_np() to narrow down used
> > cores when none of below options is specified:
> >   * coremask (-c)
> >   * corelist (-l)
> >   * and coremap (--lcores)
> > 
> > The purpose of this patch is to leave out these core related options
> > when DPDK applications are deployed under container env, so that
> > users do not need decide the core related parameters when developing
> > applications. Instead, when applications are deployed in containers,
> > use cpu-set to constrain which cores can be used inside this container
> > instance. And DPDK application inside containers just rely on this
> > auto detect mechanism to start polling threads.
> > 
> > Note: previously, some users are using isolated CPUs, which could
> > be excluded by default. Please add commands like taskset to use
> > those cores.
> > 
> > Test example:
> > $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024
> 
> Bruce, what do you think of this version?
> It requires taskset only if -c, -l and --lcores are not used.
> 
I'm fine with that since it maintains backward compatibilty for those
options.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [PATCH v4] eal: restrict cores auto detection

[Thomas Monjalon]

2016-12-09 15:14, Bruce Richardson:
> On Thu, Dec 08, 2016 at 07:19:41PM +0100, Thomas Monjalon wrote:
> > 2016-12-02 17:48, Jianfeng Tan:
> > > This patch uses pthread_getaffinity_np() to narrow down used
> > > cores when none of below options is specified:
> > >   * coremask (-c)
> > >   * corelist (-l)
> > >   * and coremap (--lcores)
> > > 
> > > The purpose of this patch is to leave out these core related options
> > > when DPDK applications are deployed under container env, so that
> > > users do not need decide the core related parameters when developing
> > > applications. Instead, when applications are deployed in containers,
> > > use cpu-set to constrain which cores can be used inside this container
> > > instance. And DPDK application inside containers just rely on this
> > > auto detect mechanism to start polling threads.
> > > 
> > > Note: previously, some users are using isolated CPUs, which could
> > > be excluded by default. Please add commands like taskset to use
> > > those cores.
> > > 
> > > Test example:
> > > $ taskset 0xc0000 ./examples/helloworld/build/helloworld -m 1024
> > 
> > Bruce, what do you think of this version?
> > It requires taskset only if -c, -l and --lcores are not used.
> > 
> I'm fine with that since it maintains backward compatibilty for those
> options.
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Applied with "s/specified/parsed/", thanks