* [dpdk-dev] [PATCH] mk: add support for UBSAN
@ 2019-08-19 13:48 Harman Kalra
2019-09-13 11:40 ` Harman Kalra
2019-11-11 7:07 ` Thomas Monjalon
0 siblings, 2 replies; 10+ messages in thread
From: Harman Kalra @ 2019-08-19 13:48 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson
Cc: dev, Harman Kalra
UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
detector. UBSan modifies the program at compile-time to catch
various kinds of undefined behavior during program execution.
This patch implements support for UBSan to the DPDK.
See: doc/guides/prog_guide/ubsan.rst for more information.
Signed-off-by: Harman Kalra <hkalra@marvell.com>
---
config/common_base | 6 ++
config/meson.build | 15 ++++
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
doc/guides/rel_notes/release_19_11.rst | 7 ++
meson_options.txt | 2 +
mk/rte.app.mk | 10 +++
mk/rte.lib.mk | 12 +++
mk/toolchain/clang/rte.vars.mk | 4 +
mk/toolchain/gcc/rte.vars.mk | 8 ++
10 files changed, 177 insertions(+)
create mode 100644 doc/guides/prog_guide/ubsan.rst
diff --git a/config/common_base b/config/common_base
index 8ef75c203..b47a6f28e 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1067,3 +1067,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
# Compile the eventdev application
#
CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Enable undefined behavior sanitizer
+#
+CONFIG_RTE_UBSAN=n
+CONFIG_RTE_UBSAN_SANITIZE_ALL=n
diff --git a/config/meson.build b/config/meson.build
index 2bafea530..6a2fa117d 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -196,3 +196,18 @@ add_project_arguments('-D_GNU_SOURCE', language: 'c')
if is_freebsd
add_project_arguments('-D__BSD_VISIBLE', language: 'c')
endif
+
+# enable ubsan
+if get_option('enable_ubsan')
+ if cc.has_argument('-fsanitize=undefined')
+ ubsan_dep = cc.find_library('libubsan', required: false)
+ if ubsan_dep.found()
+ add_project_arguments('-fsanitize=undefined', language: 'c')
+ add_project_link_arguments('-fsanitize=undefined', language: 'c')
+ else
+ message('libubsan not found, UBSAN cannot be enabled')
+ endif
+ else
+ message('gcc version does not support UBSAN')
+ endif
+endif
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 692409af8..970901e9a 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -66,4 +66,5 @@ Programmer's Guide
perf_opt_guidelines
writing_efficient_code
profile_app
+ ubsan
glossary
diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
new file mode 100644
index 000000000..cb19f3bd9
--- /dev/null
+++ b/doc/guides/prog_guide/ubsan.rst
@@ -0,0 +1,112 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2019 Marvell International Ltd.
+
+The Undefined Behavior Sanitizer - UBSan
+========================================
+
+UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
+UBSan uses compile-time instrumentation and modifies the program by adding
+some stubs which perform certain checks before operations that might cause
+undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
+(which are defined in libUBSan library) are called to prints the error message.
+
+Some examples of undefined behaviour checks:
+
+* Misaligned memory access
+* Signed integer overflow
+* Load from/store to an object with insufficient space.
+* Integer divide by zero as well as INT_MIN / -1 division
+* Out-of-bounds memory accesses.
+* Null argument declared with nonnull attribute, returned null from function
+ which never returns null, null ptr dereference
+* Variable size array with non-positive length
+
+GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
+checkers implemented.
+
+Example UBSan error
+--------------------
+
+Following error was reported when UBSan was enabled:
+
+.. code-block:: console
+
+ drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
+ 1 by 31 places cannot be represented in type 'int'
+
+Code responsible for this error:
+
+.. code-block:: c
+
+ if (dev->txmap[i] & (1 << 31)) {
+
+To fix this error:
+
+.. code-block:: c
+
+ if (dev->txmap[i] & (1U << 31)) {
+
+Usage
+-----
+
+make build
+^^^^^^^^^^
+
+To enable UBSan, enable following configuration:
+
+.. code-block:: console
+
+ CONFIG_RTE_UBSAN=y
+
+UBSan framework supports three modes:
+
+1. Enable UBSan on the entire DPDK source code - set following configuration:
+
+.. code-block:: console
+
+ CONFIG_RTE_UBSAN_SANITIZE_ALL=y
+
+2. Enable UBSan on a particular library or PMD - add the following line to the
+ respective Makefile of the library or PMD
+ (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
+ the library or PMD and not the entire repository.
+
+.. code-block:: console
+
+ UBSAN_SANITIZE := y
+
+3. Disable UBSan for a particular library or PMD - add the following line to
+ the respective Makefile of the library or PMD. Make sure
+ ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
+ entire DPDK repository but not this specific library or PMD.
+
+.. code-block:: console
+
+ UBSAN_SANITIZE := n
+
+.. Note::
+
+ Standard DPDK applications like test, testpmd, etc. cannot be
+ chosen explicitly for UBSan check, like libraries or PMD. The reason is,
+ say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
+ in Makefile of app Y which uses X APIs. This will lead to undefined
+ reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
+ Hence UBSan check is enabled for all standard DPDK applications as soon as
+ ``CONFIG_RTE_UBSAN=y`` is set.
+
+meson build
+^^^^^^^^^^^
+
+To enable UBSan in meson build system, use following meson build command:
+
+**Example usage:**
+
+.. code-block:: console
+
+ meson build -Denable_ubsan=true
+ ninja -C build
+
+.. Note::
+
+ Meson build works only in one mode i.e. UBSan can be enabled for
+ the entire DPDK sources and not individual libraries or PMD, like make build.
diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index 8490d897c..cfefdbeec 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -56,6 +56,13 @@ New Features
Also, make sure to start the actual text at the margin.
=========================================================
+* **Added Undefined Behavior Sanitizer framework.**
+
+ UBSan is a fast runtime undefined behavior detector which uses compile-time
+ instrumentation and modifies the program by adding some stubs that perform
+ certain checks before operations that might cause undefined behavior.
+
+ See :doc:`../prog_guide/ubsan` for more information:
Removed Items
-------------
diff --git a/meson_options.txt b/meson_options.txt
index 448f3e63d..fb0aead00 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -8,6 +8,8 @@ option('enable_docs', type: 'boolean', value: false,
description: 'build documentation')
option('enable_kmods', type: 'boolean', value: true,
description: 'build kernel modules')
+option('enable_ubsan', type: 'boolean', value: false,
+ description: 'Enables undefined behavior sanitizer')
option('examples', type: 'string', value: '',
description: 'Comma-separated list of examples to build by default')
option('flexran_sdk', type: 'string', value: '',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index ba5c39e01..9ea421a83 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -374,6 +374,16 @@ endif
MAPFLAGS = -Map=$@.map --cref
+#
+# If UBSAN is enabled, all application will be compiled with
+# '-fsanitize=undefined' flag
+#
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(UBSAN_ENABLE),y)
+CFLAGS += -fsanitize=undefined
+endif
+endif
+
.PHONY: all
all: install
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 4df8849a0..33f5746c8 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
endif
endif
+#
+# If UBSAN is enabled, lib to undergo check can be chosen
+# by setting UBSAN_SANITIZE=y in respective lib Makefile
+# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
+# on entire repo.
+#
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(UBSAN_ENABLE),y)
+CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
+ , -fsanitize=undefined)
+endif
+endif
_BUILD = $(LIB)
PREINSTALL = $(SYMLINK-FILES-y)
diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
index 3c49dc568..623780106 100644
--- a/mk/toolchain/clang/rte.vars.mk
+++ b/mk/toolchain/clang/rte.vars.mk
@@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
WERROR_FLAGS += -Wno-address-of-packed-member
endif
+ifeq ($(CONFIG_RTE_UBSAN),y)
+UBSAN_ENABLE := y
+endif
+
export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
index b852fcfd7..d40e14d61 100644
--- a/mk/toolchain/gcc/rte.vars.mk
+++ b/mk/toolchain/gcc/rte.vars.mk
@@ -90,5 +90,13 @@ endif
# disable packed member unalign warnings
WERROR_FLAGS += -Wno-address-of-packed-member
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
+$(warning UBSAN not supported gcc < 4.9)
+else
+UBSAN_ENABLE = y
+endif
+endif
+
export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
--
2.18.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] mk: add support for UBSAN
2019-08-19 13:48 [dpdk-dev] [PATCH] mk: add support for UBSAN Harman Kalra
@ 2019-09-13 11:40 ` Harman Kalra
2019-10-09 14:25 ` Harman Kalra
2019-11-11 7:07 ` Thomas Monjalon
1 sibling, 1 reply; 10+ messages in thread
From: Harman Kalra @ 2019-09-13 11:40 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson; +Cc: dev
Ping..
Kindly review this patch.
On Mon, Aug 19, 2019 at 07:18:21PM +0530, Harman Kalra wrote:
> UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
> detector. UBSan modifies the program at compile-time to catch
> various kinds of undefined behavior during program execution.
>
> This patch implements support for UBSan to the DPDK.
>
> See: doc/guides/prog_guide/ubsan.rst for more information.
>
> Signed-off-by: Harman Kalra <hkalra@marvell.com>
> ---
> config/common_base | 6 ++
> config/meson.build | 15 ++++
> doc/guides/prog_guide/index.rst | 1 +
> doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
> doc/guides/rel_notes/release_19_11.rst | 7 ++
> meson_options.txt | 2 +
> mk/rte.app.mk | 10 +++
> mk/rte.lib.mk | 12 +++
> mk/toolchain/clang/rte.vars.mk | 4 +
> mk/toolchain/gcc/rte.vars.mk | 8 ++
> 10 files changed, 177 insertions(+)
> create mode 100644 doc/guides/prog_guide/ubsan.rst
>
> diff --git a/config/common_base b/config/common_base
> index 8ef75c203..b47a6f28e 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -1067,3 +1067,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
> # Compile the eventdev application
> #
> CONFIG_RTE_APP_EVENTDEV=y
> +
> +#
> +# Enable undefined behavior sanitizer
> +#
> +CONFIG_RTE_UBSAN=n
> +CONFIG_RTE_UBSAN_SANITIZE_ALL=n
> diff --git a/config/meson.build b/config/meson.build
> index 2bafea530..6a2fa117d 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -196,3 +196,18 @@ add_project_arguments('-D_GNU_SOURCE', language: 'c')
> if is_freebsd
> add_project_arguments('-D__BSD_VISIBLE', language: 'c')
> endif
> +
> +# enable ubsan
> +if get_option('enable_ubsan')
> + if cc.has_argument('-fsanitize=undefined')
> + ubsan_dep = cc.find_library('libubsan', required: false)
> + if ubsan_dep.found()
> + add_project_arguments('-fsanitize=undefined', language: 'c')
> + add_project_link_arguments('-fsanitize=undefined', language: 'c')
> + else
> + message('libubsan not found, UBSAN cannot be enabled')
> + endif
> + else
> + message('gcc version does not support UBSAN')
> + endif
> +endif
> diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
> index 692409af8..970901e9a 100644
> --- a/doc/guides/prog_guide/index.rst
> +++ b/doc/guides/prog_guide/index.rst
> @@ -66,4 +66,5 @@ Programmer's Guide
> perf_opt_guidelines
> writing_efficient_code
> profile_app
> + ubsan
> glossary
> diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
> new file mode 100644
> index 000000000..cb19f3bd9
> --- /dev/null
> +++ b/doc/guides/prog_guide/ubsan.rst
> @@ -0,0 +1,112 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2019 Marvell International Ltd.
> +
> +The Undefined Behavior Sanitizer - UBSan
> +========================================
> +
> +UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
> +UBSan uses compile-time instrumentation and modifies the program by adding
> +some stubs which perform certain checks before operations that might cause
> +undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
> +(which are defined in libUBSan library) are called to prints the error message.
> +
> +Some examples of undefined behaviour checks:
> +
> +* Misaligned memory access
> +* Signed integer overflow
> +* Load from/store to an object with insufficient space.
> +* Integer divide by zero as well as INT_MIN / -1 division
> +* Out-of-bounds memory accesses.
> +* Null argument declared with nonnull attribute, returned null from function
> + which never returns null, null ptr dereference
> +* Variable size array with non-positive length
> +
> +GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
> +checkers implemented.
> +
> +Example UBSan error
> +--------------------
> +
> +Following error was reported when UBSan was enabled:
> +
> +.. code-block:: console
> +
> + drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
> + 1 by 31 places cannot be represented in type 'int'
> +
> +Code responsible for this error:
> +
> +.. code-block:: c
> +
> + if (dev->txmap[i] & (1 << 31)) {
> +
> +To fix this error:
> +
> +.. code-block:: c
> +
> + if (dev->txmap[i] & (1U << 31)) {
> +
> +Usage
> +-----
> +
> +make build
> +^^^^^^^^^^
> +
> +To enable UBSan, enable following configuration:
> +
> +.. code-block:: console
> +
> + CONFIG_RTE_UBSAN=y
> +
> +UBSan framework supports three modes:
> +
> +1. Enable UBSan on the entire DPDK source code - set following configuration:
> +
> +.. code-block:: console
> +
> + CONFIG_RTE_UBSAN_SANITIZE_ALL=y
> +
> +2. Enable UBSan on a particular library or PMD - add the following line to the
> + respective Makefile of the library or PMD
> + (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
> + the library or PMD and not the entire repository.
> +
> +.. code-block:: console
> +
> + UBSAN_SANITIZE := y
> +
> +3. Disable UBSan for a particular library or PMD - add the following line to
> + the respective Makefile of the library or PMD. Make sure
> + ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
> + entire DPDK repository but not this specific library or PMD.
> +
> +.. code-block:: console
> +
> + UBSAN_SANITIZE := n
> +
> +.. Note::
> +
> + Standard DPDK applications like test, testpmd, etc. cannot be
> + chosen explicitly for UBSan check, like libraries or PMD. The reason is,
> + say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
> + in Makefile of app Y which uses X APIs. This will lead to undefined
> + reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
> + Hence UBSan check is enabled for all standard DPDK applications as soon as
> + ``CONFIG_RTE_UBSAN=y`` is set.
> +
> +meson build
> +^^^^^^^^^^^
> +
> +To enable UBSan in meson build system, use following meson build command:
> +
> +**Example usage:**
> +
> +.. code-block:: console
> +
> + meson build -Denable_ubsan=true
> + ninja -C build
> +
> +.. Note::
> +
> + Meson build works only in one mode i.e. UBSan can be enabled for
> + the entire DPDK sources and not individual libraries or PMD, like make build.
> diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> index 8490d897c..cfefdbeec 100644
> --- a/doc/guides/rel_notes/release_19_11.rst
> +++ b/doc/guides/rel_notes/release_19_11.rst
> @@ -56,6 +56,13 @@ New Features
> Also, make sure to start the actual text at the margin.
> =========================================================
>
> +* **Added Undefined Behavior Sanitizer framework.**
> +
> + UBSan is a fast runtime undefined behavior detector which uses compile-time
> + instrumentation and modifies the program by adding some stubs that perform
> + certain checks before operations that might cause undefined behavior.
> +
> + See :doc:`../prog_guide/ubsan` for more information:
>
> Removed Items
> -------------
> diff --git a/meson_options.txt b/meson_options.txt
> index 448f3e63d..fb0aead00 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -8,6 +8,8 @@ option('enable_docs', type: 'boolean', value: false,
> description: 'build documentation')
> option('enable_kmods', type: 'boolean', value: true,
> description: 'build kernel modules')
> +option('enable_ubsan', type: 'boolean', value: false,
> + description: 'Enables undefined behavior sanitizer')
> option('examples', type: 'string', value: '',
> description: 'Comma-separated list of examples to build by default')
> option('flexran_sdk', type: 'string', value: '',
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index ba5c39e01..9ea421a83 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -374,6 +374,16 @@ endif
>
> MAPFLAGS = -Map=$@.map --cref
>
> +#
> +# If UBSAN is enabled, all application will be compiled with
> +# '-fsanitize=undefined' flag
> +#
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +ifeq ($(UBSAN_ENABLE),y)
> +CFLAGS += -fsanitize=undefined
> +endif
> +endif
> +
> .PHONY: all
> all: install
>
> diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> index 4df8849a0..33f5746c8 100644
> --- a/mk/rte.lib.mk
> +++ b/mk/rte.lib.mk
> @@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
> endif
> endif
>
> +#
> +# If UBSAN is enabled, lib to undergo check can be chosen
> +# by setting UBSAN_SANITIZE=y in respective lib Makefile
> +# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
> +# on entire repo.
> +#
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +ifeq ($(UBSAN_ENABLE),y)
> +CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
> + , -fsanitize=undefined)
> +endif
> +endif
>
> _BUILD = $(LIB)
> PREINSTALL = $(SYMLINK-FILES-y)
> diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
> index 3c49dc568..623780106 100644
> --- a/mk/toolchain/clang/rte.vars.mk
> +++ b/mk/toolchain/clang/rte.vars.mk
> @@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
> WERROR_FLAGS += -Wno-address-of-packed-member
> endif
>
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +UBSAN_ENABLE := y
> +endif
> +
> export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
> index b852fcfd7..d40e14d61 100644
> --- a/mk/toolchain/gcc/rte.vars.mk
> +++ b/mk/toolchain/gcc/rte.vars.mk
> @@ -90,5 +90,13 @@ endif
> # disable packed member unalign warnings
> WERROR_FLAGS += -Wno-address-of-packed-member
>
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
> +$(warning UBSAN not supported gcc < 4.9)
> +else
> +UBSAN_ENABLE = y
> +endif
> +endif
> +
> export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> --
> 2.18.0
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] mk: add support for UBSAN
2019-09-13 11:40 ` Harman Kalra
@ 2019-10-09 14:25 ` Harman Kalra
2019-10-28 10:50 ` Harman Kalra
0 siblings, 1 reply; 10+ messages in thread
From: Harman Kalra @ 2019-10-09 14:25 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson; +Cc: dev
Ping..
On Fri, Sep 13, 2019 at 11:40:40AM +0000, Harman Kalra wrote:
> Ping..
> Kindly review this patch.
>
> On Mon, Aug 19, 2019 at 07:18:21PM +0530, Harman Kalra wrote:
> > UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
> > detector. UBSan modifies the program at compile-time to catch
> > various kinds of undefined behavior during program execution.
> >
> > This patch implements support for UBSan to the DPDK.
> >
> > See: doc/guides/prog_guide/ubsan.rst for more information.
> >
> > Signed-off-by: Harman Kalra <hkalra@marvell.com>
> > ---
> > config/common_base | 6 ++
> > config/meson.build | 15 ++++
> > doc/guides/prog_guide/index.rst | 1 +
> > doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
> > doc/guides/rel_notes/release_19_11.rst | 7 ++
> > meson_options.txt | 2 +
> > mk/rte.app.mk | 10 +++
> > mk/rte.lib.mk | 12 +++
> > mk/toolchain/clang/rte.vars.mk | 4 +
> > mk/toolchain/gcc/rte.vars.mk | 8 ++
> > 10 files changed, 177 insertions(+)
> > create mode 100644 doc/guides/prog_guide/ubsan.rst
> >
> > diff --git a/config/common_base b/config/common_base
> > index 8ef75c203..b47a6f28e 100644
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -1067,3 +1067,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
> > # Compile the eventdev application
> > #
> > CONFIG_RTE_APP_EVENTDEV=y
> > +
> > +#
> > +# Enable undefined behavior sanitizer
> > +#
> > +CONFIG_RTE_UBSAN=n
> > +CONFIG_RTE_UBSAN_SANITIZE_ALL=n
> > diff --git a/config/meson.build b/config/meson.build
> > index 2bafea530..6a2fa117d 100644
> > --- a/config/meson.build
> > +++ b/config/meson.build
> > @@ -196,3 +196,18 @@ add_project_arguments('-D_GNU_SOURCE', language: 'c')
> > if is_freebsd
> > add_project_arguments('-D__BSD_VISIBLE', language: 'c')
> > endif
> > +
> > +# enable ubsan
> > +if get_option('enable_ubsan')
> > + if cc.has_argument('-fsanitize=undefined')
> > + ubsan_dep = cc.find_library('libubsan', required: false)
> > + if ubsan_dep.found()
> > + add_project_arguments('-fsanitize=undefined', language: 'c')
> > + add_project_link_arguments('-fsanitize=undefined', language: 'c')
> > + else
> > + message('libubsan not found, UBSAN cannot be enabled')
> > + endif
> > + else
> > + message('gcc version does not support UBSAN')
> > + endif
> > +endif
> > diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
> > index 692409af8..970901e9a 100644
> > --- a/doc/guides/prog_guide/index.rst
> > +++ b/doc/guides/prog_guide/index.rst
> > @@ -66,4 +66,5 @@ Programmer's Guide
> > perf_opt_guidelines
> > writing_efficient_code
> > profile_app
> > + ubsan
> > glossary
> > diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
> > new file mode 100644
> > index 000000000..cb19f3bd9
> > --- /dev/null
> > +++ b/doc/guides/prog_guide/ubsan.rst
> > @@ -0,0 +1,112 @@
> > +.. SPDX-License-Identifier: BSD-3-Clause
> > + Copyright(c) 2019 Marvell International Ltd.
> > +
> > +The Undefined Behavior Sanitizer - UBSan
> > +========================================
> > +
> > +UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
> > +UBSan uses compile-time instrumentation and modifies the program by adding
> > +some stubs which perform certain checks before operations that might cause
> > +undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
> > +(which are defined in libUBSan library) are called to prints the error message.
> > +
> > +Some examples of undefined behaviour checks:
> > +
> > +* Misaligned memory access
> > +* Signed integer overflow
> > +* Load from/store to an object with insufficient space.
> > +* Integer divide by zero as well as INT_MIN / -1 division
> > +* Out-of-bounds memory accesses.
> > +* Null argument declared with nonnull attribute, returned null from function
> > + which never returns null, null ptr dereference
> > +* Variable size array with non-positive length
> > +
> > +GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
> > +checkers implemented.
> > +
> > +Example UBSan error
> > +--------------------
> > +
> > +Following error was reported when UBSan was enabled:
> > +
> > +.. code-block:: console
> > +
> > + drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
> > + 1 by 31 places cannot be represented in type 'int'
> > +
> > +Code responsible for this error:
> > +
> > +.. code-block:: c
> > +
> > + if (dev->txmap[i] & (1 << 31)) {
> > +
> > +To fix this error:
> > +
> > +.. code-block:: c
> > +
> > + if (dev->txmap[i] & (1U << 31)) {
> > +
> > +Usage
> > +-----
> > +
> > +make build
> > +^^^^^^^^^^
> > +
> > +To enable UBSan, enable following configuration:
> > +
> > +.. code-block:: console
> > +
> > + CONFIG_RTE_UBSAN=y
> > +
> > +UBSan framework supports three modes:
> > +
> > +1. Enable UBSan on the entire DPDK source code - set following configuration:
> > +
> > +.. code-block:: console
> > +
> > + CONFIG_RTE_UBSAN_SANITIZE_ALL=y
> > +
> > +2. Enable UBSan on a particular library or PMD - add the following line to the
> > + respective Makefile of the library or PMD
> > + (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
> > + the library or PMD and not the entire repository.
> > +
> > +.. code-block:: console
> > +
> > + UBSAN_SANITIZE := y
> > +
> > +3. Disable UBSan for a particular library or PMD - add the following line to
> > + the respective Makefile of the library or PMD. Make sure
> > + ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
> > + entire DPDK repository but not this specific library or PMD.
> > +
> > +.. code-block:: console
> > +
> > + UBSAN_SANITIZE := n
> > +
> > +.. Note::
> > +
> > + Standard DPDK applications like test, testpmd, etc. cannot be
> > + chosen explicitly for UBSan check, like libraries or PMD. The reason is,
> > + say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
> > + in Makefile of app Y which uses X APIs. This will lead to undefined
> > + reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
> > + Hence UBSan check is enabled for all standard DPDK applications as soon as
> > + ``CONFIG_RTE_UBSAN=y`` is set.
> > +
> > +meson build
> > +^^^^^^^^^^^
> > +
> > +To enable UBSan in meson build system, use following meson build command:
> > +
> > +**Example usage:**
> > +
> > +.. code-block:: console
> > +
> > + meson build -Denable_ubsan=true
> > + ninja -C build
> > +
> > +.. Note::
> > +
> > + Meson build works only in one mode i.e. UBSan can be enabled for
> > + the entire DPDK sources and not individual libraries or PMD, like make build.
> > diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> > index 8490d897c..cfefdbeec 100644
> > --- a/doc/guides/rel_notes/release_19_11.rst
> > +++ b/doc/guides/rel_notes/release_19_11.rst
> > @@ -56,6 +56,13 @@ New Features
> > Also, make sure to start the actual text at the margin.
> > =========================================================
> >
> > +* **Added Undefined Behavior Sanitizer framework.**
> > +
> > + UBSan is a fast runtime undefined behavior detector which uses compile-time
> > + instrumentation and modifies the program by adding some stubs that perform
> > + certain checks before operations that might cause undefined behavior.
> > +
> > + See :doc:`../prog_guide/ubsan` for more information:
> >
> > Removed Items
> > -------------
> > diff --git a/meson_options.txt b/meson_options.txt
> > index 448f3e63d..fb0aead00 100644
> > --- a/meson_options.txt
> > +++ b/meson_options.txt
> > @@ -8,6 +8,8 @@ option('enable_docs', type: 'boolean', value: false,
> > description: 'build documentation')
> > option('enable_kmods', type: 'boolean', value: true,
> > description: 'build kernel modules')
> > +option('enable_ubsan', type: 'boolean', value: false,
> > + description: 'Enables undefined behavior sanitizer')
> > option('examples', type: 'string', value: '',
> > description: 'Comma-separated list of examples to build by default')
> > option('flexran_sdk', type: 'string', value: '',
> > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > index ba5c39e01..9ea421a83 100644
> > --- a/mk/rte.app.mk
> > +++ b/mk/rte.app.mk
> > @@ -374,6 +374,16 @@ endif
> >
> > MAPFLAGS = -Map=$@.map --cref
> >
> > +#
> > +# If UBSAN is enabled, all application will be compiled with
> > +# '-fsanitize=undefined' flag
> > +#
> > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > +ifeq ($(UBSAN_ENABLE),y)
> > +CFLAGS += -fsanitize=undefined
> > +endif
> > +endif
> > +
> > .PHONY: all
> > all: install
> >
> > diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> > index 4df8849a0..33f5746c8 100644
> > --- a/mk/rte.lib.mk
> > +++ b/mk/rte.lib.mk
> > @@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
> > endif
> > endif
> >
> > +#
> > +# If UBSAN is enabled, lib to undergo check can be chosen
> > +# by setting UBSAN_SANITIZE=y in respective lib Makefile
> > +# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
> > +# on entire repo.
> > +#
> > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > +ifeq ($(UBSAN_ENABLE),y)
> > +CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
> > + , -fsanitize=undefined)
> > +endif
> > +endif
> >
> > _BUILD = $(LIB)
> > PREINSTALL = $(SYMLINK-FILES-y)
> > diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
> > index 3c49dc568..623780106 100644
> > --- a/mk/toolchain/clang/rte.vars.mk
> > +++ b/mk/toolchain/clang/rte.vars.mk
> > @@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
> > WERROR_FLAGS += -Wno-address-of-packed-member
> > endif
> >
> > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > +UBSAN_ENABLE := y
> > +endif
> > +
> > export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> > export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> > diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
> > index b852fcfd7..d40e14d61 100644
> > --- a/mk/toolchain/gcc/rte.vars.mk
> > +++ b/mk/toolchain/gcc/rte.vars.mk
> > @@ -90,5 +90,13 @@ endif
> > # disable packed member unalign warnings
> > WERROR_FLAGS += -Wno-address-of-packed-member
> >
> > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > +ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
> > +$(warning UBSAN not supported gcc < 4.9)
> > +else
> > +UBSAN_ENABLE = y
> > +endif
> > +endif
> > +
> > export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> > export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> > --
> > 2.18.0
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] mk: add support for UBSAN
2019-10-09 14:25 ` Harman Kalra
@ 2019-10-28 10:50 ` Harman Kalra
0 siblings, 0 replies; 10+ messages in thread
From: Harman Kalra @ 2019-10-28 10:50 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson; +Cc: dev
Ping...
Kindly review this patch.
On Wed, Oct 09, 2019 at 02:25:52PM +0000, Harman Kalra wrote:
> Ping..
>
> On Fri, Sep 13, 2019 at 11:40:40AM +0000, Harman Kalra wrote:
> > Ping..
> > Kindly review this patch.
> >
> > On Mon, Aug 19, 2019 at 07:18:21PM +0530, Harman Kalra wrote:
> > > UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
> > > detector. UBSan modifies the program at compile-time to catch
> > > various kinds of undefined behavior during program execution.
> > >
> > > This patch implements support for UBSan to the DPDK.
> > >
> > > See: doc/guides/prog_guide/ubsan.rst for more information.
> > >
> > > Signed-off-by: Harman Kalra <hkalra@marvell.com>
> > > ---
> > > config/common_base | 6 ++
> > > config/meson.build | 15 ++++
> > > doc/guides/prog_guide/index.rst | 1 +
> > > doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
> > > doc/guides/rel_notes/release_19_11.rst | 7 ++
> > > meson_options.txt | 2 +
> > > mk/rte.app.mk | 10 +++
> > > mk/rte.lib.mk | 12 +++
> > > mk/toolchain/clang/rte.vars.mk | 4 +
> > > mk/toolchain/gcc/rte.vars.mk | 8 ++
> > > 10 files changed, 177 insertions(+)
> > > create mode 100644 doc/guides/prog_guide/ubsan.rst
> > >
> > > diff --git a/config/common_base b/config/common_base
> > > index 8ef75c203..b47a6f28e 100644
> > > --- a/config/common_base
> > > +++ b/config/common_base
> > > @@ -1067,3 +1067,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
> > > # Compile the eventdev application
> > > #
> > > CONFIG_RTE_APP_EVENTDEV=y
> > > +
> > > +#
> > > +# Enable undefined behavior sanitizer
> > > +#
> > > +CONFIG_RTE_UBSAN=n
> > > +CONFIG_RTE_UBSAN_SANITIZE_ALL=n
> > > diff --git a/config/meson.build b/config/meson.build
> > > index 2bafea530..6a2fa117d 100644
> > > --- a/config/meson.build
> > > +++ b/config/meson.build
> > > @@ -196,3 +196,18 @@ add_project_arguments('-D_GNU_SOURCE', language: 'c')
> > > if is_freebsd
> > > add_project_arguments('-D__BSD_VISIBLE', language: 'c')
> > > endif
> > > +
> > > +# enable ubsan
> > > +if get_option('enable_ubsan')
> > > + if cc.has_argument('-fsanitize=undefined')
> > > + ubsan_dep = cc.find_library('libubsan', required: false)
> > > + if ubsan_dep.found()
> > > + add_project_arguments('-fsanitize=undefined', language: 'c')
> > > + add_project_link_arguments('-fsanitize=undefined', language: 'c')
> > > + else
> > > + message('libubsan not found, UBSAN cannot be enabled')
> > > + endif
> > > + else
> > > + message('gcc version does not support UBSAN')
> > > + endif
> > > +endif
> > > diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
> > > index 692409af8..970901e9a 100644
> > > --- a/doc/guides/prog_guide/index.rst
> > > +++ b/doc/guides/prog_guide/index.rst
> > > @@ -66,4 +66,5 @@ Programmer's Guide
> > > perf_opt_guidelines
> > > writing_efficient_code
> > > profile_app
> > > + ubsan
> > > glossary
> > > diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
> > > new file mode 100644
> > > index 000000000..cb19f3bd9
> > > --- /dev/null
> > > +++ b/doc/guides/prog_guide/ubsan.rst
> > > @@ -0,0 +1,112 @@
> > > +.. SPDX-License-Identifier: BSD-3-Clause
> > > + Copyright(c) 2019 Marvell International Ltd.
> > > +
> > > +The Undefined Behavior Sanitizer - UBSan
> > > +========================================
> > > +
> > > +UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
> > > +UBSan uses compile-time instrumentation and modifies the program by adding
> > > +some stubs which perform certain checks before operations that might cause
> > > +undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
> > > +(which are defined in libUBSan library) are called to prints the error message.
> > > +
> > > +Some examples of undefined behaviour checks:
> > > +
> > > +* Misaligned memory access
> > > +* Signed integer overflow
> > > +* Load from/store to an object with insufficient space.
> > > +* Integer divide by zero as well as INT_MIN / -1 division
> > > +* Out-of-bounds memory accesses.
> > > +* Null argument declared with nonnull attribute, returned null from function
> > > + which never returns null, null ptr dereference
> > > +* Variable size array with non-positive length
> > > +
> > > +GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
> > > +checkers implemented.
> > > +
> > > +Example UBSan error
> > > +--------------------
> > > +
> > > +Following error was reported when UBSan was enabled:
> > > +
> > > +.. code-block:: console
> > > +
> > > + drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
> > > + 1 by 31 places cannot be represented in type 'int'
> > > +
> > > +Code responsible for this error:
> > > +
> > > +.. code-block:: c
> > > +
> > > + if (dev->txmap[i] & (1 << 31)) {
> > > +
> > > +To fix this error:
> > > +
> > > +.. code-block:: c
> > > +
> > > + if (dev->txmap[i] & (1U << 31)) {
> > > +
> > > +Usage
> > > +-----
> > > +
> > > +make build
> > > +^^^^^^^^^^
> > > +
> > > +To enable UBSan, enable following configuration:
> > > +
> > > +.. code-block:: console
> > > +
> > > + CONFIG_RTE_UBSAN=y
> > > +
> > > +UBSan framework supports three modes:
> > > +
> > > +1. Enable UBSan on the entire DPDK source code - set following configuration:
> > > +
> > > +.. code-block:: console
> > > +
> > > + CONFIG_RTE_UBSAN_SANITIZE_ALL=y
> > > +
> > > +2. Enable UBSan on a particular library or PMD - add the following line to the
> > > + respective Makefile of the library or PMD
> > > + (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
> > > + the library or PMD and not the entire repository.
> > > +
> > > +.. code-block:: console
> > > +
> > > + UBSAN_SANITIZE := y
> > > +
> > > +3. Disable UBSan for a particular library or PMD - add the following line to
> > > + the respective Makefile of the library or PMD. Make sure
> > > + ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
> > > + entire DPDK repository but not this specific library or PMD.
> > > +
> > > +.. code-block:: console
> > > +
> > > + UBSAN_SANITIZE := n
> > > +
> > > +.. Note::
> > > +
> > > + Standard DPDK applications like test, testpmd, etc. cannot be
> > > + chosen explicitly for UBSan check, like libraries or PMD. The reason is,
> > > + say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
> > > + in Makefile of app Y which uses X APIs. This will lead to undefined
> > > + reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
> > > + Hence UBSan check is enabled for all standard DPDK applications as soon as
> > > + ``CONFIG_RTE_UBSAN=y`` is set.
> > > +
> > > +meson build
> > > +^^^^^^^^^^^
> > > +
> > > +To enable UBSan in meson build system, use following meson build command:
> > > +
> > > +**Example usage:**
> > > +
> > > +.. code-block:: console
> > > +
> > > + meson build -Denable_ubsan=true
> > > + ninja -C build
> > > +
> > > +.. Note::
> > > +
> > > + Meson build works only in one mode i.e. UBSan can be enabled for
> > > + the entire DPDK sources and not individual libraries or PMD, like make build.
> > > diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> > > index 8490d897c..cfefdbeec 100644
> > > --- a/doc/guides/rel_notes/release_19_11.rst
> > > +++ b/doc/guides/rel_notes/release_19_11.rst
> > > @@ -56,6 +56,13 @@ New Features
> > > Also, make sure to start the actual text at the margin.
> > > =========================================================
> > >
> > > +* **Added Undefined Behavior Sanitizer framework.**
> > > +
> > > + UBSan is a fast runtime undefined behavior detector which uses compile-time
> > > + instrumentation and modifies the program by adding some stubs that perform
> > > + certain checks before operations that might cause undefined behavior.
> > > +
> > > + See :doc:`../prog_guide/ubsan` for more information:
> > >
> > > Removed Items
> > > -------------
> > > diff --git a/meson_options.txt b/meson_options.txt
> > > index 448f3e63d..fb0aead00 100644
> > > --- a/meson_options.txt
> > > +++ b/meson_options.txt
> > > @@ -8,6 +8,8 @@ option('enable_docs', type: 'boolean', value: false,
> > > description: 'build documentation')
> > > option('enable_kmods', type: 'boolean', value: true,
> > > description: 'build kernel modules')
> > > +option('enable_ubsan', type: 'boolean', value: false,
> > > + description: 'Enables undefined behavior sanitizer')
> > > option('examples', type: 'string', value: '',
> > > description: 'Comma-separated list of examples to build by default')
> > > option('flexran_sdk', type: 'string', value: '',
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index ba5c39e01..9ea421a83 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -374,6 +374,16 @@ endif
> > >
> > > MAPFLAGS = -Map=$@.map --cref
> > >
> > > +#
> > > +# If UBSAN is enabled, all application will be compiled with
> > > +# '-fsanitize=undefined' flag
> > > +#
> > > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > > +ifeq ($(UBSAN_ENABLE),y)
> > > +CFLAGS += -fsanitize=undefined
> > > +endif
> > > +endif
> > > +
> > > .PHONY: all
> > > all: install
> > >
> > > diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> > > index 4df8849a0..33f5746c8 100644
> > > --- a/mk/rte.lib.mk
> > > +++ b/mk/rte.lib.mk
> > > @@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
> > > endif
> > > endif
> > >
> > > +#
> > > +# If UBSAN is enabled, lib to undergo check can be chosen
> > > +# by setting UBSAN_SANITIZE=y in respective lib Makefile
> > > +# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
> > > +# on entire repo.
> > > +#
> > > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > > +ifeq ($(UBSAN_ENABLE),y)
> > > +CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
> > > + , -fsanitize=undefined)
> > > +endif
> > > +endif
> > >
> > > _BUILD = $(LIB)
> > > PREINSTALL = $(SYMLINK-FILES-y)
> > > diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
> > > index 3c49dc568..623780106 100644
> > > --- a/mk/toolchain/clang/rte.vars.mk
> > > +++ b/mk/toolchain/clang/rte.vars.mk
> > > @@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
> > > WERROR_FLAGS += -Wno-address-of-packed-member
> > > endif
> > >
> > > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > > +UBSAN_ENABLE := y
> > > +endif
> > > +
> > > export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> > > export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> > > diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
> > > index b852fcfd7..d40e14d61 100644
> > > --- a/mk/toolchain/gcc/rte.vars.mk
> > > +++ b/mk/toolchain/gcc/rte.vars.mk
> > > @@ -90,5 +90,13 @@ endif
> > > # disable packed member unalign warnings
> > > WERROR_FLAGS += -Wno-address-of-packed-member
> > >
> > > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > > +ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
> > > +$(warning UBSAN not supported gcc < 4.9)
> > > +else
> > > +UBSAN_ENABLE = y
> > > +endif
> > > +endif
> > > +
> > > export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> > > export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> > > --
> > > 2.18.0
> > >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH] mk: add support for UBSAN
2019-08-19 13:48 [dpdk-dev] [PATCH] mk: add support for UBSAN Harman Kalra
2019-09-13 11:40 ` Harman Kalra
@ 2019-11-11 7:07 ` Thomas Monjalon
2019-11-15 14:46 ` [dpdk-dev] [EXT] " Harman Kalra
1 sibling, 1 reply; 10+ messages in thread
From: Thomas Monjalon @ 2019-11-11 7:07 UTC (permalink / raw)
To: Harman Kalra; +Cc: dev, John McNamara, Marko Kovacevic, Bruce Richardson
Hi,
Sorry for the very late review.
I hope someone else would try it.
I tried this:
devtools/test-build.sh -v x86_64-native-linux-clang+shared+UBSAN+SANITIZE_ALL
and it triggers some link errors:
/usr/bin/ld: rte_kvargs.c:(.text+0xc65): undefined reference to `__ubsan_handle_pointer_overflow'
19/08/2019 15:48, Harman Kalra:
> UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
> detector. UBSan modifies the program at compile-time to catch
> various kinds of undefined behavior during program execution.
>
> This patch implements support for UBSan to the DPDK.
>
> See: doc/guides/prog_guide/ubsan.rst for more information.
>
> Signed-off-by: Harman Kalra <hkalra@marvell.com>
> ---
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +ifeq ($(UBSAN_ENABLE),y)
This can be replaced with an oneline:
ifeq ($(CONFIG_RTE_UBSAN)$(UBSAN_ENABLE),yy)
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH] mk: add support for UBSAN
2019-11-11 7:07 ` Thomas Monjalon
@ 2019-11-15 14:46 ` Harman Kalra
2019-11-15 14:54 ` [dpdk-dev] [PATCH v2] " Harman Kalra
2019-11-16 8:31 ` [dpdk-dev] [EXT] Re: [PATCH] " Thomas Monjalon
0 siblings, 2 replies; 10+ messages in thread
From: Harman Kalra @ 2019-11-15 14:46 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: dev, John McNamara, Marko Kovacevic, Bruce Richardson
On Mon, Nov 11, 2019 at 08:07:00AM +0100, Thomas Monjalon wrote:
> External Email
>
> ----------------------------------------------------------------------
> Hi,
>
> Sorry for the very late review.
> I hope someone else would try it.
>
> I tried this:
> devtools/test-build.sh -v x86_64-native-linux-clang+shared+UBSAN+SANITIZE_ALL
> and it triggers some link errors:
> /usr/bin/ld: rte_kvargs.c:(.text+0xc65): undefined reference to `__ubsan_handle_pointer_overflow'
Hi,
Thanks for trying it out. I came across these errors when compiler
versions doesn't supports UBSAN
Can you please with latest clang version if issue still persists.
>
>
> 19/08/2019 15:48, Harman Kalra:
> > UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
> > detector. UBSan modifies the program at compile-time to catch
> > various kinds of undefined behavior during program execution.
> >
> > This patch implements support for UBSan to the DPDK.
> >
> > See: doc/guides/prog_guide/ubsan.rst for more information.
> >
> > Signed-off-by: Harman Kalra <hkalra@marvell.com>
> > ---
> > +ifeq ($(CONFIG_RTE_UBSAN),y)
> > +ifeq ($(UBSAN_ENABLE),y)
>
> This can be replaced with an oneline:
>
> ifeq ($(CONFIG_RTE_UBSAN)$(UBSAN_ENABLE),yy)
Ack, will do these changes in V2
>
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH v2] mk: add support for UBSAN
2019-11-15 14:46 ` [dpdk-dev] [EXT] " Harman Kalra
@ 2019-11-15 14:54 ` Harman Kalra
2019-11-15 15:34 ` [dpdk-dev] [PATCH v3] " Harman Kalra
2019-11-16 8:31 ` [dpdk-dev] [EXT] Re: [PATCH] " Thomas Monjalon
1 sibling, 1 reply; 10+ messages in thread
From: Harman Kalra @ 2019-11-15 14:54 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson
Cc: dev, Harman Kalra
UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
detector. UBSan modifies the program at compile-time to catch
various kinds of undefined behavior during program execution.
This patch introduces support for UBSan to the DPDK.
See: doc/guides/prog_guide/ubsan.rst for more information.
Signed-off-by: Harman Kalra <hkalra@marvell.com>
---
config/common_base | 6 ++
config/meson.build | 15 ++++
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
doc/guides/rel_notes/release_19_11.rst | 7 ++
meson_options.txt | 2 +
mk/rte.app.mk | 8 ++
mk/rte.lib.mk | 12 +++
mk/toolchain/clang/rte.vars.mk | 4 +
mk/toolchain/gcc/rte.vars.mk | 8 ++
10 files changed, 175 insertions(+)
create mode 100644 doc/guides/prog_guide/ubsan.rst
diff --git a/config/common_base b/config/common_base
index 914277856..f1bb3e0b2 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1098,3 +1098,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
# Compile the eventdev application
#
CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Enable undefined behavior sanitizer
+#
+CONFIG_RTE_UBSAN=n
+CONFIG_RTE_UBSAN_SANITIZE_ALL=n
diff --git a/config/meson.build b/config/meson.build
index 2b1cb92e7..a43c23f50 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -238,3 +238,18 @@ if get_option('b_lto')
add_project_link_arguments('-Wno-lto-type-mismatch', language: 'c')
endif
endif
+
+# enable ubsan
+if get_option('enable_ubsan')
+ if cc.has_argument('-fsanitize=undefined')
+ ubsan_dep = cc.find_library('libubsan', required: false)
+ if ubsan_dep.found()
+ add_project_arguments('-fsanitize=undefined', language: 'c')
+ add_project_link_arguments('-fsanitize=undefined', language: 'c')
+ else
+ message('libubsan not found, UBSAN cannot be enabled')
+ endif
+ else
+ message('gcc version does not support UBSAN')
+ endif
+endif
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index dc4851c57..911b82a41 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -67,4 +67,5 @@ Programmer's Guide
writing_efficient_code
lto
profile_app
+ ubsan
glossary
diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
new file mode 100644
index 000000000..cb19f3bd9
--- /dev/null
+++ b/doc/guides/prog_guide/ubsan.rst
@@ -0,0 +1,112 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2019 Marvell International Ltd.
+
+The Undefined Behavior Sanitizer - UBSan
+========================================
+
+UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
+UBSan uses compile-time instrumentation and modifies the program by adding
+some stubs which perform certain checks before operations that might cause
+undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
+(which are defined in libUBSan library) are called to prints the error message.
+
+Some examples of undefined behaviour checks:
+
+* Misaligned memory access
+* Signed integer overflow
+* Load from/store to an object with insufficient space.
+* Integer divide by zero as well as INT_MIN / -1 division
+* Out-of-bounds memory accesses.
+* Null argument declared with nonnull attribute, returned null from function
+ which never returns null, null ptr dereference
+* Variable size array with non-positive length
+
+GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
+checkers implemented.
+
+Example UBSan error
+--------------------
+
+Following error was reported when UBSan was enabled:
+
+.. code-block:: console
+
+ drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
+ 1 by 31 places cannot be represented in type 'int'
+
+Code responsible for this error:
+
+.. code-block:: c
+
+ if (dev->txmap[i] & (1 << 31)) {
+
+To fix this error:
+
+.. code-block:: c
+
+ if (dev->txmap[i] & (1U << 31)) {
+
+Usage
+-----
+
+make build
+^^^^^^^^^^
+
+To enable UBSan, enable following configuration:
+
+.. code-block:: console
+
+ CONFIG_RTE_UBSAN=y
+
+UBSan framework supports three modes:
+
+1. Enable UBSan on the entire DPDK source code - set following configuration:
+
+.. code-block:: console
+
+ CONFIG_RTE_UBSAN_SANITIZE_ALL=y
+
+2. Enable UBSan on a particular library or PMD - add the following line to the
+ respective Makefile of the library or PMD
+ (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
+ the library or PMD and not the entire repository.
+
+.. code-block:: console
+
+ UBSAN_SANITIZE := y
+
+3. Disable UBSan for a particular library or PMD - add the following line to
+ the respective Makefile of the library or PMD. Make sure
+ ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
+ entire DPDK repository but not this specific library or PMD.
+
+.. code-block:: console
+
+ UBSAN_SANITIZE := n
+
+.. Note::
+
+ Standard DPDK applications like test, testpmd, etc. cannot be
+ chosen explicitly for UBSan check, like libraries or PMD. The reason is,
+ say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
+ in Makefile of app Y which uses X APIs. This will lead to undefined
+ reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
+ Hence UBSan check is enabled for all standard DPDK applications as soon as
+ ``CONFIG_RTE_UBSAN=y`` is set.
+
+meson build
+^^^^^^^^^^^
+
+To enable UBSan in meson build system, use following meson build command:
+
+**Example usage:**
+
+.. code-block:: console
+
+ meson build -Denable_ubsan=true
+ ninja -C build
+
+.. Note::
+
+ Meson build works only in one mode i.e. UBSan can be enabled for
+ the entire DPDK sources and not individual libraries or PMD, like make build.
diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index c0045a91f..61fd1bcc2 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -294,6 +294,13 @@ New Features
See :doc:`../prog_guide/lto` for more information:
+* **Added Undefined Behavior Sanitizer framework.**
+
+ UBSan is a fast runtime undefined behavior detector which uses compile-time
+ instrumentation and modifies the program by adding some stubs that perform
+ certain checks before operations that might cause undefined behavior.
+
+ See :doc:`../prog_guide/ubsan` for more information:
Removed Items
diff --git a/meson_options.txt b/meson_options.txt
index 89650b0e9..f3b42d2b1 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -10,6 +10,8 @@ option('enable_docs', type: 'boolean', value: false,
description: 'build documentation')
option('enable_kmods', type: 'boolean', value: true,
description: 'build kernel modules')
+option('enable_ubsan', type: 'boolean', value: false,
+ description: 'Enables undefined behavior sanitizer')
option('examples', type: 'string', value: '',
description: 'Comma-separated list of examples to build by default')
option('flexran_sdk', type: 'string', value: '',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 683e3a4e3..1304227cf 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -385,6 +385,14 @@ endif
MAPFLAGS = -Map=$@.map --cref
+#
+# If UBSAN is enabled, all application will be compiled with
+# '-fsanitize=undefined' flag
+#
+ifeq ($(CONFIG_RTE_UBSAN)$(UBSAN_ENABLE),yy)
+CFLAGS += -fsanitize=undefined
+endif
+
.PHONY: all
all: install
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 4df8849a0..33f5746c8 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
endif
endif
+#
+# If UBSAN is enabled, lib to undergo check can be chosen
+# by setting UBSAN_SANITIZE=y in respective lib Makefile
+# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
+# on entire repo.
+#
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(UBSAN_ENABLE),y)
+CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
+ , -fsanitize=undefined)
+endif
+endif
_BUILD = $(LIB)
PREINSTALL = $(SYMLINK-FILES-y)
diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
index 3c49dc568..623780106 100644
--- a/mk/toolchain/clang/rte.vars.mk
+++ b/mk/toolchain/clang/rte.vars.mk
@@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
WERROR_FLAGS += -Wno-address-of-packed-member
endif
+ifeq ($(CONFIG_RTE_UBSAN),y)
+UBSAN_ENABLE := y
+endif
+
export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
index 9fc704193..43e7d139b 100644
--- a/mk/toolchain/gcc/rte.vars.mk
+++ b/mk/toolchain/gcc/rte.vars.mk
@@ -102,5 +102,13 @@ endif
# disable packed member unalign warnings
WERROR_FLAGS += -Wno-address-of-packed-member
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
+$(warning UBSAN not supported gcc < 4.9)
+else
+UBSAN_ENABLE = y
+endif
+endif
+
export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
--
2.18.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH v3] mk: add support for UBSAN
2019-11-15 14:54 ` [dpdk-dev] [PATCH v2] " Harman Kalra
@ 2019-11-15 15:34 ` Harman Kalra
2019-11-27 14:23 ` Aaron Conole
0 siblings, 1 reply; 10+ messages in thread
From: Harman Kalra @ 2019-11-15 15:34 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson
Cc: dev, Harman Kalra
UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
detector. UBSan modifies the program at compile-time to catch
various kinds of undefined behavior during program execution.
This patch introduces support for UBSan to the DPDK.
See: doc/guides/prog_guide/ubsan.rst for more information.
Signed-off-by: Harman Kalra <hkalra@marvell.com>
---
V2:
* Addressed review comment regarding combining two
ifeq into one.
V3:
* Added version change logs.
config/common_base | 6 ++
config/meson.build | 15 ++++
doc/guides/prog_guide/index.rst | 1 +
doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
doc/guides/rel_notes/release_19_11.rst | 7 ++
meson_options.txt | 2 +
mk/rte.app.mk | 8 ++
mk/rte.lib.mk | 12 +++
mk/toolchain/clang/rte.vars.mk | 4 +
mk/toolchain/gcc/rte.vars.mk | 8 ++
10 files changed, 175 insertions(+)
create mode 100644 doc/guides/prog_guide/ubsan.rst
diff --git a/config/common_base b/config/common_base
index 914277856..f1bb3e0b2 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1098,3 +1098,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
# Compile the eventdev application
#
CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Enable undefined behavior sanitizer
+#
+CONFIG_RTE_UBSAN=n
+CONFIG_RTE_UBSAN_SANITIZE_ALL=n
diff --git a/config/meson.build b/config/meson.build
index 2b1cb92e7..a43c23f50 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -238,3 +238,18 @@ if get_option('b_lto')
add_project_link_arguments('-Wno-lto-type-mismatch', language: 'c')
endif
endif
+
+# enable ubsan
+if get_option('enable_ubsan')
+ if cc.has_argument('-fsanitize=undefined')
+ ubsan_dep = cc.find_library('libubsan', required: false)
+ if ubsan_dep.found()
+ add_project_arguments('-fsanitize=undefined', language: 'c')
+ add_project_link_arguments('-fsanitize=undefined', language: 'c')
+ else
+ message('libubsan not found, UBSAN cannot be enabled')
+ endif
+ else
+ message('gcc version does not support UBSAN')
+ endif
+endif
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index dc4851c57..911b82a41 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -67,4 +67,5 @@ Programmer's Guide
writing_efficient_code
lto
profile_app
+ ubsan
glossary
diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
new file mode 100644
index 000000000..cb19f3bd9
--- /dev/null
+++ b/doc/guides/prog_guide/ubsan.rst
@@ -0,0 +1,112 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2019 Marvell International Ltd.
+
+The Undefined Behavior Sanitizer - UBSan
+========================================
+
+UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
+UBSan uses compile-time instrumentation and modifies the program by adding
+some stubs which perform certain checks before operations that might cause
+undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
+(which are defined in libUBSan library) are called to prints the error message.
+
+Some examples of undefined behaviour checks:
+
+* Misaligned memory access
+* Signed integer overflow
+* Load from/store to an object with insufficient space.
+* Integer divide by zero as well as INT_MIN / -1 division
+* Out-of-bounds memory accesses.
+* Null argument declared with nonnull attribute, returned null from function
+ which never returns null, null ptr dereference
+* Variable size array with non-positive length
+
+GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
+checkers implemented.
+
+Example UBSan error
+--------------------
+
+Following error was reported when UBSan was enabled:
+
+.. code-block:: console
+
+ drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
+ 1 by 31 places cannot be represented in type 'int'
+
+Code responsible for this error:
+
+.. code-block:: c
+
+ if (dev->txmap[i] & (1 << 31)) {
+
+To fix this error:
+
+.. code-block:: c
+
+ if (dev->txmap[i] & (1U << 31)) {
+
+Usage
+-----
+
+make build
+^^^^^^^^^^
+
+To enable UBSan, enable following configuration:
+
+.. code-block:: console
+
+ CONFIG_RTE_UBSAN=y
+
+UBSan framework supports three modes:
+
+1. Enable UBSan on the entire DPDK source code - set following configuration:
+
+.. code-block:: console
+
+ CONFIG_RTE_UBSAN_SANITIZE_ALL=y
+
+2. Enable UBSan on a particular library or PMD - add the following line to the
+ respective Makefile of the library or PMD
+ (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
+ the library or PMD and not the entire repository.
+
+.. code-block:: console
+
+ UBSAN_SANITIZE := y
+
+3. Disable UBSan for a particular library or PMD - add the following line to
+ the respective Makefile of the library or PMD. Make sure
+ ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
+ entire DPDK repository but not this specific library or PMD.
+
+.. code-block:: console
+
+ UBSAN_SANITIZE := n
+
+.. Note::
+
+ Standard DPDK applications like test, testpmd, etc. cannot be
+ chosen explicitly for UBSan check, like libraries or PMD. The reason is,
+ say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
+ in Makefile of app Y which uses X APIs. This will lead to undefined
+ reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
+ Hence UBSan check is enabled for all standard DPDK applications as soon as
+ ``CONFIG_RTE_UBSAN=y`` is set.
+
+meson build
+^^^^^^^^^^^
+
+To enable UBSan in meson build system, use following meson build command:
+
+**Example usage:**
+
+.. code-block:: console
+
+ meson build -Denable_ubsan=true
+ ninja -C build
+
+.. Note::
+
+ Meson build works only in one mode i.e. UBSan can be enabled for
+ the entire DPDK sources and not individual libraries or PMD, like make build.
diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index c0045a91f..61fd1bcc2 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -294,6 +294,13 @@ New Features
See :doc:`../prog_guide/lto` for more information:
+* **Added Undefined Behavior Sanitizer framework.**
+
+ UBSan is a fast runtime undefined behavior detector which uses compile-time
+ instrumentation and modifies the program by adding some stubs that perform
+ certain checks before operations that might cause undefined behavior.
+
+ See :doc:`../prog_guide/ubsan` for more information:
Removed Items
diff --git a/meson_options.txt b/meson_options.txt
index 89650b0e9..f3b42d2b1 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -10,6 +10,8 @@ option('enable_docs', type: 'boolean', value: false,
description: 'build documentation')
option('enable_kmods', type: 'boolean', value: true,
description: 'build kernel modules')
+option('enable_ubsan', type: 'boolean', value: false,
+ description: 'Enables undefined behavior sanitizer')
option('examples', type: 'string', value: '',
description: 'Comma-separated list of examples to build by default')
option('flexran_sdk', type: 'string', value: '',
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 683e3a4e3..1304227cf 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -385,6 +385,14 @@ endif
MAPFLAGS = -Map=$@.map --cref
+#
+# If UBSAN is enabled, all application will be compiled with
+# '-fsanitize=undefined' flag
+#
+ifeq ($(CONFIG_RTE_UBSAN)$(UBSAN_ENABLE),yy)
+CFLAGS += -fsanitize=undefined
+endif
+
.PHONY: all
all: install
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 4df8849a0..33f5746c8 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
endif
endif
+#
+# If UBSAN is enabled, lib to undergo check can be chosen
+# by setting UBSAN_SANITIZE=y in respective lib Makefile
+# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
+# on entire repo.
+#
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(UBSAN_ENABLE),y)
+CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
+ , -fsanitize=undefined)
+endif
+endif
_BUILD = $(LIB)
PREINSTALL = $(SYMLINK-FILES-y)
diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
index 3c49dc568..623780106 100644
--- a/mk/toolchain/clang/rte.vars.mk
+++ b/mk/toolchain/clang/rte.vars.mk
@@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
WERROR_FLAGS += -Wno-address-of-packed-member
endif
+ifeq ($(CONFIG_RTE_UBSAN),y)
+UBSAN_ENABLE := y
+endif
+
export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
index 9fc704193..43e7d139b 100644
--- a/mk/toolchain/gcc/rte.vars.mk
+++ b/mk/toolchain/gcc/rte.vars.mk
@@ -102,5 +102,13 @@ endif
# disable packed member unalign warnings
WERROR_FLAGS += -Wno-address-of-packed-member
+ifeq ($(CONFIG_RTE_UBSAN),y)
+ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
+$(warning UBSAN not supported gcc < 4.9)
+else
+UBSAN_ENABLE = y
+endif
+endif
+
export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
--
2.18.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH] mk: add support for UBSAN
2019-11-15 14:46 ` [dpdk-dev] [EXT] " Harman Kalra
2019-11-15 14:54 ` [dpdk-dev] [PATCH v2] " Harman Kalra
@ 2019-11-16 8:31 ` Thomas Monjalon
1 sibling, 0 replies; 10+ messages in thread
From: Thomas Monjalon @ 2019-11-16 8:31 UTC (permalink / raw)
To: Harman Kalra; +Cc: dev, John McNamara, Marko Kovacevic, Bruce Richardson
15/11/2019 15:46, Harman Kalra:
> On Mon, Nov 11, 2019 at 08:07:00AM +0100, Thomas Monjalon wrote:
> > External Email
> >
> > ----------------------------------------------------------------------
> > Hi,
> >
> > Sorry for the very late review.
> > I hope someone else would try it.
> >
> > I tried this:
> > devtools/test-build.sh -v x86_64-native-linux-clang+shared+UBSAN+SANITIZE_ALL
> > and it triggers some link errors:
> > /usr/bin/ld: rte_kvargs.c:(.text+0xc65): undefined reference to `__ubsan_handle_pointer_overflow'
> Hi,
>
> Thanks for trying it out. I came across these errors when compiler
> versions doesn't supports UBSAN
If the support is not available, we should print a clear error message,
not random link errors.
> Can you please with latest clang version if issue still persists.
I am using clang 8.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH v3] mk: add support for UBSAN
2019-11-15 15:34 ` [dpdk-dev] [PATCH v3] " Harman Kalra
@ 2019-11-27 14:23 ` Aaron Conole
0 siblings, 0 replies; 10+ messages in thread
From: Aaron Conole @ 2019-11-27 14:23 UTC (permalink / raw)
To: Harman Kalra
Cc: Thomas Monjalon, John McNamara, Marko Kovacevic, Bruce Richardson, dev
Harman Kalra <hkalra@marvell.com> writes:
> UndefinedBehaviorSanitizer (UBSan) is a fast undefined behavior
> detector. UBSan modifies the program at compile-time to catch
> various kinds of undefined behavior during program execution.
>
> This patch introduces support for UBSan to the DPDK.
>
> See: doc/guides/prog_guide/ubsan.rst for more information.
>
> Signed-off-by: Harman Kalra <hkalra@marvell.com>
> ---
Sorry I am coming to this late.
> V2:
> * Addressed review comment regarding combining two
> ifeq into one.
>
> V3:
> * Added version change logs.
>
> config/common_base | 6 ++
> config/meson.build | 15 ++++
> doc/guides/prog_guide/index.rst | 1 +
> doc/guides/prog_guide/ubsan.rst | 112 +++++++++++++++++++++++++
> doc/guides/rel_notes/release_19_11.rst | 7 ++
> meson_options.txt | 2 +
> mk/rte.app.mk | 8 ++
> mk/rte.lib.mk | 12 +++
> mk/toolchain/clang/rte.vars.mk | 4 +
> mk/toolchain/gcc/rte.vars.mk | 8 ++
> 10 files changed, 175 insertions(+)
> create mode 100644 doc/guides/prog_guide/ubsan.rst
>
> diff --git a/config/common_base b/config/common_base
> index 914277856..f1bb3e0b2 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -1098,3 +1098,9 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
> # Compile the eventdev application
> #
> CONFIG_RTE_APP_EVENTDEV=y
> +
> +#
> +# Enable undefined behavior sanitizer
> +#
> +CONFIG_RTE_UBSAN=n
> +CONFIG_RTE_UBSAN_SANITIZE_ALL=n
> diff --git a/config/meson.build b/config/meson.build
> index 2b1cb92e7..a43c23f50 100644
> --- a/config/meson.build
> +++ b/config/meson.build
> @@ -238,3 +238,18 @@ if get_option('b_lto')
> add_project_link_arguments('-Wno-lto-type-mismatch', language: 'c')
> endif
> endif
> +
> +# enable ubsan
> +if get_option('enable_ubsan')
> + if cc.has_argument('-fsanitize=undefined')
> + ubsan_dep = cc.find_library('libubsan', required: false)
> + if ubsan_dep.found()
> + add_project_arguments('-fsanitize=undefined', language: 'c')
> + add_project_link_arguments('-fsanitize=undefined', language: 'c')
> + else
> + message('libubsan not found, UBSAN cannot be enabled')
> + endif
> + else
> + message('gcc version does not support UBSAN')
> + endif
> +endif
Why is this needed? AFAIK, meson supports
-Db_sanitize=undefined,address,... so what do we gain by this?
Especially since for ubsan and asan, -fno-omit-frame-pointer is needed
for useful backtraces (which the meson module does for us). This
support has been in since 2017, afaict.
> diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
> index dc4851c57..911b82a41 100644
> --- a/doc/guides/prog_guide/index.rst
> +++ b/doc/guides/prog_guide/index.rst
> @@ -67,4 +67,5 @@ Programmer's Guide
> writing_efficient_code
> lto
> profile_app
> + ubsan
> glossary
> diff --git a/doc/guides/prog_guide/ubsan.rst b/doc/guides/prog_guide/ubsan.rst
> new file mode 100644
> index 000000000..cb19f3bd9
> --- /dev/null
> +++ b/doc/guides/prog_guide/ubsan.rst
> @@ -0,0 +1,112 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2019 Marvell International Ltd.
> +
> +The Undefined Behavior Sanitizer - UBSan
> +=======================================+
> +UndefinedBehaviorSanitizer (UBSan) is a runtime undefined behavior detector.
> +UBSan uses compile-time instrumentation and modifies the program by adding
> +some stubs which perform certain checks before operations that might cause
> +undefined behaviour. If some UB detected, respective _UBSan_handle_* handlers
> +(which are defined in libUBSan library) are called to prints the error message.
> +
> +Some examples of undefined behaviour checks:
> +
> +* Misaligned memory access
> +* Signed integer overflow
> +* Load from/store to an object with insufficient space.
> +* Integer divide by zero as well as INT_MIN / -1 division
> +* Out-of-bounds memory accesses.
> +* Null argument declared with nonnull attribute, returned null from function
> + which never returns null, null ptr dereference
> +* Variable size array with non-positive length
> +
> +GCC supports this feature since 4.9, however GCC 5.0 onwards has many more
> +checkers implemented.
> +
> +Example UBSan error
> +--------------------
> +
> +Following error was reported when UBSan was enabled:
> +
> +.. code-block:: console
> +
> + drivers/net/octeontx2/otx2_stats.c:82:26: runtime error: left shift of
> + 1 by 31 places cannot be represented in type 'int'
> +
> +Code responsible for this error:
> +
> +.. code-block:: c
> +
> + if (dev->txmap[i] & (1 << 31)) {
> +
> +To fix this error:
> +
> +.. code-block:: c
> +
> + if (dev->txmap[i] & (1U << 31)) {
> +
> +Usage
> +-----
> +
> +make build
> +^^^^^^^^^^
> +
> +To enable UBSan, enable following configuration:
> +
> +.. code-block:: console
> +
> + CONFIG_RTE_UBSAN=y
> +
> +UBSan framework supports three modes:
> +
> +1. Enable UBSan on the entire DPDK source code - set following configuration:
> +
> +.. code-block:: console
> +
> + CONFIG_RTE_UBSAN_SANITIZE_ALL=y
> +
> +2. Enable UBSan on a particular library or PMD - add the following line to the
> + respective Makefile of the library or PMD
> + (make sure ``CONFIG_RTE_UBSAN_SANITIZE_ALL=n``). This will instrument only
> + the library or PMD and not the entire repository.
> +
> +.. code-block:: console
> +
> + UBSAN_SANITIZE := y
> +
> +3. Disable UBSan for a particular library or PMD - add the following line to
> + the respective Makefile of the library or PMD. Make sure
> + ``CONFIG_RTE_UBSAN_SANITIZE_ALL=y`` config is set. This will instrument
> + entire DPDK repository but not this specific library or PMD.
> +
> +.. code-block:: console
> +
> + UBSAN_SANITIZE := n
> +
> +.. Note::
> +
> + Standard DPDK applications like test, testpmd, etc. cannot be
> + chosen explicitly for UBSan check, like libraries or PMD. The reason is,
> + say UBSan is enabled for library X, and ``UBSAN_SANITIZE=y`` is not added
> + in Makefile of app Y which uses X APIs. This will lead to undefined
> + reference to _UBSan_handle_* handlers as Y is not compiled with UBSan flags.
> + Hence UBSan check is enabled for all standard DPDK applications as soon as
> + ``CONFIG_RTE_UBSAN=y`` is set.
> +
> +meson build
> +^^^^^^^^^^^
> +
> +To enable UBSan in meson build system, use following meson build command:
> +
> +**Example usage:**
> +
> +.. code-block:: console
> +
> + meson build -Denable_ubsan=true
> + ninja -C build
> +
> +.. Note::
> +
> + Meson build works only in one mode i.e. UBSan can be enabled for
> + the entire DPDK sources and not individual libraries or PMD, like make build.
> diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> index c0045a91f..61fd1bcc2 100644
> --- a/doc/guides/rel_notes/release_19_11.rst
> +++ b/doc/guides/rel_notes/release_19_11.rst
> @@ -294,6 +294,13 @@ New Features
>
> See :doc:`../prog_guide/lto` for more information:
>
> +* **Added Undefined Behavior Sanitizer framework.**
> +
> + UBSan is a fast runtime undefined behavior detector which uses compile-time
> + instrumentation and modifies the program by adding some stubs that perform
> + certain checks before operations that might cause undefined behavior.
> +
> + See :doc:`../prog_guide/ubsan` for more information:
>
>
> Removed Items
> diff --git a/meson_options.txt b/meson_options.txt
> index 89650b0e9..f3b42d2b1 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -10,6 +10,8 @@ option('enable_docs', type: 'boolean', value: false,
> description: 'build documentation')
> option('enable_kmods', type: 'boolean', value: true,
> description: 'build kernel modules')
> +option('enable_ubsan', type: 'boolean', value: false,
> + description: 'Enables undefined behavior sanitizer')
> option('examples', type: 'string', value: '',
> description: 'Comma-separated list of examples to build by default')
> option('flexran_sdk', type: 'string', value: '',
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index 683e3a4e3..1304227cf 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -385,6 +385,14 @@ endif
>
> MAPFLAGS = -Map=$@.map --cref
>
> +#
> +# If UBSAN is enabled, all application will be compiled with
> +# '-fsanitize=undefined' flag
> +#
> +ifeq ($(CONFIG_RTE_UBSAN)$(UBSAN_ENABLE),yy)
> +CFLAGS += -fsanitize=undefined
> +endif
> +
> .PHONY: all
> all: install
>
> diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
> index 4df8849a0..33f5746c8 100644
> --- a/mk/rte.lib.mk
> +++ b/mk/rte.lib.mk
> @@ -29,6 +29,18 @@ CPU_LDFLAGS += --version-script=$(SRCDIR)/$(EXPORT_MAP)
> endif
> endif
>
> +#
> +# If UBSAN is enabled, lib to undergo check can be chosen
> +# by setting UBSAN_SANITIZE=y in respective lib Makefile
> +# else set CONFIG_RTE_UBSAN_SANITIZE_ALL=y to enforce check
> +# on entire repo.
> +#
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +ifeq ($(UBSAN_ENABLE),y)
> +CFLAGS += $(if $(patsubst %n,,$(CONFIG_RTE_UBSAN_SANITIZE_ALL)$(UBSAN_SANITIZE)) \
> + , -fsanitize=undefined)
> +endif
> +endif
>
> _BUILD = $(LIB)
> PREINSTALL = $(SYMLINK-FILES-y)
> diff --git a/mk/toolchain/clang/rte.vars.mk b/mk/toolchain/clang/rte.vars.mk
> index 3c49dc568..623780106 100644
> --- a/mk/toolchain/clang/rte.vars.mk
> +++ b/mk/toolchain/clang/rte.vars.mk
> @@ -56,5 +56,9 @@ ifeq ($(shell test $(CLANG_MAJOR_VERSION) -ge 4 && echo 1), 1)
> WERROR_FLAGS += -Wno-address-of-packed-member
> endif
>
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +UBSAN_ENABLE := y
> +endif
> +
> export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
> diff --git a/mk/toolchain/gcc/rte.vars.mk b/mk/toolchain/gcc/rte.vars.mk
> index 9fc704193..43e7d139b 100644
> --- a/mk/toolchain/gcc/rte.vars.mk
> +++ b/mk/toolchain/gcc/rte.vars.mk
> @@ -102,5 +102,13 @@ endif
> # disable packed member unalign warnings
> WERROR_FLAGS += -Wno-address-of-packed-member
>
> +ifeq ($(CONFIG_RTE_UBSAN),y)
> +ifeq ($(shell test $(GCC_VERSION) -lt 49 && echo 1), 1)
> +$(warning UBSAN not supported gcc < 4.9)
> +else
> +UBSAN_ENABLE = y
> +endif
> +endif
> +
> export CC AS AR LD OBJCOPY OBJDUMP STRIP READELF
> export TOOLCHAIN_CFLAGS TOOLCHAIN_LDFLAGS TOOLCHAIN_ASFLAGS
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-11-27 14:23 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-19 13:48 [dpdk-dev] [PATCH] mk: add support for UBSAN Harman Kalra
2019-09-13 11:40 ` Harman Kalra
2019-10-09 14:25 ` Harman Kalra
2019-10-28 10:50 ` Harman Kalra
2019-11-11 7:07 ` Thomas Monjalon
2019-11-15 14:46 ` [dpdk-dev] [EXT] " Harman Kalra
2019-11-15 14:54 ` [dpdk-dev] [PATCH v2] " Harman Kalra
2019-11-15 15:34 ` [dpdk-dev] [PATCH v3] " Harman Kalra
2019-11-27 14:23 ` Aaron Conole
2019-11-16 8:31 ` [dpdk-dev] [EXT] Re: [PATCH] " Thomas Monjalon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).