From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iN8ko-0002BM-II for linux-um@lists.infradead.org; Wed, 23 Oct 2019 05:00:23 +0000 Received: by mail-pl1-x642.google.com with SMTP id q15so9467860pll.11 for ; Tue, 22 Oct 2019 22:00:14 -0700 (PDT) From: Hajime Tazaki Subject: [RFC PATCH 30/47] lkl: add documentation Date: Wed, 23 Oct 2019 13:38:04 +0900 Message-Id: In-Reply-To: References: MIME-Version: 1.0 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: linux-um@lists.infradead.org Cc: "H . K . Jerry Chu" , Conrad Meyer , Octavian Purdila , Motomu Utsumi , Akira Moroo , Thomas Liebetraut , Patrick Collins , Chenyang Zhong , Yuan Liu , Gustavo Bittencourt From: Octavian Purdila We also added a symlink to README.md to display at github. Signed-off-by: Chenyang Zhong Signed-off-by: Conrad Meyer Signed-off-by: Gustavo Bittencourt Signed-off-by: H.K. Jerry Chu Signed-off-by: Motomu Utsumi Signed-off-by: Patrick Collins Signed-off-by: Thomas Liebetraut Signed-off-by: Yuan Liu Signed-off-by: Octavian Purdila --- Documentation/lkl.txt | 470 ++++++++++++++++++++++++++++++++++++++++++ README.md | 1 + 2 files changed, 471 insertions(+) create mode 100644 Documentation/lkl.txt create mode 120000 README.md diff --git a/Documentation/lkl.txt b/Documentation/lkl.txt new file mode 100644 index 000000000000..97fe407b0bc6 --- /dev/null +++ b/Documentation/lkl.txt @@ -0,0 +1,470 @@ + +Introduction +============ + +LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code as +extensively as possible with minimal effort and reduced maintenance overhead. + +Examples of how LKL can be used are: creating userspace applications (running on +Linux and other operating systems) that can read or write Linux filesystems or +can use the Linux networking stack, creating kernel drivers for other operating +systems that can read Linux filesystems, bootloaders support for reading/writing +Linux filesystems, etc. + +With LKL, the kernel code is compiled into an object file that can be directly +linked by applications. The API offered by LKL is based on the Linux system call +interface. + +LKL is implemented as an architecture port in arch/lkl. It uses host operations +defined by the application or a host library (tools/lkl/lib). + + +Supported hosts +=============== + +The supported hosts for now are POSIX and Windows userspace applications. + + +Building LKL, the host library and LKL based tools +================================================== + + $ make -C tools/lkl + +will build LKL as a object file, it will install it in tools/lkl/lib together +with the headers files in tools/lkl/include then will build the host library, +tests and a few of application examples: + +* tests/boot - a simple applications that uses LKL and exercises the basic LKL +APIs + +* fs2tar - a tool that converts a filesystem image to a tar archive + +* cptofs/cpfromfs - a tool that copies files to/from a filesystem image + +* lklfuse - a tool that can mount a filesystem image in userspace, + without root privileges, using FUSE + + +Building LKL on FreeBSD +----------------------- + + $ pkg install binutils gcc gnubc gmake gsed coreutils bison flex python argp-standalone + + #Prefer ports binutils and GNU bc(1): + $ export PATH=/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/lib64/ccache + + $ gmake -C tools/lkl + +Building LKL on Ubuntu +----------------------- + + $ sudo apt-get install libfuse-dev libarchive-dev xfsprogs + + # Optional, if you would like to be able to run tests + $ sudo apt-get install btrfs-tools + $ pip install yamlish junit_xml + + $ make -C tools/lkl + + # To check that everything works: + $ cd tools/lkl + $ make run-tests + + +Building LKL for Windows +------------------------ + +In order to build LKL for Win32 the mingw cross compiler needs to be installed +on the host (e.g. on Ubuntu the following packages are required: +binutils-mingw-w64-i686, gcc-mingw-w64-base, gcc-mingw-w64-i686 +mingw-w64-common, mingw-w64-i686-dev). + +Due to a bug in mingw regarding weak symbols the following patches needs to be +applied to mingw-binutils: + +https://sourceware.org/ml/binutils/2015-10/msg00234.html + +and i686-w64-mingw32-gas, i686-w64-mingw32-ld and i686-w64-mingw32-objcopy need +to be recompiled. + +With that pre-requisites fullfilled you can now build LKL for Win32 with the +following command: + + $ make CROSS_COMPILE=i686-w64-mingw32- -C tools/lkl + + + +Building LKL on Windows +------------------------ + +To build on Windows, certain GNU tools need to be installed. These tools can come +from several different projects, such as cygwin, unxutils, gnu-win32 or busybox-w32. +Below is one minimal/modular set-up based on msys2. + +### Common build dependencies: +* [MSYS2](https://sourceforge.net/projects/msys2/) (provides GNU bash and many other utilities) +* Extra utilities from MSYS2/pacman: bc, base-devel + +### General considerations: +* No spaces in pathnames (source, prefix, destination,...)! +* Make sure that all utilities are in the PATH. +* Win64 (and MinGW 64-bit crt) is LLP64, which causes conflicts in size of "long" in the +Linux source. Linux (and lkl) can (currently) not +be built on LLP64. +* Cygwin (and msys2) are LP64, like linux. + +### For MSYS2 (and Cygwin): +Msys2 will install a gcc tool chain as part of the base-devel bundle. Binutils (2.26) is already +patched for NT weak externals. Using the msys2 shell, cd to the lkl sources and run: + + $ make -C tools/lkl + +### For MinGW: +Install mingw-w64-i686-toolchain via pacman, mingw-w64-i686-binutils (2.26) is already patched +for NT weak externals. Start a MinGW Win32 shell (64-bit will not work, see above) +and run: + + $ make -C tools/lkl + + +LKL hijack library +================== + +LKL hijack library (liblkl-hijack.so) is used to replace system calls used by an +application on the fly so that the application can use LKL instead of the kernel +of host operating system. LD_PRELOAD is used to dynamically override system +calls with this library when you execute a program. + +You can usually use this library via a wrapper script. + + $ cd tools/lkl + $ ./bin/lkl-hijack.sh ip address show + +In order to configure the behavior of LKL, a json file can be used. You can +specify json file with environmental variables (LKL_HIJACK_CONFIG_FILE). If +there is nothing specified, LKL tries to find with the name 'lkl-hijack.json' +for the configuration file. You can also use the old-style configuration with +environmental variables (e.g., LKL_HIJACK_NET_IFTYPE) but those are overridden +if a json file is specified. + +``` + $ cat conf.json + { + "gateway":"192.168.0.1", + "gateway6":"2001:db8:0:f101::1", + "debug":"1", + "singlecpu":"1", + "sysctl":"net.ipv4.tcp_wmem=4096 87380 2147483647", + "boot_cmdline":"ip=dhcp", + "interfaces":[ + { + "mac":"12:34:56:78:9a:bc", + "type":"tap", + "param":"tap7", + "ip":"192.168.0.2", + "masklen":"24", + "ifgateway":"192.168.0.1", + "ipv6":"2001:db8:0:f101::2", + "masklen6":"64", + "ifgateway6":"2001:db8:0:f101::1", + "offload":"0xc803" + }, + { + "mac":"12:34:56:78:9a:bd", + "type":"tap", + "param":"tap77", + "ip":"192.168.1.2", + "masklen":"24", + "ifgateway":"192.168.1.1", + "ipv6":"2001:db8:0:f102::2", + "masklen6":"64", + "ifgateway6":"2001:db8:0:f102::1", + "offload":"0xc803" + } + ] + } + $ LKL_HIJACK_CONFIG_FILE="conf.json" lkl-hijack.sh ip addr s +``` + +The following are the list of keys to describe a JSON file. + +* IPv4 gateway address + + key: "gateway" + value type: string + + the gateway IPv4 address of LKL network stack. +``` + "gateway":"192.168.0.1" +``` + +* IPv6 gateway address + + key: "gateway6" + value type: string + + the gateway IPv6 address of LKL network stack. +``` + "gateway6":"2001:db8:0:f101::1" +``` + +* Debug + + key: "debug" + value type: string + + Setting it causes some debug information (both from the kernel and the + LKL library) to be enabled. If zero' is specified it is disabled. + It is also used as a bit mask to turn on specific debugging facilities. + E.g., setting it to "0x100" will cause the LKL kernel to pause after + the hijack'ed app exits. This allows one to debug or collect info from + the LKL kernel before it quits. +``` + "debug":"1" +``` + +* Single CPU pinning + + key: "singlecpu" + value type: string + + Pin LKL kernel threads on to a single host cpu. value "1" pins + only LKL kernel threads while value "2" also pins polling + threads. +``` + "singlecpu":"1" +``` + +* SYSCTL + + key: "sysctl" + value type: string + + Configure sysctl values of the booted kernel via the hijack library. Multiple + entries can be specified. +``` + "sysctl":"net.ipv4.tcp_wmem=4096 87380 2147483647" +``` + +* Boot command line + + key: "boot_cmdline" + value type: string + + Specify the command line to the kernel boot so that change the configuration + on a kernel instance. For instance, you can change the memory size with + below. +``` + "boot_cmdline": "mem=1G" +``` + +* Mount + + key: "mount" + value type: string + +``` + "mount": "proc,sysfs" +``` + +* Network Interface Configuration + + key: "interfaces" + value type: array of objects + + This key takes a set of sub-keys to configure a single interface. Each key is defined as follows. + ``` + "interfaces":[{....},{....}] + ``` + + + * Interface type + + key: "type" + value type: string + + The interface type in host operating system to connect to LKL. + The following example specifies a tap interface. + ``` + "type":"tap" + ``` + + * Interface parameter + + key: "param" + value type: string + + Additional configuration parameters for the interface specified by Interface type (type). + The parameters depend on the interface type. + ``` + "type":"tap", + "param":"tap0" + ``` + + * Interface MTU size + + key: "mtu" + value type: string + + the MTU size of the interface. + ``` + "mtu":"1280" + ``` + + * Interface IPv4 address + + key: "ip" + value type: string + + the IPv4 address of the interface. + If you want to use DHCP for the IP address assignment, + use "boot_cmdline" with "ip=dhcp" option. + ``` + "ip":"192.168.0.2" + ``` + ``` + "boot_cmdline":"ip=dhcp" + ``` + + * Interface IPv4 netmask length + + key: "masklen" + value type: string + + the network mask length of the interface. + ``` + "ip":"192.168.0.2", + "masklen":"24" + ``` + + * Interface IPv4 gateway on routing policy table + + key: "ifgateway" + value type: string + + If you specify this parameter, LKL adds routing policy table. + And then LKL creates link local and gateway route on this table. + Table SELECTOR is "from" and PREFIX is address you assigned to this interface. + Table id is 2 * (interface index). + This parameter could be used to configure LKL for mptcp, for example. + + ``` + "ip":"192.168.0.2", + "masklen":"24", + "ifgateway":"192.168.0.1" + ``` + + * Interface IPv6 address + + key: "ipv6" + value type: string + + the IPv6 address of the interface. + ``` + "ipv6":"2001:db8:0:f101::2" + ``` + + * Interface IPv6 netmask length + + key: "masklen6" + value type: string + + the network mask length of the interface. + ``` + "ipv6":"2001:db8:0:f101::2", + "masklen":"64" + ``` + + * Interface IPv6 gateway on routing policy table + + key: "ifgateway6" + value type: string + + If you specify this parameter, LKL adds routing policy table. + And then LKL creates link local and gateway route on this table. + Table SELECTOR is "from" and PREFIX is address you assigned to this interface. + Table id is 2 * (interface index) + 1. + This parameter could be used to configure LKL for mptcp, for example. + ``` + "ipv6":"2001:db8:0:f101::2", + "masklen":"64" + "ifgateway6":"2001:db8:0:f101::1", + ``` + + * Interface MAC address + + key: "mac" + value type: string + + the MAC address of the interface. + ``` + "mac":"12:34:56:78:9a:bc" + ``` + + * Interfac neighbor entries + + key: "neigh" + value type: string + + Add a list of permanent neighbor entries in the form of "ip|mac;ip|mac;...". ipv6 are supported + ``` + "neigh":"192.168.0.1|12:34:56:78:9a:bc;2001:db8:0:f101::1|12:34:56:78:9a:be" + ``` + + * Interface qdisc entries + + key: "qdisc" + value type: string + + Add a qdisc entry in the form of "root|type;root|type;...". + ``` + "qdisc":"root|fq" + ``` + + * Interface offload + + key: "offload" + value type: string + + Work as a bit mask to enable selective device offload features. E.g., + to enable "mergeable RX buffer" (LKL_VIRTIO_NET_F_MRG_RXBUF) + + "guest csum" (LKL_VIRTIO_NET_F_GUEST_CSUM) device features, simply set + it to 0x8002. + See virtio_net.h for a list of offload features and their bit masks. + ``` + "offload":"0x8002" + ``` + +* Delay + + key: "delay_main" + value type: string + + The delay before calling main() function of the application after the + initialization of LKL. Some subsystems in Linux tree require a certain + amount of time before accepting a request from application, such as + delivery of address assignment to an network interface. This parameter + is used in such case. The value is described as a microsecond value. +``` + "delay_main":"500000" +``` + +FAQ +=== + +Q: How is LKL different from UML? + +A: UML prodivides a full OS environment (e.g. user/kernel separation, user +processes) and also has requirements (a filesystem, processes, etc.) that makes +it hard to use it for standalone applications. UML also relies heavily on Linux +hosts. On the other hand LKL is designed to be linked directly with the +application and hence does not have user/kernel separation which makes it easier +to use it in standalone applications. + + +Q: How is LKL different from LibOS? + +A: LibOS re-implements high-level kernel APIs for timers, softirqs, scheduling, +sysctl, SLAB/SLUB, etc. LKL behaves like any arch port, implementing the arch +level operations requested by the Linux kernel. LKL also offers a host interface +so that support for multiple hosts can be implemented. diff --git a/README.md b/README.md new file mode 120000 index 000000000000..35c47a921f67 --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +Documentation/lkl.txt \ No newline at end of file -- 2.20.1 (Apple Git-117) _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um