* [RFC PATCH 01/28] Add Zlib License
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
2021-07-01 6:15 ` [RFC PATCH 02/28] cli: Add LIL shell Sean Anderson
` (28 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds the Zlib License which is compatible with GPLv2 as long as its
terms are met. Files originally licensed as Zlib should use the "GPL-2.0+
AND Zlib" SPDX identifier.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
Licenses/README | 1 +
Licenses/zlib.txt | 14 ++++++++++++++
2 files changed, 15 insertions(+)
create mode 100644 Licenses/zlib.txt
diff --git a/Licenses/README b/Licenses/README
index d09f0fe2cb..d5af353520 100644
--- a/Licenses/README
+++ b/Licenses/README
@@ -152,4 +152,5 @@ IBM PIBS (PowerPC Initialization and IBM-pibs ibm-pibs.txt
ISC License ISC Y isc.txt https://spdx.org/licenses/ISC
MIT License MIT Y mit.txt https://spdx.org/licenses/MIT.html
SIL OPEN FONT LICENSE (OFL-1.1) OFL-1.1 Y OFL.txt https://spdx.org/licenses/OFL-1.1.html
+zlib License Zlib Y zlib.txt https://spdx.org/licenses/Zlib.html
X11 License X11 x11.txt https://spdx.org/licenses/X11.html
diff --git a/Licenses/zlib.txt b/Licenses/zlib.txt
new file mode 100644
index 0000000000..7ea8dde871
--- /dev/null
+++ b/Licenses/zlib.txt
@@ -0,0 +1,14 @@
+This software is provided 'as-is', without any express or implied warranty. In
+no event will the authors be held liable for any damages arising from the use
+of this software.
+
+Permission is granted to anyone to use this software for any purpose, including
+commercial applications, and to alter it and redistribute it freely, subject
+to the following restrictions:
+ 1. The origin of this software must not be misrepresented; you must not
+claim that you wrote the original software. If you use this software in a
+product, an acknowledgment in the product documentation would be appreciated
+but is not required.
+ 2. Altered source versions must be plainly marked as such, and must not be
+misrepresented as being the original software.
+ 3. This notice may not be removed or altered from any source distribution.
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 01/28] Add Zlib License
2021-07-01 6:15 ` [RFC PATCH 01/28] Add Zlib License Sean Anderson
@ 2021-07-05 15:29 ` Simon Glass
0 siblings, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>
> This adds the Zlib License which is compatible with GPLv2 as long as its
> terms are met. Files originally licensed as Zlib should use the "GPL-2.0+
> AND Zlib" SPDX identifier.
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
> ---
>
> Licenses/README | 1 +
> Licenses/zlib.txt | 14 ++++++++++++++
> 2 files changed, 15 insertions(+)
> create mode 100644 Licenses/zlib.txt
Reviewed-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 02/28] cli: Add LIL shell
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 01/28] Add Zlib License Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-02 11:03 ` Wolfgang Denk
2021-07-01 6:15 ` [RFC PATCH 03/28] cli: lil: Replace strclone with strdup Sean Anderson
` (27 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This is the LIL programming language [1] as originally written by Kostas
Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
variant. Many syntax features are very similar to shell:
=> echo Hello from $stdout
Hello from serial,vidconsole
LIL supports functions, scoped variables, and command
substitution. For a short example, running the following script
func fact {n} {
if {$n} {
expr {$n * [fact [expr {$n - 1}]]}
} {
return 1
}
}
echo [fact 10]
echo [fact 20]
would result in the output
fact
3628800
2432902008176640000
For more examples, please refer to the "test: Add tests for LIL" later in
this series.
I have made several mostly-mechanical modifications to the source before
importing it to reduce patches to functional changes. If you are interested
in what has been changed before this commit, refer to my lil git
repository [2]. In addition, some functions and features have been removed
from this patch as I have better determined what is necessary for U-Boot
and what is not. If there is sufficient interest, I can add those changes
to the repository as well.
[1] http://runtimeterror.com/tech/lil/
[2] https://github.com/Forty-Bot/lil/commits/master
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
MAINTAINERS | 6 +
cmd/Kconfig | 11 +-
common/Makefile | 1 +
common/cli_lil.c | 2991 +++++++++++++++++++++++++++++++++++++++++++++
include/cli_lil.h | 111 ++
5 files changed, 3119 insertions(+), 1 deletion(-)
create mode 100644 common/cli_lil.c
create mode 100644 include/cli_lil.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 2d267aeff2..0184de5f93 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -766,6 +766,12 @@ S: Maintained
T: git https://source.denx.de/u-boot/custodians/u-boot-i2c.git
F: drivers/i2c/
+LIL SHELL
+M: Sean Anderson <seanga2@gmail.com>
+S: Maintained
+F: common/cli_lil.c
+F: include/cli_lil.h
+
LOGGING
M: Simon Glass <sjg@chromium.org>
S: Maintained
diff --git a/cmd/Kconfig b/cmd/Kconfig
index 9e8b69258f..8bccc572af 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -23,6 +23,15 @@ config HUSH_PARSER
If disabled, you get the old, much simpler behaviour with a somewhat
smaller memory footprint.
+config LIL
+ bool "Use LIL shell"
+ depends on CMDLINE
+ help
+ This options enables the "Little Interpreted Language" (LIL) shell as
+ command line interpreter, thus enabling powerful command line syntax
+ like `proc name {args} {body}' functions or `echo [some command]`
+ command substitution ("tcl scripts").
+
config CMDLINE_EDITING
bool "Enable command line editing"
depends on CMDLINE
@@ -1379,7 +1388,7 @@ config CMD_ECHO
config CMD_ITEST
bool "itest"
- default y
+ default y if !LIL
help
Return true/false on integer compare.
diff --git a/common/Makefile b/common/Makefile
index 829ea5fb42..dce04b305e 100644
--- a/common/Makefile
+++ b/common/Makefile
@@ -10,6 +10,7 @@ obj-y += main.o
obj-y += exports.o
obj-$(CONFIG_HASH) += hash.o
obj-$(CONFIG_HUSH_PARSER) += cli_hush.o
+obj-$(CONFIG_LIL) += cli_lil.o
obj-$(CONFIG_AUTOBOOT) += autoboot.o
# This option is not just y/n - it can have a numeric value
diff --git a/common/cli_lil.c b/common/cli_lil.c
new file mode 100644
index 0000000000..c8c6986fe1
--- /dev/null
+++ b/common/cli_lil.c
@@ -0,0 +1,2991 @@
+// SPDX-License-Identifier: GPL-2.0+ AND Zlib
+/*
+ * LIL - Little Interpreted Language
+ * Copyright (C) 2021 Sean Anderson <seanga2@gmail.com>
+ * Copyright (C) 2010-2013 Kostas Michalopoulos <badsector@runtimelegend.com>
+ *
+ * This file originated from the LIL project, licensed under Zlib. All
+ * modifications are licensed under GPL-2.0+
+ */
+
+#include <common.h>
+#include <cli_lil.h>
+#include <ctype.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+
+/* Enable pools for reusing values, lists and environments. This will use more memory and
+ * will rely on the runtime/OS to free the pools once the program ends, but will cause
+ * considerably less memory fragmentation and improve the script execution performance. */
+/*#define LIL_ENABLE_POOLS*/
+
+/* Enable limiting recursive calls to lil_parse - this can be used to avoid call stack
+ * overflows and is also useful when running through an automated fuzzer like AFL */
+/*#define LIL_ENABLE_RECLIMIT 10000*/
+
+#define ERROR_NOERROR 0
+#define ERROR_DEFAULT 1
+#define ERROR_FIXHEAD 2
+
+#define CALLBACKS 8
+#define HASHMAP_CELLS 256
+#define HASHMAP_CELLMASK 0xFF
+
+struct hashentry {
+ char *k;
+ void *v;
+};
+
+struct hashcell {
+ struct hashentry *e;
+ size_t c;
+};
+
+struct hashmap {
+ struct hashcell cell[HASHMAP_CELLS];
+};
+
+struct lil_value {
+ size_t l;
+#ifdef LIL_ENABLE_POOLS
+ size_t c;
+#endif
+ char *d;
+};
+
+struct lil_var {
+ char *n;
+ struct lil_env *env;
+ struct lil_value *v;
+};
+
+struct lil_env {
+ struct lil_env *parent;
+ struct lil_func *func;
+ struct lil_var **var;
+ size_t vars;
+ struct hashmap varmap;
+ struct lil_value *retval;
+ int retval_set;
+ int breakrun;
+};
+
+struct lil_list {
+ struct lil_value **v;
+ size_t c;
+ size_t cap;
+};
+
+struct lil_func {
+ char *name;
+ struct lil_value *code;
+ struct lil_list *argnames;
+ lil_func_proc_t proc;
+};
+
+struct lil {
+ const char *code; /* need save on parse */
+ const char *rootcode;
+ size_t clen; /* need save on parse */
+ size_t head; /* need save on parse */
+ int ignoreeol;
+ struct lil_func **cmd;
+ size_t cmds;
+ struct hashmap cmdmap;
+ struct lil_env *env;
+ struct lil_env *rootenv;
+ struct lil_env *downenv;
+ struct lil_value *empty;
+ int error;
+ size_t err_head;
+ char *err_msg;
+ lil_callback_proc_t callback[CALLBACKS];
+ size_t parse_depth;
+};
+
+struct expreval {
+ const char *code;
+ size_t len, head;
+ ssize_t ival;
+ int error;
+};
+
+static struct lil_value *next_word(struct lil *lil);
+static void register_stdcmds(struct lil *lil);
+
+#ifdef LIL_ENABLE_POOLS
+static struct lil_value **pool;
+static int poolsize, poolcap;
+static struct lil_list **listpool;
+static size_t listpoolsize, listpoolcap;
+static struct lil_env **envpool;
+static size_t envpoolsize, envpoolcap;
+#endif
+
+static char *strclone(const char *s)
+{
+ size_t len = strlen(s) + 1;
+ char *ns = malloc(len);
+
+ if (!ns)
+ return NULL;
+ memcpy(ns, s, len);
+ return ns;
+}
+
+static unsigned long hm_hash(const char *key)
+{
+ unsigned long hash = 5381;
+ int c;
+
+ while ((c = *key++))
+ hash = ((hash << 5) + hash) + c;
+ return hash;
+}
+
+static void hm_init(struct hashmap *hm)
+{
+ memset(hm, 0, sizeof(struct hashmap));
+}
+
+static void hm_destroy(struct hashmap *hm)
+{
+ size_t i, j;
+
+ for (i = 0; i < HASHMAP_CELLS; i++) {
+ for (j = 0; j < hm->cell[i].c; j++)
+ free(hm->cell[i].e[j].k);
+ free(hm->cell[i].e);
+ }
+}
+
+static void hm_put(struct hashmap *hm, const char *key, void *value)
+{
+ struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+ size_t i;
+
+ for (i = 0; i < cell->c; i++) {
+ if (!strcmp(key, cell->e[i].k)) {
+ cell->e[i].v = value;
+ return;
+ }
+ }
+
+ cell->e = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
+ cell->e[cell->c].k = strclone(key);
+ cell->e[cell->c].v = value;
+ cell->c++;
+}
+
+static void *hm_get(struct hashmap *hm, const char *key)
+{
+ struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+ size_t i;
+
+ for (i = 0; i < cell->c; i++)
+ if (!strcmp(key, cell->e[i].k))
+ return cell->e[i].v;
+ return NULL;
+}
+
+static int hm_has(struct hashmap *hm, const char *key)
+{
+ struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+ size_t i;
+
+ for (i = 0; i < cell->c; i++)
+ if (!strcmp(key, cell->e[i].k))
+ return 1;
+ return 0;
+}
+
+#ifdef LIL_ENABLE_POOLS
+static struct lil_value *alloc_from_pool(void)
+{
+ if (poolsize > 0) {
+ poolsize--;
+ return pool[poolsize];
+ } else {
+ struct lil_value *val = calloc(1, sizeof(struct lil_value));
+
+ return val;
+ }
+}
+
+static void release_to_pool(struct lil_value *val)
+{
+ if (poolsize == poolcap) {
+ poolcap = poolcap ? (poolcap + poolcap / 2) : 64;
+ pool = realloc(pool, sizeof(struct lil_value *) * poolcap);
+ }
+ pool[poolsize++] = val;
+}
+
+static void ensure_capacity(struct lil_value *val, size_t cap)
+{
+ if (val->c < cap) {
+ val->c = cap + 128;
+ val->d = realloc(val->d, val->c);
+ }
+}
+#endif
+
+static struct lil_value *alloc_value_len(const char *str, size_t len)
+{
+#ifdef LIL_ENABLE_POOLS
+ struct lil_value *val = alloc_from_pool();
+#else
+ struct lil_value *val = calloc(1, sizeof(struct lil_value));
+#endif
+
+ if (!val)
+ return NULL;
+ if (str) {
+ val->l = len;
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, len + 1);
+#else
+ val->d = malloc(len + 1);
+ if (!val->d) {
+ free(val);
+ return NULL;
+ }
+#endif
+ memcpy(val->d, str, len);
+ val->d[len] = 0;
+ } else {
+ val->l = 0;
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, 1);
+ val->d[0] = '\0';
+#else
+ val->d = NULL;
+#endif
+ }
+ return val;
+}
+
+static struct lil_value *alloc_value(const char *str)
+{
+ return alloc_value_len(str, str ? strlen(str) : 0);
+}
+
+struct lil_value *lil_clone_value(struct lil_value *src)
+{
+ struct lil_value *val;
+
+ if (!src)
+ return NULL;
+#ifdef LIL_ENABLE_POOLS
+ val = alloc_from_pool();
+#else
+ val = calloc(1, sizeof(struct lil_value));
+#endif
+ if (!val)
+ return NULL;
+
+ val->l = src->l;
+ if (src->l) {
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, val->l + 1);
+#else
+ val->d = malloc(val->l + 1);
+ if (!val->d) {
+ free(val);
+ return NULL;
+ }
+#endif
+ memcpy(val->d, src->d, val->l + 1);
+ } else {
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, 1);
+ val->d[0] = '\0';
+#else
+ val->d = NULL;
+#endif
+ }
+ return val;
+}
+
+int lil_append_char(struct lil_value *val, char ch)
+{
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, val->l + 2);
+ val->d[val->l++] = ch;
+ val->d[val->l] = '\0';
+#else
+ char *new = realloc(val->d, val->l + 2);
+
+ if (!new)
+ return 0;
+
+ new[val->l++] = ch;
+ new[val->l] = 0;
+ val->d = new;
+#endif
+ return 1;
+}
+
+int lil_append_string_len(struct lil_value *val, const char *s, size_t len)
+{
+#ifndef LIL_ENABLE_POOLS
+ char *new;
+#endif
+
+ if (!s || !s[0])
+ return 1;
+
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, val->l + len + 1);
+ memcpy(val->d + val->l, s, len + 1);
+#else
+ new = realloc(val->d, val->l + len + 1);
+ if (!new)
+ return 0;
+
+ memcpy(new + val->l, s, len + 1);
+ val->d = new;
+#endif
+ val->l += len;
+ return 1;
+}
+
+int lil_append_string(struct lil_value *val, const char *s)
+{
+ return lil_append_string_len(val, s, strlen(s));
+}
+
+int lil_append_val(struct lil_value *val, struct lil_value *v)
+{
+#ifndef LIL_ENABLE_POOLS
+ char *new;
+#endif
+
+ if (!v || !v->l)
+ return 1;
+
+#ifdef LIL_ENABLE_POOLS
+ ensure_capacity(val, val->l + v->l + 1);
+ memcpy(val->d + val->l, v->d, v->l + 1);
+#else
+ new = realloc(val->d, val->l + v->l + 1);
+ if (!new)
+ return 0;
+
+ memcpy(new + val->l, v->d, v->l + 1);
+ val->d = new;
+#endif
+ val->l += v->l;
+ return 1;
+}
+
+void lil_free_value(struct lil_value *val)
+{
+ if (!val)
+ return;
+
+#ifdef LIL_ENABLE_POOLS
+ release_to_pool(val);
+#else
+ free(val->d);
+ free(val);
+#endif
+}
+
+struct lil_list *lil_alloc_list(void)
+{
+ struct lil_list *list;
+
+#ifdef LIL_ENABLE_POOLS
+ if (listpoolsize > 0)
+ return listpool[--listpoolsize];
+#endif
+ list = calloc(1, sizeof(struct lil_list));
+ list->v = NULL;
+ return list;
+}
+
+void lil_free_list(struct lil_list *list)
+{
+ size_t i;
+
+ if (!list)
+ return;
+
+ for (i = 0; i < list->c; i++)
+ lil_free_value(list->v[i]);
+
+#ifdef LIL_ENABLE_POOLS
+ list->c = 0;
+ if (listpoolsize == listpoolcap) {
+ listpoolcap =
+ listpoolcap ? (listpoolcap + listpoolcap / 2) : 32;
+ listpool = realloc(listpool,
+ sizeof(struct lil_list *) * listpoolcap);
+ }
+ listpool[listpoolsize++] = list;
+#else
+ free(list->v);
+ free(list);
+#endif
+}
+
+void lil_list_append(struct lil_list *list, struct lil_value *val)
+{
+ if (list->c == list->cap) {
+ size_t cap = list->cap ? (list->cap + list->cap / 2) : 32;
+ struct lil_value **nv =
+ realloc(list->v, sizeof(struct lil_value *) * cap);
+
+ if (!nv)
+ return;
+
+ list->cap = cap;
+ list->v = nv;
+ }
+ list->v[list->c++] = val;
+}
+
+size_t lil_list_size(struct lil_list *list)
+{
+ return list->c;
+}
+
+struct lil_value *lil_list_get(struct lil_list *list, size_t index)
+{
+ return index >= list->c ? NULL : list->v[index];
+}
+
+static int needs_escape(const char *str)
+{
+ size_t i;
+
+ if (!str || !str[0])
+ return 1;
+
+ for (i = 0; str[i]; i++)
+ if (ispunct(str[i]) || isspace(str[i]))
+ return 1;
+
+ return 0;
+}
+
+struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape)
+{
+ struct lil_value *val = alloc_value(NULL);
+ size_t i, j;
+
+ for (i = 0; i < list->c; i++) {
+ int escape =
+ do_escape ? needs_escape(lil_to_string(list->v[i])) : 0;
+
+ if (i)
+ lil_append_char(val, ' ');
+
+ if (escape) {
+ lil_append_char(val, '{');
+ for (j = 0; j < list->v[i]->l; j++) {
+ if (list->v[i]->d[j] == '{')
+ lil_append_string(val, "}\"\\o\"{");
+ else if (list->v[i]->d[j] == '}')
+ lil_append_string(val, "}\"\\c\"{");
+ else
+ lil_append_char(val, list->v[i]->d[j]);
+ }
+ lil_append_char(val, '}');
+ } else {
+ lil_append_val(val, list->v[i]);
+ }
+ }
+ return val;
+}
+
+struct lil_env *lil_alloc_env(struct lil_env *parent)
+{
+ struct lil_env *env;
+
+#ifdef LIL_ENABLE_POOLS
+ if (envpoolsize > 0) {
+ size_t i, j;
+
+ env = envpool[--envpoolsize];
+ env->parent = parent;
+ env->func = NULL;
+ env->var = NULL;
+ env->vars = 0;
+ env->retval = NULL;
+ env->retval_set = 0;
+ env->breakrun = 0;
+ for (i = 0; i < HASHMAP_CELLS; i++) {
+ for (j = 0; j < env->varmap.cell[i].c; j++)
+ free(env->varmap.cell[i].e[j].k);
+ env->varmap.cell[i].c = 0;
+ }
+ return env;
+ }
+#endif
+ env = calloc(1, sizeof(struct lil_env));
+ env->parent = parent;
+ return env;
+}
+
+void lil_free_env(struct lil_env *env)
+{
+ size_t i;
+
+ if (!env)
+ return;
+
+ lil_free_value(env->retval);
+#ifdef LIL_ENABLE_POOLS
+ for (i = 0; i < env->vars; i++) {
+ free(env->var[i]->n);
+ lil_free_value(env->var[i]->v);
+ free(env->var[i]);
+ }
+ free(env->var);
+
+ if (envpoolsize == envpoolcap) {
+ envpoolcap = envpoolcap ? (envpoolcap + envpoolcap / 2) : 64;
+ envpool =
+ realloc(envpool, sizeof(struct lil_env *) * envpoolcap);
+ }
+ envpool[envpoolsize++] = env;
+#else
+ hm_destroy(&env->varmap);
+ for (i = 0; i < env->vars; i++) {
+ free(env->var[i]->n);
+ lil_free_value(env->var[i]->v);
+ free(env->var[i]);
+ }
+ free(env->var);
+ free(env);
+#endif
+}
+
+static struct lil_var *lil_find_local_var(struct lil *lil, struct lil_env *env,
+ const char *name)
+{
+ return hm_get(&env->varmap, name);
+}
+
+static struct lil_var *lil_find_var(struct lil *lil, struct lil_env *env,
+ const char *name)
+{
+ struct lil_var *r = lil_find_local_var(lil, env, name);
+
+ if (r)
+ return r;
+
+ if (env == lil->rootenv)
+ return NULL;
+
+ return lil_find_var(lil, lil->rootenv, name);
+}
+
+static struct lil_func *lil_find_cmd(struct lil *lil, const char *name)
+{
+ return hm_get(&lil->cmdmap, name);
+}
+
+static struct lil_func *add_func(struct lil *lil, const char *name)
+{
+ struct lil_func *cmd;
+ struct lil_func **ncmd;
+
+ cmd = lil_find_cmd(lil, name);
+ if (cmd) {
+ if (cmd->argnames)
+ lil_free_list(cmd->argnames);
+ lil_free_value(cmd->code);
+ cmd->argnames = NULL;
+ cmd->code = NULL;
+ cmd->proc = NULL;
+ return cmd;
+ }
+
+ cmd = calloc(1, sizeof(struct lil_func));
+ cmd->name = strclone(name);
+
+ ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
+ if (!ncmd) {
+ free(cmd);
+ return NULL;
+ }
+
+ lil->cmd = ncmd;
+ ncmd[lil->cmds++] = cmd;
+ hm_put(&lil->cmdmap, name, cmd);
+ return cmd;
+}
+
+static void del_func(struct lil *lil, struct lil_func *cmd)
+{
+ size_t i, index = lil->cmds;
+
+ for (i = 0; i < lil->cmds; i++) {
+ if (lil->cmd[i] == cmd) {
+ index = i;
+ break;
+ }
+ }
+ if (index == lil->cmds)
+ return;
+
+ hm_put(&lil->cmdmap, cmd->name, 0);
+ if (cmd->argnames)
+ lil_free_list(cmd->argnames);
+
+ lil_free_value(cmd->code);
+ free(cmd->name);
+ free(cmd);
+ lil->cmds--;
+ for (i = index; i < lil->cmds; i++)
+ lil->cmd[i] = lil->cmd[i + 1];
+}
+
+int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)
+{
+ struct lil_func *cmd = add_func(lil, name);
+
+ if (!cmd)
+ return 0;
+ cmd->proc = proc;
+ return 1;
+}
+
+struct lil_var *lil_set_var(struct lil *lil, const char *name,
+ struct lil_value *val, int local)
+{
+ struct lil_var **nvar;
+ struct lil_env *env =
+ local == LIL_SETVAR_GLOBAL ? lil->rootenv : lil->env;
+ int freeval = 0;
+
+ if (!name[0])
+ return NULL;
+
+ if (local != LIL_SETVAR_LOCAL_NEW) {
+ struct lil_var *var = lil_find_var(lil, env, name);
+
+ if (local == LIL_SETVAR_LOCAL_ONLY && var &&
+ var->env == lil->rootenv && var->env != env)
+ var = NULL;
+
+ if (((!var && env == lil->rootenv) ||
+ (var && var->env == lil->rootenv)) &&
+ lil->callback[LIL_CALLBACK_SETVAR]) {
+ lil_setvar_callback_proc_t proc =
+ (lil_setvar_callback_proc_t)
+ lil->callback[LIL_CALLBACK_SETVAR];
+ struct lil_value *newval = val;
+ int r = proc(lil, name, &newval);
+
+ if (r < 0) {
+ return NULL;
+ } else if (r > 0) {
+ val = newval;
+ freeval = 1;
+ }
+ }
+
+ if (var) {
+ lil_free_value(var->v);
+ var->v = freeval ? val : lil_clone_value(val);
+ return var;
+ }
+ }
+
+ nvar = realloc(env->var, sizeof(struct lil_var *) * (env->vars + 1));
+ if (!nvar) {
+ /* TODO: report memory error */
+ return NULL;
+ }
+
+ env->var = nvar;
+ nvar[env->vars] = calloc(1, sizeof(struct lil_var));
+ nvar[env->vars]->n = strclone(name);
+ nvar[env->vars]->env = env;
+ nvar[env->vars]->v = freeval ? val : lil_clone_value(val);
+ hm_put(&env->varmap, name, nvar[env->vars]);
+ return nvar[env->vars++];
+}
+
+struct lil_value *lil_get_var(struct lil *lil, const char *name)
+{
+ return lil_get_var_or(lil, name, lil->empty);
+}
+
+struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
+ struct lil_value *defvalue)
+{
+ struct lil_var *var = lil_find_var(lil, lil->env, name);
+ struct lil_value *retval = var ? var->v : defvalue;
+
+ if (lil->callback[LIL_CALLBACK_GETVAR] &&
+ (!var || var->env == lil->rootenv)) {
+ lil_getvar_callback_proc_t proc =
+ (lil_getvar_callback_proc_t)
+ lil->callback[LIL_CALLBACK_GETVAR];
+ struct lil_value *newretval = retval;
+
+ if (proc(lil, name, &newretval))
+ retval = newretval;
+ }
+ return retval;
+}
+
+struct lil_env *lil_push_env(struct lil *lil)
+{
+ struct lil_env *env = lil_alloc_env(lil->env);
+
+ lil->env = env;
+ return env;
+}
+
+void lil_pop_env(struct lil *lil)
+{
+ if (lil->env->parent) {
+ struct lil_env *next = lil->env->parent;
+
+ lil_free_env(lil->env);
+ lil->env = next;
+ }
+}
+
+struct lil *lil_new(void)
+{
+ struct lil *lil = calloc(1, sizeof(struct lil));
+
+ lil->rootenv = lil->env = lil_alloc_env(NULL);
+ lil->empty = alloc_value(NULL);
+ hm_init(&lil->cmdmap);
+ register_stdcmds(lil);
+ return lil;
+}
+
+static int islilspecial(char ch)
+{
+ return ch == '$' || ch == '{' || ch == '}' || ch == '[' || ch == ']' ||
+ ch == '"' || ch == '\'' || ch == ';';
+}
+
+static int eolchar(char ch)
+{
+ return ch == '\n' || ch == '\r' || ch == ';';
+}
+
+static int ateol(struct lil *lil)
+{
+ return !(lil->ignoreeol) && eolchar(lil->code[lil->head]);
+}
+
+static void lil_skip_spaces(struct lil *lil)
+{
+ while (lil->head < lil->clen) {
+ if (lil->code[lil->head] == '#') {
+ if (lil->code[lil->head + 1] == '#' &&
+ lil->code[lil->head + 2] != '#') {
+ lil->head += 2;
+ while (lil->head < lil->clen) {
+ if ((lil->code[lil->head] == '#') &&
+ (lil->code[lil->head + 1] == '#') &&
+ (lil->code[lil->head + 2] != '#')) {
+ lil->head += 2;
+ break;
+ }
+ lil->head++;
+ }
+ } else {
+ while (lil->head < lil->clen &&
+ !eolchar(lil->code[lil->head]))
+ lil->head++;
+ }
+ } else if (lil->code[lil->head] == '\\' &&
+ eolchar(lil->code[lil->head + 1])) {
+ lil->head++;
+ while (lil->head < lil->clen &&
+ eolchar(lil->code[lil->head]))
+ lil->head++;
+ } else if (eolchar(lil->code[lil->head])) {
+ if (lil->ignoreeol)
+ lil->head++;
+ else
+ break;
+ } else if (isspace(lil->code[lil->head])) {
+ lil->head++;
+ } else {
+ break;
+ }
+ }
+}
+
+static struct lil_value *get_bracketpart(struct lil *lil)
+{
+ size_t cnt = 1;
+ struct lil_value *val, *cmd = alloc_value(NULL);
+ int save_eol = lil->ignoreeol;
+
+ lil->ignoreeol = 0;
+ lil->head++;
+ while (lil->head < lil->clen) {
+ if (lil->code[lil->head] == '[') {
+ lil->head++;
+ cnt++;
+ lil_append_char(cmd, '[');
+ } else if (lil->code[lil->head] == ']') {
+ lil->head++;
+ if (--cnt == 0)
+ break;
+ else
+ lil_append_char(cmd, ']');
+ } else {
+ lil_append_char(cmd, lil->code[lil->head++]);
+ }
+ }
+
+ val = lil_parse_value(lil, cmd, 0);
+ lil_free_value(cmd);
+ lil->ignoreeol = save_eol;
+ return val;
+}
+
+static struct lil_value *get_dollarpart(struct lil *lil)
+{
+ struct lil_value *val, *name, *tmp;
+
+ lil->head++;
+ name = next_word(lil);
+ tmp = alloc_value("set ");
+ lil_append_val(tmp, name);
+ lil_free_value(name);
+
+ val = lil_parse_value(lil, tmp, 0);
+ lil_free_value(tmp);
+ return val;
+}
+
+static struct lil_value *next_word(struct lil *lil)
+{
+ struct lil_value *val;
+ size_t start;
+
+ lil_skip_spaces(lil);
+ if (lil->code[lil->head] == '$') {
+ val = get_dollarpart(lil);
+ } else if (lil->code[lil->head] == '{') {
+ size_t cnt = 1;
+
+ lil->head++;
+ val = alloc_value(NULL);
+ while (lil->head < lil->clen) {
+ if (lil->code[lil->head] == '{') {
+ lil->head++;
+ cnt++;
+ lil_append_char(val, '{');
+ } else if (lil->code[lil->head] == '}') {
+ lil->head++;
+ if (--cnt == 0)
+ break;
+ else
+ lil_append_char(val, '}');
+ } else {
+ lil_append_char(val, lil->code[lil->head++]);
+ }
+ }
+ } else if (lil->code[lil->head] == '[') {
+ val = get_bracketpart(lil);
+ } else if (lil->code[lil->head] == '"' ||
+ lil->code[lil->head] == '\'') {
+ char sc = lil->code[lil->head++];
+
+ val = alloc_value(NULL);
+ while (lil->head < lil->clen) {
+ if (lil->code[lil->head] == '[' ||
+ lil->code[lil->head] == '$') {
+ struct lil_value *tmp =
+ lil->code[lil->head] == '$' ?
+ get_dollarpart(lil) :
+ get_bracketpart(lil);
+
+ lil_append_val(val, tmp);
+ lil_free_value(tmp);
+ lil->head--; /* avoid skipping the char below */
+ } else if (lil->code[lil->head] == '\\') {
+ lil->head++;
+ switch (lil->code[lil->head]) {
+ case 'b':
+ lil_append_char(val, '\b');
+ break;
+ case 't':
+ lil_append_char(val, '\t');
+ break;
+ case 'n':
+ lil_append_char(val, '\n');
+ break;
+ case 'v':
+ lil_append_char(val, '\v');
+ break;
+ case 'f':
+ lil_append_char(val, '\f');
+ break;
+ case 'r':
+ lil_append_char(val, '\r');
+ break;
+ case '0':
+ lil_append_char(val, 0);
+ break;
+ case 'a':
+ lil_append_char(val, '\a');
+ break;
+ case 'c':
+ lil_append_char(val, '}');
+ break;
+ case 'o':
+ lil_append_char(val, '{');
+ break;
+ default:
+ lil_append_char(val,
+ lil->code[lil->head]);
+ break;
+ }
+ } else if (lil->code[lil->head] == sc) {
+ lil->head++;
+ break;
+ } else {
+ lil_append_char(val, lil->code[lil->head]);
+ }
+ lil->head++;
+ }
+ } else {
+ start = lil->head;
+ while (lil->head < lil->clen &&
+ !isspace(lil->code[lil->head]) &&
+ !islilspecial(lil->code[lil->head]))
+ lil->head++;
+ val = alloc_value_len(lil->code + start, lil->head - start);
+ }
+ return val ? val : alloc_value(NULL);
+}
+
+static struct lil_list *substitute(struct lil *lil)
+{
+ struct lil_list *words = lil_alloc_list();
+
+ lil_skip_spaces(lil);
+ while (lil->head < lil->clen && !ateol(lil) && !lil->error) {
+ struct lil_value *w = alloc_value(NULL);
+
+ do {
+ size_t head = lil->head;
+ struct lil_value *wp = next_word(lil);
+
+ if (head ==
+ lil->head) { /* something wrong, the parser can't proceed */
+ lil_free_value(w);
+ lil_free_value(wp);
+ lil_free_list(words);
+ return NULL;
+ }
+
+ lil_append_val(w, wp);
+ lil_free_value(wp);
+ } while (lil->head < lil->clen &&
+ !eolchar(lil->code[lil->head]) &&
+ !isspace(lil->code[lil->head]) && !lil->error);
+ lil_skip_spaces(lil);
+
+ lil_list_append(words, w);
+ }
+
+ return words;
+}
+
+struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code)
+{
+ const char *save_code = lil->code;
+ size_t save_clen = lil->clen;
+ size_t save_head = lil->head;
+ int save_igeol = lil->ignoreeol;
+ struct lil_list *words;
+
+ lil->code = lil_to_string(code);
+ lil->clen = code->l;
+ lil->head = 0;
+ lil->ignoreeol = 1;
+
+ words = substitute(lil);
+ if (!words)
+ words = lil_alloc_list();
+
+ lil->code = save_code;
+ lil->clen = save_clen;
+ lil->head = save_head;
+ lil->ignoreeol = save_igeol;
+ return words;
+}
+
+struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code)
+{
+ struct lil_list *words = lil_subst_to_list(lil, code);
+ struct lil_value *val;
+
+ val = lil_list_to_value(words, 0);
+ lil_free_list(words);
+ return val;
+}
+
+static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
+ struct lil_list *words)
+{
+ struct lil_value *r;
+
+ if (cmd->proc) {
+ size_t shead = lil->head;
+
+ r = cmd->proc(lil, words->c - 1, words->v + 1);
+ if (lil->error == ERROR_FIXHEAD) {
+ lil->error = ERROR_DEFAULT;
+ lil->err_head = shead;
+ }
+ } else {
+ lil_push_env(lil);
+ lil->env->func = cmd;
+
+ if (cmd->argnames->c == 1 &&
+ !strcmp(lil_to_string(cmd->argnames->v[0]), "args")) {
+ struct lil_value *args = lil_list_to_value(words, 1);
+
+ lil_set_var(lil, "args", args, LIL_SETVAR_LOCAL_NEW);
+ lil_free_value(args);
+ } else {
+ size_t i;
+
+ for (i = 0; i < cmd->argnames->c; i++) {
+ struct lil_value *val;
+
+ if (i < words->c - 1)
+ val = words->v[i + 1];
+ else
+ val = lil->empty;
+
+ lil_set_var(lil,
+ lil_to_string(cmd->argnames->v[i]),
+ val, LIL_SETVAR_LOCAL_NEW);
+ }
+ }
+ r = lil_parse_value(lil, cmd->code, 1);
+
+ lil_pop_env(lil);
+ }
+
+ return r;
+}
+
+struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
+ int funclevel)
+{
+ const char *save_code = lil->code;
+ size_t save_clen = lil->clen;
+ size_t save_head = lil->head;
+ struct lil_value *val = NULL;
+ struct lil_list *words = NULL;
+
+ if (!save_code)
+ lil->rootcode = code;
+ lil->code = code;
+ lil->clen = codelen ? codelen : strlen(code);
+ lil->head = 0;
+
+ lil_skip_spaces(lil);
+ lil->parse_depth++;
+#ifdef LIL_ENABLE_RECLIMIT
+ if (lil->parse_depth > LIL_ENABLE_RECLIMIT) {
+ lil_set_error(lil, "Too many recursive calls");
+ goto cleanup;
+ }
+#endif
+
+ if (lil->parse_depth == 1)
+ lil->error = 0;
+
+ if (funclevel)
+ lil->env->breakrun = 0;
+
+ while (lil->head < lil->clen && !lil->error) {
+ if (words)
+ lil_free_list(words);
+
+ if (val)
+ lil_free_value(val);
+ val = NULL;
+
+ words = substitute(lil);
+ if (!words || lil->error)
+ goto cleanup;
+
+ if (words->c) {
+ struct lil_func *cmd =
+ lil_find_cmd(lil, lil_to_string(words->v[0]));
+
+ if (!cmd) {
+ if (words->v[0]->l) {
+ char msg[64];
+
+ snprintf(msg, sizeof(msg),
+ "unknown function %s",
+ words->v[0]->d);
+ lil_set_error_at(lil, lil->head, msg);
+ goto cleanup;
+ }
+ } else {
+ val = run_cmd(lil, cmd, words);
+ }
+
+ if (lil->env->breakrun)
+ goto cleanup;
+ }
+
+ lil_skip_spaces(lil);
+ while (ateol(lil))
+ lil->head++;
+ lil_skip_spaces(lil);
+ }
+
+cleanup:
+ if (lil->error && lil->callback[LIL_CALLBACK_ERROR] &&
+ lil->parse_depth == 1) {
+ lil_error_callback_proc_t proc =
+ (lil_error_callback_proc_t)
+ lil->callback[LIL_CALLBACK_ERROR];
+
+ proc(lil, lil->err_head, lil->err_msg);
+ }
+
+ if (words)
+ lil_free_list(words);
+ lil->code = save_code;
+ lil->clen = save_clen;
+ lil->head = save_head;
+
+ if (funclevel && lil->env->retval_set) {
+ if (val)
+ lil_free_value(val);
+ val = lil->env->retval;
+ lil->env->retval = NULL;
+ lil->env->retval_set = 0;
+ lil->env->breakrun = 0;
+ }
+
+ lil->parse_depth--;
+ return val ? val : alloc_value(NULL);
+}
+
+struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
+ int funclevel)
+{
+ if (!val || !val->d || !val->l)
+ return alloc_value(NULL);
+
+ return lil_parse(lil, val->d, val->l, funclevel);
+}
+
+void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc)
+{
+ if (cb < 0 || cb > CALLBACKS)
+ return;
+
+ lil->callback[cb] = proc;
+}
+
+void lil_set_error(struct lil *lil, const char *msg)
+{
+ if (lil->error)
+ return;
+
+ free(lil->err_msg);
+ lil->error = ERROR_FIXHEAD;
+ lil->err_head = 0;
+ lil->err_msg = strclone(msg ? msg : "");
+}
+
+void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
+{
+ if (lil->error)
+ return;
+
+ free(lil->err_msg);
+ lil->error = ERROR_DEFAULT;
+ lil->err_head = pos;
+ lil->err_msg = strclone(msg ? msg : "");
+}
+
+int lil_error(struct lil *lil, const char **msg, size_t *pos)
+{
+ if (!lil->error)
+ return 0;
+
+ *msg = lil->err_msg;
+ *pos = lil->err_head;
+ lil->error = ERROR_NOERROR;
+
+ return 1;
+}
+
+#define EERR_NO_ERROR 0
+#define EERR_SYNTAX_ERROR 1
+#define EERR_DIVISION_BY_ZERO 3
+#define EERR_INVALID_EXPRESSION 4
+
+static void ee_expr(struct expreval *ee);
+
+static int ee_invalidpunct(int ch)
+{
+ return ispunct(ch) && ch != '!' && ch != '~' && ch != '(' &&
+ ch != ')' && ch != '-' && ch != '+';
+}
+
+static void ee_skip_spaces(struct expreval *ee)
+{
+ while (ee->head < ee->len && isspace(ee->code[ee->head]))
+ ee->head++;
+}
+
+static void ee_numeric_element(struct expreval *ee)
+{
+ ee_skip_spaces(ee);
+ ee->ival = 0;
+ while (ee->head < ee->len) {
+ if (!isdigit(ee->code[ee->head]))
+ break;
+
+ ee->ival = ee->ival * 10 + (ee->code[ee->head] - '0');
+ ee->head++;
+ }
+}
+
+static void ee_element(struct expreval *ee)
+{
+ if (isdigit(ee->code[ee->head])) {
+ ee_numeric_element(ee);
+ return;
+ }
+
+ /*
+ * for anything else that might creep in (usually from strings), we set
+ * the value to 1 so that strings evaluate as "true" when used in
+ * conditional expressions
+ */
+ ee->ival = 1;
+ ee->error = EERR_INVALID_EXPRESSION; /* special flag, will be cleared */
+}
+
+static void ee_paren(struct expreval *ee)
+{
+ ee_skip_spaces(ee);
+ if (ee->code[ee->head] == '(') {
+ ee->head++;
+ ee_expr(ee);
+ ee_skip_spaces(ee);
+
+ if (ee->code[ee->head] == ')')
+ ee->head++;
+ else
+ ee->error = EERR_SYNTAX_ERROR;
+ } else {
+ ee_element(ee);
+ }
+}
+
+static void ee_unary(struct expreval *ee)
+{
+ ee_skip_spaces(ee);
+ if (ee->head < ee->len && !ee->error &&
+ (ee->code[ee->head] == '-' || ee->code[ee->head] == '+' ||
+ ee->code[ee->head] == '~' || ee->code[ee->head] == '!')) {
+ char op = ee->code[ee->head++];
+
+ ee_unary(ee);
+ if (ee->error)
+ return;
+
+ switch (op) {
+ case '-':
+ ee->ival = -ee->ival;
+ break;
+ case '+':
+ /* ignore it, doesn't change a thing */
+ break;
+ case '~':
+ ee->ival = ~ee->ival;
+ break;
+ case '!':
+ ee->ival = !ee->ival;
+ break;
+ }
+ } else {
+ ee_paren(ee);
+ }
+}
+
+static void ee_muldiv(struct expreval *ee)
+{
+ ee_unary(ee);
+ if (ee->error)
+ return;
+
+ ee_skip_spaces(ee);
+ while (ee->head < ee->len && !ee->error &&
+ !ee_invalidpunct(ee->code[ee->head + 1]) &&
+ (ee->code[ee->head] == '*' || ee->code[ee->head] == '/' ||
+ ee->code[ee->head] == '\\' || ee->code[ee->head] == '%')) {
+ ssize_t oival = ee->ival;
+
+ switch (ee->code[ee->head]) {
+ case '*':
+ ee->head++;
+ ee_unary(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = ee->ival * oival;
+ break;
+ case '%':
+ ee->head++;
+ ee_unary(ee);
+ if (ee->error)
+ return;
+
+ if (ee->ival == 0)
+ ee->error = EERR_DIVISION_BY_ZERO;
+ else
+ ee->ival = oival % ee->ival;
+ break;
+ case '/':
+ ee->head++;
+ ee_unary(ee);
+ if (ee->error)
+ return;
+
+ if (ee->ival == 0)
+ ee->error = EERR_DIVISION_BY_ZERO;
+ else
+ ee->ival = oival / ee->ival;
+ break;
+ case '\\':
+ ee->head++;
+ ee_unary(ee);
+ if (ee->error)
+ return;
+
+ if (ee->ival == 0)
+ ee->error = EERR_DIVISION_BY_ZERO;
+ else
+ ee->ival = oival / ee->ival;
+ break;
+ }
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_addsub(struct expreval *ee)
+{
+ ee_muldiv(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ !ee_invalidpunct(ee->code[ee->head + 1]) &&
+ (ee->code[ee->head] == '+' || ee->code[ee->head] == '-')) {
+ ssize_t oival = ee->ival;
+
+ switch (ee->code[ee->head]) {
+ case '+':
+ ee->head++;
+ ee_muldiv(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = ee->ival + oival;
+ break;
+ case '-':
+ ee->head++;
+ ee_muldiv(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = oival - ee->ival;
+ break;
+ }
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_shift(struct expreval *ee)
+{
+ ee_addsub(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ ((ee->code[ee->head] == '<' && ee->code[ee->head + 1] == '<') ||
+ (ee->code[ee->head] == '>' && ee->code[ee->head + 1] == '>'))) {
+ ssize_t oival = ee->ival;
+
+ ee->head++;
+ switch (ee->code[ee->head]) {
+ case '<':
+ ee->head++;
+ ee_addsub(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = oival << ee->ival;
+ break;
+ case '>':
+ ee->head++;
+ ee_addsub(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = oival >> ee->ival;
+ break;
+ }
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_compare(struct expreval *ee)
+{
+ ee_shift(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ ((ee->code[ee->head] == '<' &&
+ !ee_invalidpunct(ee->code[ee->head + 1])) ||
+ (ee->code[ee->head] == '>' &&
+ !ee_invalidpunct(ee->code[ee->head + 1])) ||
+ (ee->code[ee->head] == '<' && ee->code[ee->head + 1] == '=') ||
+ (ee->code[ee->head] == '>' && ee->code[ee->head + 1] == '='))) {
+ ssize_t oival = ee->ival;
+ int op = 4;
+
+ if (ee->code[ee->head] == '<' &&
+ !ee_invalidpunct(ee->code[ee->head + 1]))
+ op = 1;
+ else if (ee->code[ee->head] == '>' &&
+ !ee_invalidpunct(ee->code[ee->head + 1]))
+ op = 2;
+ else if (ee->code[ee->head] == '<' &&
+ ee->code[ee->head + 1] == '=')
+ op = 3;
+
+ ee->head += op > 2 ? 2 : 1;
+
+ switch (op) {
+ case 1:
+ ee_shift(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival < ee->ival) ? 1 : 0;
+ break;
+ case 2:
+ ee_shift(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival > ee->ival) ? 1 : 0;
+ break;
+ case 3:
+ ee_shift(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival <= ee->ival) ? 1 : 0;
+ break;
+ case 4:
+ ee_shift(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival >= ee->ival) ? 1 : 0;
+ break;
+ }
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_equals(struct expreval *ee)
+{
+ ee_compare(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ ((ee->code[ee->head] == '=' && ee->code[ee->head + 1] == '=') ||
+ (ee->code[ee->head] == '!' && ee->code[ee->head + 1] == '='))) {
+ ssize_t oival = ee->ival;
+ int op = ee->code[ee->head] == '=' ? 1 : 2;
+
+ ee->head += 2;
+
+ switch (op) {
+ case 1:
+ ee_compare(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival == ee->ival) ? 1 : 0;
+ break;
+ case 2:
+ ee_compare(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival != ee->ival) ? 1 : 0;
+ break;
+ }
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_bitand(struct expreval *ee)
+{
+ ee_equals(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ (ee->code[ee->head] == '&' &&
+ !ee_invalidpunct(ee->code[ee->head + 1]))) {
+ ssize_t oival = ee->ival;
+ ee->head++;
+
+ ee_equals(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = oival & ee->ival;
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_bitor(struct expreval *ee)
+{
+ ee_bitand(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ (ee->code[ee->head] == '|' &&
+ !ee_invalidpunct(ee->code[ee->head + 1]))) {
+ ssize_t oival = ee->ival;
+
+ ee->head++;
+
+ ee_bitand(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = oival | ee->ival;
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_logand(struct expreval *ee)
+{
+ ee_bitor(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ (ee->code[ee->head] == '&' && ee->code[ee->head + 1] == '&')) {
+ ssize_t oival = ee->ival;
+
+ ee->head += 2;
+
+ ee_bitor(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival && ee->ival) ? 1 : 0;
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_logor(struct expreval *ee)
+{
+ ee_logand(ee);
+ ee_skip_spaces(ee);
+
+ while (ee->head < ee->len && !ee->error &&
+ (ee->code[ee->head] == '|' && ee->code[ee->head + 1] == '|')) {
+ ssize_t oival = ee->ival;
+
+ ee->head += 2;
+
+ ee_logand(ee);
+ if (ee->error)
+ return;
+
+ ee->ival = (oival || ee->ival) ? 1 : 0;
+
+ ee_skip_spaces(ee);
+ }
+}
+
+static void ee_expr(struct expreval *ee)
+{
+ ee_logor(ee);
+ if (ee->error == EERR_INVALID_EXPRESSION) {
+ /*
+ * invalid expression doesn't really matter, it is only used to
+ * stop the expression parsing.
+ */
+ ee->error = EERR_NO_ERROR;
+ ee->ival = 1;
+ }
+}
+
+struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
+{
+ struct expreval ee;
+
+ code = lil_subst_to_value(lil, code);
+ if (lil->error)
+ return NULL;
+
+ ee.code = lil_to_string(code);
+ if (!ee.code[0]) {
+ /*
+ * an empty expression equals to 0 so that it can be used as a
+ * false value in conditionals
+ */
+ lil_free_value(code);
+ return lil_alloc_integer(0);
+ }
+
+ ee.head = 0;
+ ee.len = code->l;
+ ee.ival = 0;
+ ee.error = 0;
+
+ ee_expr(&ee);
+ lil_free_value(code);
+ if (ee.error) {
+ switch (ee.error) {
+ case EERR_DIVISION_BY_ZERO:
+ lil_set_error(lil, "division by zero in expression");
+ break;
+ case EERR_SYNTAX_ERROR:
+ lil_set_error(lil, "expression syntax error");
+ break;
+ }
+ return NULL;
+ }
+ return lil_alloc_integer(ee.ival);
+}
+
+struct lil_value *lil_unused_name(struct lil *lil, const char *part)
+{
+ char *name = malloc(strlen(part) + 64);
+ struct lil_value *val;
+ size_t i;
+
+ for (i = 0; i < (size_t)-1; i++) {
+ sprintf(name, "!!un!%s!%09u!nu!!", part, (unsigned int)i);
+ if (lil_find_cmd(lil, name))
+ continue;
+
+ if (lil_find_var(lil, lil->env, name))
+ continue;
+
+ val = lil_alloc_string(name);
+ free(name);
+ return val;
+ }
+ return NULL;
+}
+
+struct lil_value *lil_arg(struct lil_value **argv, size_t index)
+{
+ return argv ? argv[index] : NULL;
+}
+
+const char *lil_to_string(struct lil_value *val)
+{
+ return (val && val->l) ? val->d : "";
+}
+
+ssize_t lil_to_integer(struct lil_value *val)
+{
+ return simple_strtol(lil_to_string(val), NULL, 0);
+}
+
+int lil_to_boolean(struct lil_value *val)
+{
+ const char *s = lil_to_string(val);
+ size_t i, dots = 0;
+
+ if (!s[0])
+ return 0;
+
+ for (i = 0; s[i]; i++) {
+ if (s[i] != '0' && s[i] != '.')
+ return 1;
+
+ if (s[i] == '.') {
+ if (dots)
+ return 1;
+
+ dots = 1;
+ }
+ }
+
+ return 0;
+}
+
+struct lil_value *lil_alloc_string(const char *str)
+{
+ return alloc_value(str);
+}
+
+struct lil_value *lil_alloc_string_len(const char *str, size_t len)
+{
+ return alloc_value_len(str, len);
+}
+
+struct lil_value *lil_alloc_integer(ssize_t num)
+{
+ char buff[128];
+
+ sprintf(buff, "%zd", num);
+ return alloc_value(buff);
+}
+
+void lil_free(struct lil *lil)
+{
+ size_t i;
+
+ if (!lil)
+ return;
+
+ free(lil->err_msg);
+ lil_free_value(lil->empty);
+ while (lil->env) {
+ struct lil_env *next = lil->env->parent;
+
+ lil_free_env(lil->env);
+ lil->env = next;
+ }
+
+ for (i = 0; i < lil->cmds; i++) {
+ if (lil->cmd[i]->argnames)
+ lil_free_list(lil->cmd[i]->argnames);
+
+ lil_free_value(lil->cmd[i]->code);
+ free(lil->cmd[i]->name);
+ free(lil->cmd[i]);
+ }
+
+ hm_destroy(&lil->cmdmap);
+ free(lil->cmd);
+ free(lil);
+}
+
+static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_func *func;
+ const char *type;
+ size_t i;
+ struct lil_value *r;
+
+ if (!argc)
+ return NULL;
+
+ type = lil_to_string(argv[0]);
+ if (!strcmp(type, "version"))
+ return lil_alloc_string(LIL_VERSION_STRING);
+
+ if (!strcmp(type, "args")) {
+ if (argc < 2)
+ return NULL;
+ func = lil_find_cmd(lil, lil_to_string(argv[1]));
+ if (!func || !func->argnames)
+ return NULL;
+ return lil_list_to_value(func->argnames, 1);
+ }
+
+ if (!strcmp(type, "body")) {
+ if (argc < 2)
+ return NULL;
+
+ func = lil_find_cmd(lil, lil_to_string(argv[1]));
+ if (!func || func->proc)
+ return NULL;
+
+ return lil_clone_value(func->code);
+ }
+
+ if (!strcmp(type, "func-count"))
+ return lil_alloc_integer(lil->cmds);
+
+ if (!strcmp(type, "funcs")) {
+ struct lil_list *funcs = lil_alloc_list();
+
+ for (i = 0; i < lil->cmds; i++)
+ lil_list_append(funcs,
+ lil_alloc_string(lil->cmd[i]->name));
+
+ r = lil_list_to_value(funcs, 1);
+ lil_free_list(funcs);
+ return r;
+ }
+
+ if (!strcmp(type, "vars")) {
+ struct lil_list *vars = lil_alloc_list();
+ struct lil_env *env = lil->env;
+
+ while (env) {
+ for (i = 0; i < env->vars; i++) {
+ struct lil_value *var =
+ lil_alloc_string(env->var[i]->n);
+
+ lil_list_append(vars, var);
+ }
+ env = env->parent;
+ }
+
+ r = lil_list_to_value(vars, 1);
+ lil_free_list(vars);
+ return r;
+ }
+
+ if (!strcmp(type, "globals")) {
+ struct lil_list *vars = lil_alloc_list();
+
+ for (i = 0; i < lil->rootenv->vars; i++) {
+ struct lil_value *var =
+ lil_alloc_string(lil->rootenv->var[i]->n);
+
+ lil_list_append(vars, var);
+ }
+
+ r = lil_list_to_value(vars, 1);
+ lil_free_list(vars);
+ return r;
+ }
+
+ if (!strcmp(type, "has-func")) {
+ const char *target;
+
+ if (argc == 1)
+ return NULL;
+
+ target = lil_to_string(argv[1]);
+ return hm_has(&lil->cmdmap, target) ? lil_alloc_string("1") :
+ NULL;
+ }
+
+ if (!strcmp(type, "has-var")) {
+ const char *target;
+ struct lil_env *env = lil->env;
+
+ if (argc == 1)
+ return NULL;
+
+ target = lil_to_string(argv[1]);
+ while (env) {
+ if (hm_has(&env->varmap, target))
+ return lil_alloc_string("1");
+ env = env->parent;
+ }
+ return NULL;
+ }
+
+ if (!strcmp(type, "has-global")) {
+ const char *target;
+
+ if (argc == 1)
+ return NULL;
+
+ target = lil_to_string(argv[1]);
+ for (i = 0; i < lil->rootenv->vars; i++)
+ if (!strcmp(target, lil->rootenv->var[i]->n))
+ return lil_alloc_string("1");
+ return NULL;
+ }
+
+ if (!strcmp(type, "error"))
+ return lil->err_msg ? lil_alloc_string(lil->err_msg) : NULL;
+
+ if (!strcmp(type, "this")) {
+ struct lil_env *env = lil->env;
+
+ while (env != lil->rootenv && !env->func)
+ env = env->parent;
+
+ if (env == lil->rootenv)
+ return lil_alloc_string(lil->rootcode);
+
+ return env->func ? env->func->code : NULL;
+ }
+
+ if (!strcmp(type, "name")) {
+ struct lil_env *env = lil->env;
+
+ while (env != lil->rootenv && !env->func)
+ env = env->parent;
+
+ if (env == lil->rootenv)
+ return NULL;
+
+ return env->func ? lil_alloc_string(env->func->name) : NULL;
+ }
+
+ return NULL;
+}
+
+static struct lil_value *fnc_func(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *name;
+ struct lil_func *cmd;
+ struct lil_list *fargs;
+
+ if (argc < 1)
+ return NULL;
+
+ if (argc >= 3) {
+ name = lil_clone_value(argv[0]);
+ fargs = lil_subst_to_list(lil, argv[1]);
+ cmd = add_func(lil, lil_to_string(argv[0]));
+ cmd->argnames = fargs;
+ cmd->code = lil_clone_value(argv[2]);
+ } else {
+ name = lil_unused_name(lil, "anonymous-function");
+ if (argc < 2) {
+ struct lil_value *tmp = lil_alloc_string("args");
+
+ fargs = lil_subst_to_list(lil, tmp);
+ lil_free_value(tmp);
+ cmd = add_func(lil, lil_to_string(name));
+ cmd->argnames = fargs;
+ cmd->code = lil_clone_value(argv[0]);
+ } else {
+ fargs = lil_subst_to_list(lil, argv[0]);
+ cmd = add_func(lil, lil_to_string(name));
+ cmd->argnames = fargs;
+ cmd->code = lil_clone_value(argv[1]);
+ }
+ }
+
+ return name;
+}
+
+static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *r;
+ struct lil_func *func;
+ const char *oldname;
+ const char *newname;
+
+ if (argc < 2)
+ return NULL;
+
+ oldname = lil_to_string(argv[0]);
+ newname = lil_to_string(argv[1]);
+ func = lil_find_cmd(lil, oldname);
+ if (!func) {
+ char *msg = malloc(24 + strlen(oldname));
+
+ sprintf(msg, "unknown function '%s'", oldname);
+ lil_set_error_at(lil, lil->head, msg);
+ free(msg);
+ return NULL;
+ }
+
+ r = lil_alloc_string(func->name);
+ if (newname[0]) {
+ hm_put(&lil->cmdmap, oldname, 0);
+ hm_put(&lil->cmdmap, newname, func);
+ free(func->name);
+ func->name = strclone(newname);
+ } else {
+ del_func(lil, func);
+ }
+
+ return r;
+}
+
+static struct lil_value *fnc_unusedname(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ return lil_unused_name(lil, argc > 0 ? lil_to_string(argv[0]) :
+ "unusedname");
+}
+
+static struct lil_value *fnc_quote(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *r;
+ size_t i;
+
+ if (argc < 1)
+ return NULL;
+
+ r = alloc_value(NULL);
+ for (i = 0; i < argc; i++) {
+ if (i)
+ lil_append_char(r, ' ');
+ lil_append_val(r, argv[i]);
+ }
+
+ return r;
+}
+
+static struct lil_value *fnc_set(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ size_t i = 0;
+ struct lil_var *var = NULL;
+ int access = LIL_SETVAR_LOCAL;
+
+ if (!argc)
+ return NULL;
+
+ if (!strcmp(lil_to_string(argv[0]), "global")) {
+ i = 1;
+ access = LIL_SETVAR_GLOBAL;
+ }
+
+ while (i < argc) {
+ if (argc == i + 1) {
+ struct lil_value *val =
+ lil_get_var(lil, lil_to_string(argv[i]));
+
+ return lil_clone_value(val);
+ }
+
+ var = lil_set_var(lil, lil_to_string(argv[i]), argv[i + 1],
+ access);
+ i += 2;
+ }
+
+ return var ? lil_clone_value(var->v) : NULL;
+}
+
+static struct lil_value *fnc_local(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ size_t i;
+
+ for (i = 0; i < argc; i++) {
+ const char *varname = lil_to_string(argv[i]);
+
+ if (!lil_find_local_var(lil, lil->env, varname))
+ lil_set_var(lil, varname, lil->empty,
+ LIL_SETVAR_LOCAL_NEW);
+ }
+
+ return NULL;
+}
+
+static struct lil_value *fnc_eval(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc == 1)
+ return lil_parse_value(lil, argv[0], 0);
+
+ if (argc > 1) {
+ struct lil_value *val = alloc_value(NULL), *r;
+ size_t i;
+
+ for (i = 0; i < argc; i++) {
+ if (i)
+ lil_append_char(val, ' ');
+ lil_append_val(val, argv[i]);
+ }
+
+ r = lil_parse_value(lil, val, 0);
+ lil_free_value(val);
+ return r;
+ }
+
+ return NULL;
+}
+
+static struct lil_value *fnc_topeval(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_env *thisenv = lil->env;
+ struct lil_env *thisdownenv = lil->downenv;
+ struct lil_value *r;
+
+ lil->env = lil->rootenv;
+ lil->downenv = thisenv;
+
+ r = fnc_eval(lil, argc, argv);
+ lil->downenv = thisdownenv;
+ lil->env = thisenv;
+ return r;
+}
+
+static struct lil_value *fnc_upeval(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_env *thisenv = lil->env;
+ struct lil_env *thisdownenv = lil->downenv;
+ struct lil_value *r;
+
+ if (lil->rootenv == thisenv)
+ return fnc_eval(lil, argc, argv);
+
+ lil->env = thisenv->parent;
+ lil->downenv = thisenv;
+
+ r = fnc_eval(lil, argc, argv);
+ lil->env = thisenv;
+ lil->downenv = thisdownenv;
+ return r;
+}
+
+static struct lil_value *fnc_downeval(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *r;
+ struct lil_env *upenv = lil->env;
+ struct lil_env *downenv = lil->downenv;
+
+ if (!downenv)
+ return fnc_eval(lil, argc, argv);
+
+ lil->downenv = NULL;
+ lil->env = downenv;
+
+ r = fnc_eval(lil, argc, argv);
+ lil->downenv = downenv;
+ lil->env = upenv;
+ return r;
+}
+
+static struct lil_value *fnc_count(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ char buff[64];
+
+ if (!argc)
+ return alloc_value("0");
+
+ list = lil_subst_to_list(lil, argv[0]);
+ sprintf(buff, "%u", (unsigned int)list->c);
+ lil_free_list(list);
+ return alloc_value(buff);
+}
+
+static struct lil_value *fnc_index(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ size_t index;
+ struct lil_value *r;
+
+ if (argc < 2)
+ return NULL;
+
+ list = lil_subst_to_list(lil, argv[0]);
+ index = (size_t)lil_to_integer(argv[1]);
+ if (index >= list->c)
+ r = NULL;
+ else
+ r = lil_clone_value(list->v[index]);
+ lil_free_list(list);
+ return r;
+}
+
+static struct lil_value *fnc_indexof(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ size_t index;
+ struct lil_value *r = NULL;
+
+ if (argc < 2)
+ return NULL;
+
+ list = lil_subst_to_list(lil, argv[0]);
+ for (index = 0; index < list->c; index++) {
+ if (!strcmp(lil_to_string(list->v[index]),
+ lil_to_string(argv[1]))) {
+ r = lil_alloc_integer(index);
+ break;
+ }
+ }
+
+ lil_free_list(list);
+ return r;
+}
+
+static struct lil_value *fnc_append(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ struct lil_value *r;
+ size_t i, base = 1;
+ int access = LIL_SETVAR_LOCAL;
+ const char *varname;
+
+ if (argc < 2)
+ return NULL;
+
+ varname = lil_to_string(argv[0]);
+ if (!strcmp(varname, "global")) {
+ if (argc < 3)
+ return NULL;
+
+ varname = lil_to_string(argv[1]);
+ base = 2;
+ access = LIL_SETVAR_GLOBAL;
+ }
+
+ list = lil_subst_to_list(lil, lil_get_var(lil, varname));
+ for (i = base; i < argc; i++)
+ lil_list_append(list, lil_clone_value(argv[i]));
+
+ r = lil_list_to_value(list, 1);
+ lil_free_list(list);
+ lil_set_var(lil, varname, r, access);
+ return r;
+}
+
+static struct lil_value *fnc_slice(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list, *slice;
+ size_t i;
+ ssize_t from, to;
+ struct lil_value *r;
+
+ if (argc < 1)
+ return NULL;
+ if (argc < 2)
+ return lil_clone_value(argv[0]);
+
+ from = lil_to_integer(argv[1]);
+ if (from < 0)
+ from = 0;
+
+ list = lil_subst_to_list(lil, argv[0]);
+ to = argc > 2 ? lil_to_integer(argv[2]) : (ssize_t)list->c;
+ if (to > (ssize_t)list->c)
+ to = list->c;
+ else if (to < from)
+ to = from;
+
+ slice = lil_alloc_list();
+ for (i = (size_t)from; i < (size_t)to; i++)
+ lil_list_append(slice, lil_clone_value(list->v[i]));
+ lil_free_list(list);
+
+ r = lil_list_to_value(slice, 1);
+ lil_free_list(slice);
+ return r;
+}
+
+static struct lil_value *fnc_filter(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list, *filtered;
+ size_t i;
+ struct lil_value *r;
+ const char *varname = "x";
+ int base = 0;
+
+ if (argc < 1)
+ return NULL;
+
+ if (argc < 2)
+ return lil_clone_value(argv[0]);
+
+ if (argc > 2) {
+ base = 1;
+ varname = lil_to_string(argv[0]);
+ }
+
+ list = lil_subst_to_list(lil, argv[base]);
+ filtered = lil_alloc_list();
+ for (i = 0; i < list->c && !lil->env->breakrun; i++) {
+ lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+ r = lil_eval_expr(lil, argv[base + 1]);
+ if (lil_to_boolean(r))
+ lil_list_append(filtered, lil_clone_value(list->v[i]));
+ lil_free_value(r);
+ }
+ lil_free_list(list);
+
+ r = lil_list_to_value(filtered, 1);
+ lil_free_list(filtered);
+ return r;
+}
+
+static struct lil_value *fnc_list(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list = lil_alloc_list();
+ struct lil_value *r;
+ size_t i;
+
+ for (i = 0; i < argc; i++)
+ lil_list_append(list, lil_clone_value(argv[i]));
+
+ r = lil_list_to_value(list, 1);
+ lil_free_list(list);
+ return r;
+}
+
+static struct lil_value *fnc_subst(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc < 1)
+ return NULL;
+
+ return lil_subst_to_value(lil, argv[0]);
+}
+
+static struct lil_value *fnc_concat(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ struct lil_value *r, *tmp;
+ size_t i;
+
+ if (argc < 1)
+ return NULL;
+
+ r = lil_alloc_string("");
+ for (i = 0; i < argc; i++) {
+ list = lil_subst_to_list(lil, argv[i]);
+ tmp = lil_list_to_value(list, 1);
+ lil_free_list(list);
+ lil_append_val(r, tmp);
+ lil_free_value(tmp);
+ }
+ return r;
+}
+
+static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list, *rlist;
+ struct lil_value *r;
+ size_t i, listidx = 0, codeidx = 1;
+ const char *varname = "i";
+
+ if (argc < 2)
+ return NULL;
+
+ if (argc >= 3) {
+ varname = lil_to_string(argv[0]);
+ listidx = 1;
+ codeidx = 2;
+ }
+
+ rlist = lil_alloc_list();
+ list = lil_subst_to_list(lil, argv[listidx]);
+ for (i = 0; i < list->c; i++) {
+ struct lil_value *rv;
+
+ lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+ rv = lil_parse_value(lil, argv[codeidx], 0);
+ if (rv->l)
+ lil_list_append(rlist, rv);
+ else
+ lil_free_value(rv);
+
+ if (lil->env->breakrun || lil->error)
+ break;
+ }
+
+ r = lil_list_to_value(rlist, 1);
+ lil_free_list(list);
+ lil_free_list(rlist);
+ return r;
+}
+
+static struct lil_value *fnc_return(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ lil->env->breakrun = 1;
+ lil_free_value(lil->env->retval);
+ lil->env->retval = argc < 1 ? NULL : lil_clone_value(argv[0]);
+ lil->env->retval_set = 1;
+ return argc < 1 ? NULL : lil_clone_value(argv[0]);
+}
+
+static struct lil_value *fnc_result(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc > 0) {
+ lil_free_value(lil->env->retval);
+ lil->env->retval = lil_clone_value(argv[0]);
+ lil->env->retval_set = 1;
+ }
+
+ return lil->env->retval_set ? lil_clone_value(lil->env->retval) : NULL;
+}
+
+static struct lil_value *fnc_expr(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc == 1)
+ return lil_eval_expr(lil, argv[0]);
+
+ if (argc > 1) {
+ struct lil_value *val = alloc_value(NULL), *r;
+ size_t i;
+
+ for (i = 0; i < argc; i++) {
+ if (i)
+ lil_append_char(val, ' ');
+ lil_append_val(val, argv[i]);
+ }
+
+ r = lil_eval_expr(lil, val);
+ lil_free_value(val);
+ return r;
+ }
+
+ return NULL;
+}
+
+static struct lil_value *real_inc(struct lil *lil, const char *varname,
+ ssize_t v)
+{
+ struct lil_value *pv = lil_get_var(lil, varname);
+
+ pv = lil_alloc_integer(lil_to_integer(pv) + v);
+ lil_set_var(lil, varname, pv, LIL_SETVAR_LOCAL);
+ return pv;
+}
+
+static struct lil_value *fnc_inc(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc < 1)
+ return NULL;
+
+ return real_inc(lil, lil_to_string(argv[0]),
+ argc > 1 ? lil_to_integer(argv[1]) : 1);
+}
+
+static struct lil_value *fnc_dec(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc < 1)
+ return NULL;
+
+ return real_inc(lil, lil_to_string(argv[0]),
+ -(argc > 1 ? lil_to_integer(argv[1]) : 1));
+}
+
+static struct lil_value *fnc_if(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *val, *r = NULL;
+ int base = 0, not = 0, v;
+
+ if (argc < 1)
+ return NULL;
+
+ if (!strcmp(lil_to_string(argv[0]), "not"))
+ base = not = 1;
+
+ if (argc < (size_t)base + 2)
+ return NULL;
+
+ val = lil_eval_expr(lil, argv[base]);
+ if (!val || lil->error)
+ return NULL;
+
+ v = lil_to_boolean(val);
+ if (not)
+ v = !v;
+
+ if (v)
+ r = lil_parse_value(lil, argv[base + 1], 0);
+ else if (argc > (size_t)base + 2)
+ r = lil_parse_value(lil, argv[base + 2], 0);
+
+ lil_free_value(val);
+ return r;
+}
+
+static struct lil_value *fnc_while(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *val, *r = NULL;
+ int base = 0, not = 0, v;
+
+ if (argc < 1)
+ return NULL;
+
+ if (!strcmp(lil_to_string(argv[0]), "not"))
+ base = not = 1;
+
+ if (argc < (size_t)base + 2)
+ return NULL;
+
+ while (!lil->error && !lil->env->breakrun) {
+ val = lil_eval_expr(lil, argv[base]);
+ if (!val || lil->error)
+ return NULL;
+
+ v = lil_to_boolean(val);
+ if (not)
+ v = !v;
+
+ if (!v) {
+ lil_free_value(val);
+ break;
+ }
+
+ if (r)
+ lil_free_value(r);
+ r = lil_parse_value(lil, argv[base + 1], 0);
+ lil_free_value(val);
+ }
+
+ return r;
+}
+
+static struct lil_value *fnc_for(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *val, *r = NULL;
+
+ if (argc < 4)
+ return NULL;
+
+ lil_free_value(lil_parse_value(lil, argv[0], 0));
+ while (!lil->error && !lil->env->breakrun) {
+ val = lil_eval_expr(lil, argv[1]);
+ if (!val || lil->error)
+ return NULL;
+
+ if (!lil_to_boolean(val)) {
+ lil_free_value(val);
+ break;
+ }
+
+ if (r)
+ lil_free_value(r);
+ r = lil_parse_value(lil, argv[3], 0);
+ lil_free_value(val);
+ lil_free_value(lil_parse_value(lil, argv[2], 0));
+ }
+
+ return r;
+}
+
+static struct lil_value *fnc_char(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ char s[2];
+
+ if (!argc)
+ return NULL;
+
+ s[0] = (char)lil_to_integer(argv[0]);
+ s[1] = 0;
+ return lil_alloc_string(s);
+}
+
+static struct lil_value *fnc_charat(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ size_t index;
+ char chstr[2];
+ const char *str;
+
+ if (argc < 2)
+ return NULL;
+
+ str = lil_to_string(argv[0]);
+ index = (size_t)lil_to_integer(argv[1]);
+ if (index >= strlen(str))
+ return NULL;
+
+ chstr[0] = str[index];
+ chstr[1] = 0;
+ return lil_alloc_string(chstr);
+}
+
+static struct lil_value *fnc_codeat(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ size_t index;
+ const char *str;
+
+ if (argc < 2)
+ return NULL;
+
+ str = lil_to_string(argv[0]);
+ index = (size_t)lil_to_integer(argv[1]);
+ if (index >= strlen(str))
+ return NULL;
+
+ return lil_alloc_integer(str[index]);
+}
+
+static struct lil_value *fnc_substr(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ const char *str;
+ struct lil_value *r;
+ size_t start, end, i, slen;
+
+ if (argc < 2)
+ return NULL;
+
+ str = lil_to_string(argv[0]);
+ if (!str[0])
+ return NULL;
+
+ slen = strlen(str);
+ start = simple_strtol(lil_to_string(argv[1]), NULL, 0);
+ end = argc > 2 ? simple_strtol(lil_to_string(argv[2]), NULL, 0) : slen;
+ if (end > slen)
+ end = slen;
+
+ if (start >= end)
+ return NULL;
+
+ r = lil_alloc_string("");
+ for (i = start; i < end; i++)
+ lil_append_char(r, str[i]);
+ return r;
+}
+
+static struct lil_value *fnc_strpos(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ const char *hay;
+ const char *str;
+ size_t min = 0;
+
+ if (argc < 2)
+ return lil_alloc_integer(-1);
+
+ hay = lil_to_string(argv[0]);
+ if (argc > 2) {
+ min = simple_strtol(lil_to_string(argv[2]), NULL, 0);
+ if (min >= strlen(hay))
+ return lil_alloc_integer(-1);
+ }
+
+ str = strstr(hay + min, lil_to_string(argv[1]));
+ if (!str)
+ return lil_alloc_integer(-1);
+
+ return lil_alloc_integer(str - hay);
+}
+
+static struct lil_value *fnc_length(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ size_t i, total = 0;
+
+ for (i = 0; i < argc; i++) {
+ if (i)
+ total++;
+ total += strlen(lil_to_string(argv[i]));
+ }
+
+ return lil_alloc_integer((ssize_t)total);
+}
+
+static struct lil_value *real_trim(const char *str, const char *chars, int left,
+ int right)
+{
+ int base = 0;
+ struct lil_value *r = NULL;
+
+ if (left) {
+ while (str[base] && strchr(chars, str[base]))
+ base++;
+ if (!right)
+ r = lil_alloc_string(str[base] ? str + base : NULL);
+ }
+
+ if (right) {
+ size_t len;
+ char *s;
+
+ s = strclone(str + base);
+ len = strlen(s);
+ while (len && strchr(chars, s[len - 1]))
+ len--;
+
+ s[len] = 0;
+ r = lil_alloc_string(s);
+ free(s);
+ }
+
+ return r;
+}
+
+static struct lil_value *fnc_trim(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ const char *chars;
+
+ if (!argc)
+ return NULL;
+
+ if (argc < 2)
+ chars = " \f\n\r\t\v";
+ else
+ chars = lil_to_string(argv[1]);
+
+ return real_trim(lil_to_string(argv[0]), chars, 1, 1);
+}
+
+static struct lil_value *fnc_ltrim(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ const char *chars;
+
+ if (!argc)
+ return NULL;
+
+ if (argc < 2)
+ chars = " \f\n\r\t\v";
+ else
+ chars = lil_to_string(argv[1]);
+
+ return real_trim(lil_to_string(argv[0]), chars, 1, 0);
+}
+
+static struct lil_value *fnc_rtrim(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ const char *chars;
+
+ if (!argc)
+ return NULL;
+
+ if (argc < 2)
+ chars = " \f\n\r\t\v";
+ else
+ chars = lil_to_string(argv[1]);
+
+ return real_trim(lil_to_string(argv[0]), chars, 0, 1);
+}
+
+static struct lil_value *fnc_strcmp(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc < 2)
+ return NULL;
+
+ return lil_alloc_integer(strcmp(lil_to_string(argv[0]),
+ lil_to_string(argv[1])));
+}
+
+static struct lil_value *fnc_streq(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ if (argc < 2)
+ return NULL;
+
+ return lil_alloc_integer(strcmp(lil_to_string(argv[0]),
+ lil_to_string(argv[1])) ? 0 : 1);
+}
+
+static struct lil_value *fnc_repstr(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ const char *from;
+ const char *to;
+ char *src;
+ const char *sub;
+ size_t idx;
+ size_t fromlen;
+ size_t tolen;
+ size_t srclen;
+ struct lil_value *r;
+
+ if (argc < 1)
+ return NULL;
+
+ if (argc < 3)
+ return lil_clone_value(argv[0]);
+
+ from = lil_to_string(argv[1]);
+ to = lil_to_string(argv[2]);
+ if (!from[0])
+ return NULL;
+
+ src = strclone(lil_to_string(argv[0]));
+ srclen = strlen(src);
+ fromlen = strlen(from);
+ tolen = strlen(to);
+ while ((sub = strstr(src, from))) {
+ char *newsrc = malloc(srclen - fromlen + tolen + 1);
+
+ idx = sub - src;
+ if (idx)
+ memcpy(newsrc, src, idx);
+
+ memcpy(newsrc + idx, to, tolen);
+ memcpy(newsrc + idx + tolen, src + idx + fromlen,
+ srclen - idx - fromlen);
+ srclen = srclen - fromlen + tolen;
+ free(src);
+ src = newsrc;
+ src[srclen] = 0;
+ }
+
+ r = lil_alloc_string(src);
+ free(src);
+ return r;
+}
+
+static struct lil_value *fnc_split(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ const char *sep = " ";
+ size_t i;
+ struct lil_value *val;
+ const char *str;
+
+ if (argc == 0)
+ return NULL;
+ if (argc > 1) {
+ sep = lil_to_string(argv[1]);
+ if (!sep || !sep[0])
+ return lil_clone_value(argv[0]);
+ }
+
+ val = lil_alloc_string("");
+ str = lil_to_string(argv[0]);
+ list = lil_alloc_list();
+ for (i = 0; str[i]; i++) {
+ if (strchr(sep, str[i])) {
+ lil_list_append(list, val);
+ val = lil_alloc_string("");
+ } else {
+ lil_append_char(val, str[i]);
+ }
+ }
+
+ lil_list_append(list, val);
+ val = lil_list_to_value(list, 1);
+ lil_free_list(list);
+ return val;
+}
+
+static struct lil_value *fnc_try(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_value *r;
+
+ if (argc < 1)
+ return NULL;
+
+ if (lil->error)
+ return NULL;
+
+ r = lil_parse_value(lil, argv[0], 0);
+ if (lil->error) {
+ lil->error = ERROR_NOERROR;
+ lil_free_value(r);
+
+ if (argc > 1)
+ r = lil_parse_value(lil, argv[1], 0);
+ else
+ r = 0;
+ }
+ return r;
+}
+
+static struct lil_value *fnc_error(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ lil_set_error(lil, argc > 0 ? lil_to_string(argv[0]) : NULL);
+ return NULL;
+}
+
+static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
+ struct lil_value **argv)
+{
+ struct lil_list *list;
+ size_t i;
+
+ if (argc < 2)
+ return NULL;
+
+ list = lil_subst_to_list(lil, argv[0]);
+ for (i = 1; i < argc; i++)
+ lil_set_var(lil, lil_to_string(argv[i]),
+ lil_list_get(list, i - 1), LIL_SETVAR_LOCAL);
+ lil_free_list(list);
+ return NULL;
+}
+
+static void register_stdcmds(struct lil *lil)
+{
+ lil_register(lil, "append", fnc_append);
+ lil_register(lil, "char", fnc_char);
+ lil_register(lil, "charat", fnc_charat);
+ lil_register(lil, "codeat", fnc_codeat);
+ lil_register(lil, "concat", fnc_concat);
+ lil_register(lil, "count", fnc_count);
+ lil_register(lil, "dec", fnc_dec);
+ lil_register(lil, "downeval", fnc_downeval);
+ lil_register(lil, "error", fnc_error);
+ lil_register(lil, "eval", fnc_eval);
+ lil_register(lil, "expr", fnc_expr);
+ lil_register(lil, "filter", fnc_filter);
+ lil_register(lil, "for", fnc_for);
+ lil_register(lil, "foreach", fnc_foreach);
+ lil_register(lil, "func", fnc_func);
+ lil_register(lil, "if", fnc_if);
+ lil_register(lil, "inc", fnc_inc);
+ lil_register(lil, "index", fnc_index);
+ lil_register(lil, "indexof", fnc_indexof);
+ lil_register(lil, "length", fnc_length);
+ lil_register(lil, "list", fnc_list);
+ lil_register(lil, "lmap", fnc_lmap);
+ lil_register(lil, "local", fnc_local);
+ lil_register(lil, "ltrim", fnc_ltrim);
+ lil_register(lil, "quote", fnc_quote);
+ lil_register(lil, "reflect", fnc_reflect);
+ lil_register(lil, "rename", fnc_rename);
+ lil_register(lil, "repstr", fnc_repstr);
+ lil_register(lil, "result", fnc_result);
+ lil_register(lil, "return", fnc_return);
+ lil_register(lil, "rtrim", fnc_rtrim);
+ lil_register(lil, "set", fnc_set);
+ lil_register(lil, "slice", fnc_slice);
+ lil_register(lil, "split", fnc_split);
+ lil_register(lil, "strcmp", fnc_strcmp);
+ lil_register(lil, "streq", fnc_streq);
+ lil_register(lil, "strpos", fnc_strpos);
+ lil_register(lil, "subst", fnc_subst);
+ lil_register(lil, "substr", fnc_substr);
+ lil_register(lil, "topeval", fnc_topeval);
+ lil_register(lil, "trim", fnc_trim);
+ lil_register(lil, "try", fnc_try);
+ lil_register(lil, "unusedname", fnc_unusedname);
+ lil_register(lil, "upeval", fnc_upeval);
+ lil_register(lil, "while", fnc_while);
+}
diff --git a/include/cli_lil.h b/include/cli_lil.h
new file mode 100644
index 0000000000..c72977ea5c
--- /dev/null
+++ b/include/cli_lil.h
@@ -0,0 +1,111 @@
+/* SPDX-License-Identifier: GPL-2.0+ AND Zlib */
+/*
+ * LIL - Little Interpreted Language
+ * Copyright (C) 2021 Sean Anderson <seanga2@gmail.com>
+ * Copyright (C) 2010-2013 Kostas Michalopoulos <badsector@runtimelegend.com>
+ *
+ * This file originated from the LIL project, licensed under Zlib. All
+ * modifications are licensed under GPL-2.0+
+ */
+
+#ifndef __LIL_H_INCLUDED__
+#define __LIL_H_INCLUDED__
+
+#define LIL_VERSION_STRING "0.1"
+
+#define LIL_SETVAR_GLOBAL 0
+#define LIL_SETVAR_LOCAL 1
+#define LIL_SETVAR_LOCAL_NEW 2
+#define LIL_SETVAR_LOCAL_ONLY 3
+
+#define LIL_CALLBACK_EXIT 0
+#define LIL_CALLBACK_WRITE 1
+#define LIL_CALLBACK_READ 2
+#define LIL_CALLBACK_STORE 3
+#define LIL_CALLBACK_SOURCE 4
+#define LIL_CALLBACK_ERROR 5
+#define LIL_CALLBACK_SETVAR 6
+#define LIL_CALLBACK_GETVAR 7
+
+#include <stdint.h>
+#include <inttypes.h>
+
+struct lil_value;
+struct lil_func;
+struct lil_var;
+struct lil_env;
+struct lil_list;
+struct lil;
+typedef struct lil_value *(*lil_func_proc_t)(struct lil *lil, size_t argc,
+ struct lil_value **argv);
+typedef void (*lil_exit_callback_proc_t)(struct lil *lil,
+ struct lil_value *arg);
+typedef void (*lil_write_callback_proc_t)(struct lil *lil, const char *msg);
+typedef char *(*lil_read_callback_proc_t)(struct lil *lil, const char *name);
+typedef char *(*lil_source_callback_proc_t)(struct lil *lil, const char *name);
+typedef void (*lil_store_callback_proc_t)(struct lil *lil, const char *name,
+ const char *data);
+typedef void (*lil_error_callback_proc_t)(struct lil *lil, size_t pos,
+ const char *msg);
+typedef int (*lil_setvar_callback_proc_t)(struct lil *lil, const char *name,
+ struct lil_value **value);
+typedef int (*lil_getvar_callback_proc_t)(struct lil *lil, const char *name,
+ struct lil_value **value);
+typedef void (*lil_callback_proc_t)(void);
+
+struct lil *lil_new(void);
+void lil_free(struct lil *lil);
+
+int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc);
+
+struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
+ int funclevel);
+struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
+ int funclevel);
+
+void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc);
+
+void lil_set_error(struct lil *lil, const char *msg);
+void lil_set_error_at(struct lil *lil, size_t pos, const char *msg);
+int lil_error(struct lil *lil, const char **msg, size_t *pos);
+
+const char *lil_to_string(struct lil_value *val);
+ssize_t lil_to_integer(struct lil_value *val);
+int lil_to_boolean(struct lil_value *val);
+
+struct lil_value *lil_alloc_string(const char *str);
+struct lil_value *lil_alloc_integer(ssize_t num);
+void lil_free_value(struct lil_value *val);
+
+struct lil_value *lil_clone_value(struct lil_value *src);
+int lil_append_char(struct lil_value *val, char ch);
+int lil_append_string(struct lil_value *val, const char *s);
+int lil_append_val(struct lil_value *val, struct lil_value *v);
+
+struct lil_list *lil_alloc_list(void);
+void lil_free_list(struct lil_list *list);
+void lil_list_append(struct lil_list *list, struct lil_value *val);
+size_t lil_list_size(struct lil_list *list);
+struct lil_value *lil_list_get(struct lil_list *list, size_t index);
+struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape);
+
+struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code);
+struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code);
+
+struct lil_env *lil_alloc_env(struct lil_env *parent);
+void lil_free_env(struct lil_env *env);
+struct lil_env *lil_push_env(struct lil *lil);
+void lil_pop_env(struct lil *lil);
+
+struct lil_var *lil_set_var(struct lil *lil, const char *name,
+ struct lil_value *val, int local);
+struct lil_value *lil_get_var(struct lil *lil, const char *name);
+struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
+ struct lil_value *defvalue);
+
+struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code);
+struct lil_value *lil_unused_name(struct lil *lil, const char *part);
+
+struct lil_value *lil_arg(struct lil_value **argv, size_t index);
+
+#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-01 6:15 ` [RFC PATCH 02/28] cli: Add LIL shell Sean Anderson
@ 2021-07-02 11:03 ` Wolfgang Denk
2021-07-02 13:33 ` Sean Anderson
0 siblings, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-02 11:03 UTC (permalink / raw)
To: Sean Anderson
Cc: u-boot, Tom Rini, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <20210701061611.957918-3-seanga2@gmail.com> you wrote:
> This is the LIL programming language [1] as originally written by Kostas
> Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
> variant. Many syntax features are very similar to shell:
Do you have a list of the exact differencec between LIL and a
standard shell?
I wonder, if we deviate from standard shell syntax anyway, we could
also have a look at lua, for example?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
One essential to success is that you desire be an all-obsessing one,
your thoughts and aims be co-ordinated, and your energy be concentra-
ted and applied without letup.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-02 11:03 ` Wolfgang Denk
@ 2021-07-02 13:33 ` Sean Anderson
2021-07-03 2:12 ` Sean Anderson
2021-07-03 19:23 ` Wolfgang Denk
0 siblings, 2 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-02 13:33 UTC (permalink / raw)
To: Wolfgang Denk
Cc: u-boot, Tom Rini, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/2/21 7:03 AM, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <20210701061611.957918-3-seanga2@gmail.com> you wrote:
>> This is the LIL programming language [1] as originally written by Kostas
>> Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
>> variant. Many syntax features are very similar to shell:
>
> Do you have a list of the exact differencec between LIL and a
> standard shell?
For a partial list, see
[1] https://github.com/Forty-Bot/lil/commits/master
> I wonder, if we deviate from standard shell syntax anyway, we could
> also have a look at lua, for example?
I also looked at lua (see the cover letter), but I rejected it based on
size constraints (eLua was around the size of U-Boot itself).
Because of how often the shell is used to debug things, I wanted the
candidate I picked to have similar syntax to the existing shell. For
example,
load mmc ${mmcdev}:${mmcpart} $loadaddr $image
is a valid command in both Hush and LIL. Compare with lua, which might
express the above as
load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)
which I think is a much larger deviation from existing syntax.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-02 13:33 ` Sean Anderson
@ 2021-07-03 2:12 ` Sean Anderson
2021-07-03 19:33 ` Wolfgang Denk
2021-07-03 19:23 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-03 2:12 UTC (permalink / raw)
To: Wolfgang Denk
Cc: u-boot, Tom Rini, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/2/21 9:33 AM, Sean Anderson wrote:
> On 7/2/21 7:03 AM, Wolfgang Denk wrote:
>> Dear Sean,
>>
>> In message <20210701061611.957918-3-seanga2@gmail.com> you wrote:
>>> This is the LIL programming language [1] as originally written by Kostas
>>> Michalopoulos <badsector@runtimeterror.com>. LIL is a stripped-down TCL
>>> variant. Many syntax features are very similar to shell:
>>
>> Do you have a list of the exact differencec between LIL and a
>> standard shell?
>
> For a partial list, see
>
> [1] https://github.com/Forty-Bot/lil/commits/master
Whoops, looks like I completely misread what you were asking here. I
don't have an exhaustive list of differences, but here are some similar
things expressed in both languages:
sh tcl
foo=bar set foo bar
echo $foo echo $foo
if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }
foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }
for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }
fact() {
if [ $1 -eq 0 ]; then
echo 1
else
echo $(($1 * $(fact $(($1 - 1)))))
fi
}
proc fact {n} {
if {$n} {
expr {$n * [fact [expr {$n - 1}]]}
} {
return 1
}
}
Hopefully this gives you a bit of a feel for the basic differences.
--Sean
>
>> I wonder, if we deviate from standard shell syntax anyway, we could
>> also have a look at lua, for example?
>
> I also looked at lua (see the cover letter), but I rejected it based on
> size constraints (eLua was around the size of U-Boot itself).
>
> Because of how often the shell is used to debug things, I wanted the
> candidate I picked to have similar syntax to the existing shell. For
> example,
>
> load mmc ${mmcdev}:${mmcpart} $loadaddr $image
>
> is a valid command in both Hush and LIL. Compare with lua, which might
> express the above as
>
> load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)
>
> which I think is a much larger deviation from existing syntax.
>
> --Sean
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-03 2:12 ` Sean Anderson
@ 2021-07-03 19:33 ` Wolfgang Denk
2021-07-05 15:29 ` Simon Glass
2021-07-05 19:10 ` Tom Rini
0 siblings, 2 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-03 19:33 UTC (permalink / raw)
To: Sean Anderson
Cc: u-boot, Tom Rini, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
>
> > For a partial list, see
> >
> > [1] https://github.com/Forty-Bot/lil/commits/master
>
> Whoops, looks like I completely misread what you were asking here. I
> don't have an exhaustive list of differences, but here are some similar
> things expressed in both languages:
>
> sh tcl
>
> foo=bar set foo bar
> echo $foo echo $foo
>
> if [ 1 -gt 2 ]; then if {1 > 2} {
> echo a echo a
> else } {
> echo b echo b
> fi }
>
> foo() { proc foo {first second} {
> echo $1 $2 echo $first $second
> } }
>
> for file in $(ls *.c); do foreach file [glob *.c] {
> echo $file echo $file
> done }
>
> fact() {
> if [ $1 -eq 0 ]; then
> echo 1
> else
> echo $(($1 * $(fact $(($1 - 1)))))
> fi
> }
>
> proc fact {n} {
> if {$n} {
> expr {$n * [fact [expr {$n - 1}]]}
> } {
> return 1
> }
> }
>
> Hopefully this gives you a bit of a feel for the basic differences.
Well, I know TCL, and there has been a zillion of reasons to move
_from_ that language many, many years ago. Intoducing this as new
"shell" language in U-Boot does not look attractive to me.
Actually, it gives me the creeps.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Drawing on my fine command of language, I said nothing.
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-03 19:33 ` Wolfgang Denk
@ 2021-07-05 15:29 ` Simon Glass
2021-07-05 19:10 ` Tom Rini
1 sibling, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, U-Boot Mailing List, Tom Rini, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Hi,
On Sat, 3 Jul 2021 at 13:33, Wolfgang Denk <wd@denx.de> wrote:
>
> Dear Sean,
>
> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
> >
> > > For a partial list, see
> > >
> > > [1] https://github.com/Forty-Bot/lil/commits/master
> >
> > Whoops, looks like I completely misread what you were asking here. I
> > don't have an exhaustive list of differences, but here are some similar
> > things expressed in both languages:
> >
> > sh tcl
> >
> > foo=bar set foo bar
> > echo $foo echo $foo
> >
> > if [ 1 -gt 2 ]; then if {1 > 2} {
> > echo a echo a
> > else } {
> > echo b echo b
> > fi }
> >
> > foo() { proc foo {first second} {
> > echo $1 $2 echo $first $second
> > } }
> >
> > for file in $(ls *.c); do foreach file [glob *.c] {
> > echo $file echo $file
> > done }
> >
> > fact() {
> > if [ $1 -eq 0 ]; then
> > echo 1
> > else
> > echo $(($1 * $(fact $(($1 - 1)))))
> > fi
> > }
> >
> > proc fact {n} {
> > if {$n} {
> > expr {$n * [fact [expr {$n - 1}]]}
> > } {
> > return 1
> > }
> > }
> >
> > Hopefully this gives you a bit of a feel for the basic differences.
>
> Well, I know TCL, and there has been a zillion of reasons to move
> _from_ that language many, many years ago. Intoducing this as new
> "shell" language in U-Boot does not look attractive to me.
> Actually, it gives me the creeps.
I was no fan of TCL either but this does seem like a reasonable idea to me.
Obviously whoever wrote it was very concerned about compiler
performance as there seem to be no comments?
Reviewed-by: Simon Glass <sjg@chromium.org>
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-03 19:33 ` Wolfgang Denk
2021-07-05 15:29 ` Simon Glass
@ 2021-07-05 19:10 ` Tom Rini
2021-07-05 19:47 ` Sean Anderson
2021-07-06 7:44 ` Wolfgang Denk
1 sibling, 2 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-05 19:10 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 2265 bytes --]
On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
> >
> > > For a partial list, see
> > >
> > > [1] https://github.com/Forty-Bot/lil/commits/master
> >
> > Whoops, looks like I completely misread what you were asking here. I
> > don't have an exhaustive list of differences, but here are some similar
> > things expressed in both languages:
> >
> > sh tcl
> >
> > foo=bar set foo bar
> > echo $foo echo $foo
> >
> > if [ 1 -gt 2 ]; then if {1 > 2} {
> > echo a echo a
> > else } {
> > echo b echo b
> > fi }
> >
> > foo() { proc foo {first second} {
> > echo $1 $2 echo $first $second
> > } }
> >
> > for file in $(ls *.c); do foreach file [glob *.c] {
> > echo $file echo $file
> > done }
> >
> > fact() {
> > if [ $1 -eq 0 ]; then
> > echo 1
> > else
> > echo $(($1 * $(fact $(($1 - 1)))))
> > fi
> > }
> >
> > proc fact {n} {
> > if {$n} {
> > expr {$n * [fact [expr {$n - 1}]]}
> > } {
> > return 1
> > }
> > }
> >
> > Hopefully this gives you a bit of a feel for the basic differences.
Which of these things, from each column, can you do in the context of
U-Boot? That's important too.
> Well, I know TCL, and there has been a zillion of reasons to move
> _from_ that language many, many years ago. Intoducing this as new
> "shell" language in U-Boot does not look attractive to me.
> Actually, it gives me the creeps.
This is I think the hard question. A draw of the current shell is that
it it looks and acts like bash/sh/etc, for at least basic operations.
That's something that's comfortable to a large audience. That has
disadvantages when people want to start doing something complex. Sean
has shown that several times and he's not the only one. LIL being
tcl'ish is not.
Something that has "sh" syntax but also clear to the user errors when
trying to do something not supported would also be interesting to see.
It seems like a lot of the frustration from users with our shell is that
it's not clear where the line between "this is an sh-like shell" and
"no, not like that" is.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-05 19:10 ` Tom Rini
@ 2021-07-05 19:47 ` Sean Anderson
2021-07-05 19:53 ` Tom Rini
2021-07-06 7:46 ` Wolfgang Denk
2021-07-06 7:44 ` Wolfgang Denk
1 sibling, 2 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-05 19:47 UTC (permalink / raw)
To: Tom Rini, Wolfgang Denk
Cc: u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/5/21 3:10 PM, Tom Rini wrote:
> On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
>> Dear Sean,
>>
>> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
>>>
>>>> For a partial list, see
>>>>
>>>> [1] https://github.com/Forty-Bot/lil/commits/master
>>>
>>> Whoops, looks like I completely misread what you were asking here. I
>>> don't have an exhaustive list of differences, but here are some similar
>>> things expressed in both languages:
>>>
>>> sh tcl
>>>
>>> foo=bar set foo bar
>>> echo $foo echo $foo
>>>
>>> if [ 1 -gt 2 ]; then if {1 > 2} {
>>> echo a echo a
>>> else } {
>>> echo b echo b
>>> fi }
The left side is possible with something like
if itest 1 -gt 2; then # etc.
>>>
>>> foo() { proc foo {first second} {
>>> echo $1 $2 echo $first $second
>>> } }
This is not possible. We only have eval (run) as of today. I view adding
functions as one of the most important usability improvements we can
make.
>>>
>>> for file in $(ls *.c); do foreach file [glob *.c] {
>>> echo $file echo $file
>>> done }
This is possible only if you already have a list of files. For example,
one could do
part list mmc 0 -bootable parts
for p in $parts; do #etc
but the part command is one of the only ones which produces output in
the correct format. If you want to (e.g.) dynamically construct a list
you will have a much harder time.
>>> fact() {
>>> if [ $1 -eq 0 ]; then
>>> echo 1
>>> else
>>> echo $(($1 * $(fact $(($1 - 1)))))
>>> fi
>>> }
This is technically possible with run and setexpr, but fairly cumbersome
to do.
>>>
>>> proc fact {n} {
>>> if {$n} {
>>> expr {$n * [fact [expr {$n - 1}]]}
>>> } {
>>> return 1
>>> }
>>> }
>>>
>>> Hopefully this gives you a bit of a feel for the basic differences.
>
> Which of these things, from each column, can you do in the context of
> U-Boot? That's important too.
See above.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-05 19:47 ` Sean Anderson
@ 2021-07-05 19:53 ` Tom Rini
2021-07-05 19:55 ` Sean Anderson
2021-07-06 7:46 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Tom Rini @ 2021-07-05 19:53 UTC (permalink / raw)
To: Sean Anderson
Cc: Wolfgang Denk, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 2518 bytes --]
On Mon, Jul 05, 2021 at 03:47:47PM -0400, Sean Anderson wrote:
> On 7/5/21 3:10 PM, Tom Rini wrote:
> > On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
> > > Dear Sean,
> > >
> > > In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
> > > >
> > > > > For a partial list, see
> > > > >
> > > > > [1] https://github.com/Forty-Bot/lil/commits/master
> > > >
> > > > Whoops, looks like I completely misread what you were asking here. I
> > > > don't have an exhaustive list of differences, but here are some similar
> > > > things expressed in both languages:
> > > >
> > > > sh tcl
> > > >
> > > > foo=bar set foo bar
> > > > echo $foo echo $foo
> > > >
> > > > if [ 1 -gt 2 ]; then if {1 > 2} {
> > > > echo a echo a
> > > > else } {
> > > > echo b echo b
> > > > fi }
>
> The left side is possible with something like
>
> if itest 1 -gt 2; then # etc.
>
> > > >
> > > > foo() { proc foo {first second} {
> > > > echo $1 $2 echo $first $second
> > > > } }
>
> This is not possible. We only have eval (run) as of today. I view adding
> functions as one of the most important usability improvements we can
> make.
>
> > > >
> > > > for file in $(ls *.c); do foreach file [glob *.c] {
> > > > echo $file echo $file
> > > > done }
>
> This is possible only if you already have a list of files. For example,
> one could do
>
> part list mmc 0 -bootable parts
> for p in $parts; do #etc
>
> but the part command is one of the only ones which produces output in
> the correct format. If you want to (e.g.) dynamically construct a list
> you will have a much harder time.
>
> > > > fact() {
> > > > if [ $1 -eq 0 ]; then
> > > > echo 1
> > > > else
> > > > echo $(($1 * $(fact $(($1 - 1)))))
> > > > fi
> > > > }
>
> This is technically possible with run and setexpr, but fairly cumbersome
> to do.
>
> > > >
> > > > proc fact {n} {
> > > > if {$n} {
> > > > expr {$n * [fact [expr {$n - 1}]]}
> > > > } {
> > > > return 1
> > > > }
> > > > }
> > > >
> > > > Hopefully this gives you a bit of a feel for the basic differences.
> >
> > Which of these things, from each column, can you do in the context of
> > U-Boot? That's important too.
>
> See above.
And for clarity, on the LIL side, with a few things like needing to
bring in the list of files somehow, all of those would work in U-Boot?
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-05 19:53 ` Tom Rini
@ 2021-07-05 19:55 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-05 19:55 UTC (permalink / raw)
To: Tom Rini
Cc: Wolfgang Denk, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On 7/5/21 3:53 PM, Tom Rini wrote:
> On Mon, Jul 05, 2021 at 03:47:47PM -0400, Sean Anderson wrote:
>> On 7/5/21 3:10 PM, Tom Rini wrote:
>>> On Sat, Jul 03, 2021 at 09:33:30PM +0200, Wolfgang Denk wrote:
>>>> Dear Sean,
>>>>
>>>> In message <8bbdb7a1-5085-a3b7-614f-12ae9aee8e8b@gmail.com> you wrote:
>>>>>
>>>>>> For a partial list, see
>>>>>>
>>>>>> [1] https://github.com/Forty-Bot/lil/commits/master
>>>>>
>>>>> Whoops, looks like I completely misread what you were asking here. I
>>>>> don't have an exhaustive list of differences, but here are some similar
>>>>> things expressed in both languages:
>>>>>
>>>>> sh tcl
>>>>>
>>>>> foo=bar set foo bar
>>>>> echo $foo echo $foo
>>>>>
>>>>> if [ 1 -gt 2 ]; then if {1 > 2} {
>>>>> echo a echo a
>>>>> else } {
>>>>> echo b echo b
>>>>> fi }
>>
>> The left side is possible with something like
>>
>> if itest 1 -gt 2; then # etc.
>>
>>>>>
>>>>> foo() { proc foo {first second} {
>>>>> echo $1 $2 echo $first $second
>>>>> } }
>>
>> This is not possible. We only have eval (run) as of today. I view adding
>> functions as one of the most important usability improvements we can
>> make.
>>
>>>>>
>>>>> for file in $(ls *.c); do foreach file [glob *.c] {
>>>>> echo $file echo $file
>>>>> done }
>>
>> This is possible only if you already have a list of files. For example,
>> one could do
>>
>> part list mmc 0 -bootable parts
>> for p in $parts; do #etc
>>
>> but the part command is one of the only ones which produces output in
>> the correct format. If you want to (e.g.) dynamically construct a list
>> you will have a much harder time.
>>
>>>>> fact() {
>>>>> if [ $1 -eq 0 ]; then
>>>>> echo 1
>>>>> else
>>>>> echo $(($1 * $(fact $(($1 - 1)))))
>>>>> fi
>>>>> }
>>
>> This is technically possible with run and setexpr, but fairly cumbersome
>> to do.
>>
>>>>>
>>>>> proc fact {n} {
>>>>> if {$n} {
>>>>> expr {$n * [fact [expr {$n - 1}]]}
>>>>> } {
>>>>> return 1
>>>>> }
>>>>> }
>>>>>
>>>>> Hopefully this gives you a bit of a feel for the basic differences.
>>>
>>> Which of these things, from each column, can you do in the context of
>>> U-Boot? That's important too.
>>
>> See above.
>
> And for clarity, on the LIL side, with a few things like needing to
> bring in the list of files somehow, all of those would work in U-Boot?
>
Correct. The only unimplemented function is "glob".
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-05 19:47 ` Sean Anderson
2021-07-05 19:53 ` Tom Rini
@ 2021-07-06 7:46 ` Wolfgang Denk
2021-07-06 7:52 ` Michael Nazzareno Trimarchi
2021-07-06 14:54 ` Tom Rini
1 sibling, 2 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-06 7:46 UTC (permalink / raw)
To: Sean Anderson
Cc: Tom Rini, u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
>
> >>> foo() { proc foo {first second} {
> >>> echo $1 $2 echo $first $second
> >>> } }
>
> This is not possible. We only have eval (run) as of today. I view adding
> functions as one of the most important usability improvements we can
> make.
Again: this is not an issue with hush as is, but only with our
resource-limited port of a nearly 20 year old version of it.
Updating to a current version would fix this, in an almost 100%
backward compatible way.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
If you don't have time to do it right, when will you have time to do
it over? - John Wooden
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 7:46 ` Wolfgang Denk
@ 2021-07-06 7:52 ` Michael Nazzareno Trimarchi
2021-07-06 14:57 ` Simon Glass
2021-07-06 14:54 ` Tom Rini
1 sibling, 1 reply; 107+ messages in thread
From: Michael Nazzareno Trimarchi @ 2021-07-06 7:52 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, Tom Rini, U-Boot-Denx, Marek Behún,
Simon Glass, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
>
> Dear Sean,
>
> In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> >
> > >>> foo() { proc foo {first second} {
> > >>> echo $1 $2 echo $first $second
> > >>> } }
> >
> > This is not possible. We only have eval (run) as of today. I view adding
> > functions as one of the most important usability improvements we can
> > make.
>
> Again: this is not an issue with hush as is, but only with our
> resource-limited port of a nearly 20 year old version of it.
>
> Updating to a current version would fix this, in an almost 100%
> backward compatible way.
Agree on this. There is another email about the LIL and how is right now
maintained and if the solutions that we have already in place can be just
updated, I don't find any valid reason to change it but update
seems more straightforward.
Michael
>
> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> If you don't have time to do it right, when will you have time to do
> it over? - John Wooden
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________
Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 7:52 ` Michael Nazzareno Trimarchi
@ 2021-07-06 14:57 ` Simon Glass
2021-07-06 15:48 ` Tom Rini
2021-07-07 8:22 ` Michael Nazzareno Trimarchi
0 siblings, 2 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-06 14:57 UTC (permalink / raw)
To: Michael Nazzareno Trimarchi
Cc: Wolfgang Denk, Sean Anderson, Tom Rini, U-Boot-Denx,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Hi Michael,
On Tue, 6 Jul 2021 at 01:53, Michael Nazzareno Trimarchi
<michael@amarulasolutions.com> wrote:
>
> On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
> >
> > Dear Sean,
> >
> > In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> > >
> > > >>> foo() { proc foo {first second} {
> > > >>> echo $1 $2 echo $first $second
> > > >>> } }
> > >
> > > This is not possible. We only have eval (run) as of today. I view adding
> > > functions as one of the most important usability improvements we can
> > > make.
> >
> > Again: this is not an issue with hush as is, but only with our
> > resource-limited port of a nearly 20 year old version of it.
> >
> > Updating to a current version would fix this, in an almost 100%
> > backward compatible way.
>
> Agree on this. There is another email about the LIL and how is right now
> maintained and if the solutions that we have already in place can be just
> updated, I don't find any valid reason to change it but update
> seems more straightforward.
Would you be interested in taking this on? It has been talked about
for some years...
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 14:57 ` Simon Glass
@ 2021-07-06 15:48 ` Tom Rini
2021-07-07 8:22 ` Michael Nazzareno Trimarchi
1 sibling, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-06 15:48 UTC (permalink / raw)
To: Simon Glass
Cc: Michael Nazzareno Trimarchi, Wolfgang Denk, Sean Anderson,
U-Boot-Denx, Marek Behún, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]
On Tue, Jul 06, 2021 at 08:57:23AM -0600, Simon Glass wrote:
> Hi Michael,
>
> On Tue, 6 Jul 2021 at 01:53, Michael Nazzareno Trimarchi
> <michael@amarulasolutions.com> wrote:
> >
> > On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
> > >
> > > Dear Sean,
> > >
> > > In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> > > >
> > > > >>> foo() { proc foo {first second} {
> > > > >>> echo $1 $2 echo $first $second
> > > > >>> } }
> > > >
> > > > This is not possible. We only have eval (run) as of today. I view adding
> > > > functions as one of the most important usability improvements we can
> > > > make.
> > >
> > > Again: this is not an issue with hush as is, but only with our
> > > resource-limited port of a nearly 20 year old version of it.
> > >
> > > Updating to a current version would fix this, in an almost 100%
> > > backward compatible way.
> >
> > Agree on this. There is another email about the LIL and how is right now
> > maintained and if the solutions that we have already in place can be just
> > updated, I don't find any valid reason to change it but update
> > seems more straightforward.
>
> Would you be interested in taking this on? It has been talked about
> for some years...
To repeat myself, someone posting the patches to update (or more likely,
replace as a new Kconfig option and I'd be fine with a "pick old or new"
as a starting point) is the main gating factor on using a new version of
hush.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 14:57 ` Simon Glass
2021-07-06 15:48 ` Tom Rini
@ 2021-07-07 8:22 ` Michael Nazzareno Trimarchi
1 sibling, 0 replies; 107+ messages in thread
From: Michael Nazzareno Trimarchi @ 2021-07-07 8:22 UTC (permalink / raw)
To: Simon Glass, Francis Laniel
Cc: Wolfgang Denk, Sean Anderson, Tom Rini, U-Boot-Denx,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Hi
On Tue, Jul 6, 2021 at 4:57 PM Simon Glass <sjg@chromium.org> wrote:
>
> Hi Michael,
>
> On Tue, 6 Jul 2021 at 01:53, Michael Nazzareno Trimarchi
> <michael@amarulasolutions.com> wrote:
> >
> > On Tue, Jul 6, 2021 at 9:46 AM Wolfgang Denk <wd@denx.de> wrote:
> > >
> > > Dear Sean,
> > >
> > > In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> > > >
> > > > >>> foo() { proc foo {first second} {
> > > > >>> echo $1 $2 echo $first $second
> > > > >>> } }
> > > >
> > > > This is not possible. We only have eval (run) as of today. I view adding
> > > > functions as one of the most important usability improvements we can
> > > > make.
> > >
> > > Again: this is not an issue with hush as is, but only with our
> > > resource-limited port of a nearly 20 year old version of it.
> > >
> > > Updating to a current version would fix this, in an almost 100%
> > > backward compatible way.
> >
> > Agree on this. There is another email about the LIL and how is right now
> > maintained and if the solutions that we have already in place can be just
> > updated, I don't find any valid reason to change it but update
> > seems more straightforward.
>
> Would you be interested in taking this on? It has been talked about
> for some years...
>
> Regards,
> Simon
I will ask to someone in the company to do as task. Francis can you do it?
Michael
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________
Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 7:46 ` Wolfgang Denk
2021-07-06 7:52 ` Michael Nazzareno Trimarchi
@ 2021-07-06 14:54 ` Tom Rini
2021-07-07 8:15 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Tom Rini @ 2021-07-06 14:54 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 874 bytes --]
On Tue, Jul 06, 2021 at 09:46:43AM +0200, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <e7e23702-4e55-f785-c77f-2d3076fdd67b@gmail.com> you wrote:
> >
> > >>> foo() { proc foo {first second} {
> > >>> echo $1 $2 echo $first $second
> > >>> } }
> >
> > This is not possible. We only have eval (run) as of today. I view adding
> > functions as one of the most important usability improvements we can
> > make.
>
> Again: this is not an issue with hush as is, but only with our
> resource-limited port of a nearly 20 year old version of it.
>
> Updating to a current version would fix this, in an almost 100%
> backward compatible way.
Let us cut to the chase then. Who is going to port a modern version of
hush over to U-Boot, and maintain it? If we fork and forget again,
we'll be in a bad place once again in 2-3 years.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 14:54 ` Tom Rini
@ 2021-07-07 8:15 ` Wolfgang Denk
2021-07-07 13:58 ` Tom Rini
0 siblings, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-07 8:15 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Tom,
In message <20210706145420.GQ9516@bill-the-cat> you wrote:
>
> > Updating to a current version would fix this, in an almost 100%
> > backward compatible way.
>
> Let us cut to the chase then. Who is going to port a modern version of
> hush over to U-Boot, and maintain it? If we fork and forget again,
> we'll be in a bad place once again in 2-3 years.
Would we really be better off if we switch to some exotic piece of
code instead (I was not able to locate any user base, nor other
developers), which has been reported to have poor or no error
handling, and comes with an incompatible command line interface?
There is a zillion of shell scripts in the field, from non-trivial
boot sequences to complex download-and-upgrade scripts. You can't
really even think of breaking compatibility on such a level.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
There are no data that cannot be plotted on a straight line if the
axis are chosen correctly.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 8:15 ` Wolfgang Denk
@ 2021-07-07 13:58 ` Tom Rini
2021-07-07 14:10 ` Wolfgang Denk
0 siblings, 1 reply; 107+ messages in thread
From: Tom Rini @ 2021-07-07 13:58 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]
On Wed, Jul 07, 2021 at 10:15:32AM +0200, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210706145420.GQ9516@bill-the-cat> you wrote:
> >
> > > Updating to a current version would fix this, in an almost 100%
> > > backward compatible way.
> >
> > Let us cut to the chase then. Who is going to port a modern version of
> > hush over to U-Boot, and maintain it? If we fork and forget again,
> > we'll be in a bad place once again in 2-3 years.
>
> Would we really be better off if we switch to some exotic piece of
> code instead (I was not able to locate any user base, nor other
> developers), which has been reported to have poor or no error
> handling, and comes with an incompatible command line interface?
>
> There is a zillion of shell scripts in the field, from non-trivial
> boot sequences to complex download-and-upgrade scripts. You can't
> really even think of breaking compatibility on such a level.
As I've said a few times in this thread, this not being an sh-style
interpreter is a strike against it. And if we're going to insist on a
bug-for-bug upgrade to our hush (so that all of the hugely complex
existing scripts keep working) we might as well not upgrade. Frankly I
suspect that down the line IF a new cli interpreter comes in to U-Boot
we will have to keep the old one around as a "use this instead" option
for another long number of years, so that if there are any systems with
non-trivial scripts but upgrade U-Boot and don't / won't / can't
re-validate their entire sequence, they can just use the old cli.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 13:58 ` Tom Rini
@ 2021-07-07 14:10 ` Wolfgang Denk
2021-07-07 14:14 ` Tom Rini
0 siblings, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-07 14:10 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Tom,
In message <20210707135839.GW9516@bill-the-cat> you wrote:
>
> As I've said a few times in this thread, this not being an sh-style
> interpreter is a strike against it. And if we're going to insist on a
> bug-for-bug upgrade to our hush (so that all of the hugely complex
> existing scripts keep working) we might as well not upgrade. Frankly I
> suspect that down the line IF a new cli interpreter comes in to U-Boot
> we will have to keep the old one around as a "use this instead" option
> for another long number of years, so that if there are any systems with
> non-trivial scripts but upgrade U-Boot and don't / won't / can't
> re-validate their entire sequence, they can just use the old cli.
Do you actually have an example where code working on our ancient
port of hush would fail on the current upstream version?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
If you can't explain it to a six year old, you don't understand it
yourself. - Albert Einstein
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 14:10 ` Wolfgang Denk
@ 2021-07-07 14:14 ` Tom Rini
2021-07-07 14:23 ` Wolfgang Denk
0 siblings, 1 reply; 107+ messages in thread
From: Tom Rini @ 2021-07-07 14:14 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]
On Wed, Jul 07, 2021 at 04:10:43PM +0200, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210707135839.GW9516@bill-the-cat> you wrote:
> >
> > As I've said a few times in this thread, this not being an sh-style
> > interpreter is a strike against it. And if we're going to insist on a
> > bug-for-bug upgrade to our hush (so that all of the hugely complex
> > existing scripts keep working) we might as well not upgrade. Frankly I
> > suspect that down the line IF a new cli interpreter comes in to U-Boot
> > we will have to keep the old one around as a "use this instead" option
> > for another long number of years, so that if there are any systems with
> > non-trivial scripts but upgrade U-Boot and don't / won't / can't
> > re-validate their entire sequence, they can just use the old cli.
>
> Do you actually have an example where code working on our ancient
> port of hush would fail on the current upstream version?
Have you validated one of those exceedingly complex boot scripts with a
modern hush (and some fakery for u-boot commands) ? No. I'm just
saying I expect there to be enough risk-adverse groups that just
dropping our old hush entirely might not be possible right away. Of
course, if all of the current in-tree complex cases Just Work, that
might be a good argument against needing to keep such levels of
backwards compatibility.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 14:14 ` Tom Rini
@ 2021-07-07 14:23 ` Wolfgang Denk
0 siblings, 0 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-07 14:23 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Tom,
In message <20210707141418.GZ9516@bill-the-cat> you wrote:
>
> Have you validated one of those exceedingly complex boot scripts with a
> modern hush (and some fakery for u-boot commands) ? No.
No, I havent.
I also don't claim that I know all the warts and issues with our old
hus, but for the ones I am aware the woraround usually splitting
complex or nested expressions into a sequence of simpler steps. I
cannot remember any cases where the resulting code should be
incompatible to a shell without that specific bug.
And yes, I am aware of the problems with the distro_bootcmd stuff.
Thisis exactly what I had in mind when I wrote: if you run into such
situations you should lean back and reflect a bit. I can undrstand
the intentions of all this stuff, but the implementation is a
horrible mess.
> I'm just
> saying I expect there to be enough risk-adverse groups that just
> dropping our old hush entirely might not be possible right away. Of
> course, if all of the current in-tree complex cases Just Work, that
> might be a good argument against needing to keep such levels of
> backwards compatibility.
There is only one way to test this.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The inappropriate cannot be beautiful.
- Frank Lloyd Wright _The Future of Architecture_ (1953)
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-05 19:10 ` Tom Rini
2021-07-05 19:47 ` Sean Anderson
@ 2021-07-06 7:44 ` Wolfgang Denk
2021-07-06 15:43 ` Tom Rini
2021-07-08 4:56 ` Sean Anderson
1 sibling, 2 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-06 7:44 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Tom,
In message <20210705191058.GB9516@bill-the-cat> you wrote:
>
> > > foo=bar set foo bar
> > > echo $foo echo $foo
> > >
> > > if [ 1 -gt 2 ]; then if {1 > 2} {
> > > echo a echo a
> > > else } {
> > > echo b echo b
> > > fi }
> > >
> > > foo() { proc foo {first second} {
> > > echo $1 $2 echo $first $second
> > > } }
> > >
> > > for file in $(ls *.c); do foreach file [glob *.c] {
> > > echo $file echo $file
> > > done }
> > >
> > > fact() {
> > > if [ $1 -eq 0 ]; then
> > > echo 1
> > > else
> > > echo $(($1 * $(fact $(($1 - 1)))))
> > > fi
> > > }
> > >
> > > proc fact {n} {
> > > if {$n} {
> > > expr {$n * [fact [expr {$n - 1}]]}
> > > } {
> > > return 1
> > > }
> > > }
> > >
> > > Hopefully this gives you a bit of a feel for the basic differences.
>
> Which of these things, from each column, can you do in the context of
> U-Boot? That's important too.
Well, with a current version of hush we can do:
-> foo=bar
-> echo $foo
bar
-> if [ 1 -gt 2 ]; then
> echo a
> else
> echo b
> fi
b
-> foo() {
> echo $1 $2
> }
-> foo bar baz
bar baz
-> for file in $(ls *.c); do
> echo $file
> done
ls: cannot access '*.c': No such file or directory
-> fact() {
> if [ $1 -eq 0 ]; then
> echo 1
> else
> echo $(($1 * $(fact $(($1 - 1)))))
> fi
> }
-> fact 4
24
Oh, in the contect of U-Boot? Well, there are of course
limitations, but not because of the shell, but because of the fact
that we have no concept of files, for example.
But another command interpreter will not fix this.
> This is I think the hard question. A draw of the current shell is that
> it it looks and acts like bash/sh/etc, for at least basic operations.
> That's something that's comfortable to a large audience. That has
> disadvantages when people want to start doing something complex. Sean
> has shown that several times and he's not the only one. LIL being
> tcl'ish is not.
Tcl is a horror of a language for anything that is above trivial
level.
Do you really think that replacing standard shell syntax with Tcl is
"something that's comfortable to a large audience"? I seriously
doubt that.
> Something that has "sh" syntax but also clear to the user errors when
> trying to do something not supported would also be interesting to see.
> It seems like a lot of the frustration from users with our shell is that
> it's not clear where the line between "this is an sh-like shell" and
> "no, not like that" is.
Did you run some tests on the version of hush as comes with recent
busybox releases? Which of our user's requirements does it fail to
meet?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The first thing we do is kill all the lawyers.
(Shakespeare. II Henry VI, Act IV, scene ii)
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 7:44 ` Wolfgang Denk
@ 2021-07-06 15:43 ` Tom Rini
2021-07-06 16:09 ` Kostas Michalopoulos
2021-07-07 8:15 ` Wolfgang Denk
2021-07-08 4:56 ` Sean Anderson
1 sibling, 2 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-06 15:43 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 4163 bytes --]
On Tue, Jul 06, 2021 at 09:44:20AM +0200, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210705191058.GB9516@bill-the-cat> you wrote:
> >
> > > > foo=bar set foo bar
> > > > echo $foo echo $foo
> > > >
> > > > if [ 1 -gt 2 ]; then if {1 > 2} {
> > > > echo a echo a
> > > > else } {
> > > > echo b echo b
> > > > fi }
> > > >
> > > > foo() { proc foo {first second} {
> > > > echo $1 $2 echo $first $second
> > > > } }
> > > >
> > > > for file in $(ls *.c); do foreach file [glob *.c] {
> > > > echo $file echo $file
> > > > done }
> > > >
> > > > fact() {
> > > > if [ $1 -eq 0 ]; then
> > > > echo 1
> > > > else
> > > > echo $(($1 * $(fact $(($1 - 1)))))
> > > > fi
> > > > }
> > > >
> > > > proc fact {n} {
> > > > if {$n} {
> > > > expr {$n * [fact [expr {$n - 1}]]}
> > > > } {
> > > > return 1
> > > > }
> > > > }
> > > >
> > > > Hopefully this gives you a bit of a feel for the basic differences.
> >
> > Which of these things, from each column, can you do in the context of
> > U-Boot? That's important too.
>
> Well, with a current version of hush we can do:
>
> -> foo=bar
> -> echo $foo
> bar
>
> -> if [ 1 -gt 2 ]; then
> > echo a
> > else
> > echo b
> > fi
> b
>
> -> foo() {
> > echo $1 $2
> > }
> -> foo bar baz
> bar baz
>
> -> for file in $(ls *.c); do
> > echo $file
> > done
> ls: cannot access '*.c': No such file or directory
>
> -> fact() {
> > if [ $1 -eq 0 ]; then
> > echo 1
> > else
> > echo $(($1 * $(fact $(($1 - 1)))))
> > fi
> > }
> -> fact 4
> 24
>
>
> Oh, in the contect of U-Boot? Well, there are of course
> limitations, but not because of the shell, but because of the fact
> that we have no concept of files, for example.
>
> But another command interpreter will not fix this.
Yes, clearly the file based examples won't work either way, as-is. I
was asking for what things can be done today with the implementations we
have now.
I'm pretty confident that exactly zero people have written complex
U-Boot scripts and then been happy about the experience.
> > This is I think the hard question. A draw of the current shell is that
> > it it looks and acts like bash/sh/etc, for at least basic operations.
> > That's something that's comfortable to a large audience. That has
> > disadvantages when people want to start doing something complex. Sean
> > has shown that several times and he's not the only one. LIL being
> > tcl'ish is not.
>
> Tcl is a horror of a language for anything that is above trivial
> level.
TCL has its fans. csh has it's fans. The question isn't what's the
best desktop shell or general scripting language, but what's the most
useful in our environment an use cases.
> Do you really think that replacing standard shell syntax with Tcl is
> "something that's comfortable to a large audience"? I seriously
> doubt that.
I don't know if it's right either. But drawing on my comment just now
and above about complex boot scripts, I also don't know if "it's sh but
quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
TCL? I don't know that, let me hit stackoverflow and do a little
reading" as would be the common experience. Especially if we document
up-front what the quirks we have are.
> > Something that has "sh" syntax but also clear to the user errors when
> > trying to do something not supported would also be interesting to see.
> > It seems like a lot of the frustration from users with our shell is that
> > it's not clear where the line between "this is an sh-like shell" and
> > "no, not like that" is.
>
> Did you run some tests on the version of hush as comes with recent
> busybox releases? Which of our user's requirements does it fail to
> meet?
It fails to meet the requirement of having been ported to U-Boot. I'd
joke that perhaps you can bootefi busybox, but a quick search says
that particular trick looks to still just be for python
(https://lwn.net/Articles/641244/).
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 15:43 ` Tom Rini
@ 2021-07-06 16:09 ` Kostas Michalopoulos
2021-07-07 13:32 ` Sean Anderson
2021-07-07 8:15 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Kostas Michalopoulos @ 2021-07-06 16:09 UTC (permalink / raw)
To: Tom Rini, Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt
On 7/6/2021 6:43 PM, Tom Rini wrote:
> I don't know if it's right either. But drawing on my comment just now
> and above about complex boot scripts, I also don't know if "it's sh but
> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> TCL? I don't know that, let me hit stackoverflow and do a little
> reading" as would be the common experience. Especially if we document
> up-front what the quirks we have are.
I think the same would happen with Tcl and LIL. LIL might look similar
to Tcl (and it is inspired by it), but it doesn't have the same syntax.
A big difference comes from how variables are parsed with LIL allowing
more variables symbols than Tcl without requiring escaping which affects
some uses like expr, so e.g. in Tcl this
set a 10 ; set b 20 ; expr $a+$b
will give back 30 however in LIL will give back 20. This is because Tcl
parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+" "$b",
evaluates "$a+" as an empty string (there isn't any "a+" variable), "$b"
as 20 and then just runs expr with the single argument "20".
Kostas
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 16:09 ` Kostas Michalopoulos
@ 2021-07-07 13:32 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-07 13:32 UTC (permalink / raw)
To: Kostas Michalopoulos, Tom Rini, Wolfgang Denk
Cc: u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt
On 7/6/21 12:09 PM, Kostas Michalopoulos wrote:
> On 7/6/2021 6:43 PM, Tom Rini wrote:
>> I don't know if it's right either. But drawing on my comment just now
>> and above about complex boot scripts, I also don't know if "it's sh but
>> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
>> TCL? I don't know that, let me hit stackoverflow and do a little
>> reading" as would be the common experience. Especially if we document
>> up-front what the quirks we have are.
>
> I think the same would happen with Tcl and LIL. LIL might look similar
> to Tcl (and it is inspired by it), but it doesn't have the same
> syntax. A big difference comes from how variables are parsed with LIL
> allowing more variables symbols than Tcl without requiring escaping
> which affects some uses like expr, so e.g. in Tcl this
>
> set a 10 ; set b 20 ; expr $a+$b
>
> will give back 30 however in LIL will give back 20. This is because
> Tcl parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+"
> "$b", evaluates "$a+" as an empty string (there isn't any "a+"
> variable), "$b" as 20 and then just runs expr with the single argument
> "20".
IMO both of these are nuts. I think the best thing to do here is
to raise an error about a missing variable "a+" to force the user to
rewrite this as
set a 10 ; set b 20 ; expr ${a}+$b
removing the ambiguity. Although I would generally like to follow TCL's
lead, there are several places where TCL does crazy things (e.g.
comments) which I have purposely not replicated.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 15:43 ` Tom Rini
2021-07-06 16:09 ` Kostas Michalopoulos
@ 2021-07-07 8:15 ` Wolfgang Denk
2021-07-07 13:46 ` Sean Anderson
` (2 more replies)
1 sibling, 3 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-07 8:15 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Tom,
In message <20210706154346.GT9516@bill-the-cat> you wrote:
>
> I'm pretty confident that exactly zero people have written complex
> U-Boot scripts and then been happy about the experience.
I have seen many U-Boot scripts which were pretty complex, but
working absolutely reliably.
> TCL has its fans. csh has it's fans. The question isn't what's the
> best desktop shell or general scripting language, but what's the most
> useful in our environment an use cases.
Maybe you should try and do a poll of our user base which CLI they
_want_? I doubt there will be any significant percentage voting for
Tcl.
I know of a large number of systems which offer a shell interface on
their command line, and those who don't usually use completely
proprietary code. I know of very few examples where Tcl is being
used.
> I don't know if it's right either. But drawing on my comment just now
> and above about complex boot scripts, I also don't know if "it's sh but
> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> TCL? I don't know that, let me hit stackoverflow and do a little
> reading" as would be the common experience. Especially if we document
> up-front what the quirks we have are.
Point taken. But if you think this to an end, the result is: lets
write some documentation and explain the limitations of a shell in
U-Boot environment, and document the warts and bugs of this (or an
updated) version of hush. This should make more users happy than
completely new and incompatible stuff.
Frankly, I believe when you run into problems with hush in U-Boot
(even the current version) you should lean back and think about what
you are doing.
U-Boot is a boot loader, and while it is powerful enough to do
complex things, this is not necessarily the most clever approach.
15 years ago, I've written complex update scripts for U-Boot. This
was easy enough to do, and worked perfectly. But there are so many
limitations in a boot loader environment. We don't do this any
more. Instead, we use an OS suitable for such tasks (Linux with
SWUpdate).
And talking about problems and limitations in U-Boot... Is the CLI
really our biggest concern right now? None of our users (customers)
has asked for a better command interpreter - the question we hear
are more like: "When will you support IPv6?", "NFS does not work
with recent Linux distros, will this be fixed?", "Can I download
over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden
U-Boot for security-critical environments?", etc.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Always try to do things in chronological order; it's less confusing
that way.
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 8:15 ` Wolfgang Denk
@ 2021-07-07 13:46 ` Sean Anderson
2021-07-07 13:51 ` Tom Rini
2021-07-07 13:58 ` Tom Rini
2021-07-07 14:48 ` Marek Behun
2 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-07 13:46 UTC (permalink / raw)
To: Wolfgang Denk, Tom Rini
Cc: u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/7/21 4:15 AM, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210706154346.GT9516@bill-the-cat> you wrote:
>>
>> I'm pretty confident that exactly zero people have written complex
>> U-Boot scripts and then been happy about the experience.
>
> I have seen many U-Boot scripts which were pretty complex, but
> working absolutely reliably.
>
>> TCL has its fans. csh has it's fans. The question isn't what's the
>> best desktop shell or general scripting language, but what's the most
>> useful in our environment an use cases.
>
> Maybe you should try and do a poll of our user base which CLI they
> _want_? I doubt there will be any significant percentage voting for
> Tcl.
>
> I know of a large number of systems which offer a shell interface on
> their command line, and those who don't usually use completely
> proprietary code. I know of very few examples where Tcl is being
> used.
Off the top of my head, most of the tooling for FPGAs uses TCL for
scripting. OpenOCD uses it too.
>> I don't know if it's right either. But drawing on my comment just now
>> and above about complex boot scripts, I also don't know if "it's sh but
>> quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
>> TCL? I don't know that, let me hit stackoverflow and do a little
>> reading" as would be the common experience. Especially if we document
>> up-front what the quirks we have are.
>
>
> Point taken. But if you think this to an end, the result is: lets
> write some documentation and explain the limitations of a shell in
> U-Boot environment, and document the warts and bugs of this (or an
> updated) version of hush. This should make more users happy than
> completely new and incompatible stuff.
>
>
> Frankly, I believe when you run into problems with hush in U-Boot
> (even the current version) you should lean back and think about what
> you are doing.
>
> U-Boot is a boot loader, and while it is powerful enough to do
> complex things, this is not necessarily the most clever approach.
> 15 years ago, I've written complex update scripts for U-Boot. This
> was easy enough to do, and worked perfectly. But there are so many
> limitations in a boot loader environment. We don't do this any
> more. Instead, we use an OS suitable for such tasks (Linux with
> SWUpdate).
>
>
> And talking about problems and limitations in U-Boot... Is the CLI
> really our biggest concern right now? None of our users (customers)
> has asked for a better command interpreter - the question we hear
> are more like: "When will you support IPv6?", "NFS does not work
> with recent Linux distros, will this be fixed?", "Can I download
> over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden
> U-Boot for security-critical environments?", etc.
I wanted a better shell, so I worked on it.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 13:46 ` Sean Anderson
@ 2021-07-07 13:51 ` Tom Rini
0 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-07 13:51 UTC (permalink / raw)
To: Sean Anderson
Cc: Wolfgang Denk, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 3364 bytes --]
On Wed, Jul 07, 2021 at 09:46:20AM -0400, Sean Anderson wrote:
>
> On 7/7/21 4:15 AM, Wolfgang Denk wrote:
> > Dear Tom,
> >
> > In message <20210706154346.GT9516@bill-the-cat> you wrote:
> > >
> > > I'm pretty confident that exactly zero people have written complex
> > > U-Boot scripts and then been happy about the experience.
> >
> > I have seen many U-Boot scripts which were pretty complex, but
> > working absolutely reliably.
> >
> > > TCL has its fans. csh has it's fans. The question isn't what's the
> > > best desktop shell or general scripting language, but what's the most
> > > useful in our environment an use cases.
> >
> > Maybe you should try and do a poll of our user base which CLI they
> > _want_? I doubt there will be any significant percentage voting for
> > Tcl.
> >
> > I know of a large number of systems which offer a shell interface on
> > their command line, and those who don't usually use completely
> > proprietary code. I know of very few examples where Tcl is being
> > used.
>
> Off the top of my head, most of the tooling for FPGAs uses TCL for
> scripting. OpenOCD uses it too.
>
> > > I don't know if it's right either. But drawing on my comment just now
> > > and above about complex boot scripts, I also don't know if "it's sh but
> > > quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> > > TCL? I don't know that, let me hit stackoverflow and do a little
> > > reading" as would be the common experience. Especially if we document
> > > up-front what the quirks we have are.
> >
> >
> > Point taken. But if you think this to an end, the result is: lets
> > write some documentation and explain the limitations of a shell in
> > U-Boot environment, and document the warts and bugs of this (or an
> > updated) version of hush. This should make more users happy than
> > completely new and incompatible stuff.
> >
> >
> > Frankly, I believe when you run into problems with hush in U-Boot
> > (even the current version) you should lean back and think about what
> > you are doing.
> >
> > U-Boot is a boot loader, and while it is powerful enough to do
> > complex things, this is not necessarily the most clever approach.
> > 15 years ago, I've written complex update scripts for U-Boot. This
> > was easy enough to do, and worked perfectly. But there are so many
> > limitations in a boot loader environment. We don't do this any
> > more. Instead, we use an OS suitable for such tasks (Linux with
> > SWUpdate).
> >
> >
> > And talking about problems and limitations in U-Boot... Is the CLI
> > really our biggest concern right now? None of our users (customers)
> > has asked for a better command interpreter - the question we hear
> > are more like: "When will you support IPv6?", "NFS does not work
> > with recent Linux distros, will this be fixed?", "Can I download
> > over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden
> > U-Boot for security-critical environments?", etc.
>
> I wanted a better shell, so I worked on it.
This here is also an important point, and why I'm commenting on the
series. A developer sees a problem, and works on the problem. I know I
don't comment on as much stuff as I should, but for wide reaching
patches, I really really try to.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 8:15 ` Wolfgang Denk
2021-07-07 13:46 ` Sean Anderson
@ 2021-07-07 13:58 ` Tom Rini
2021-07-07 14:48 ` Marek Behun
2 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-07 13:58 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 3114 bytes --]
On Wed, Jul 07, 2021 at 10:15:34AM +0200, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210706154346.GT9516@bill-the-cat> you wrote:
> >
> > I'm pretty confident that exactly zero people have written complex
> > U-Boot scripts and then been happy about the experience.
>
> I have seen many U-Boot scripts which were pretty complex, but
> working absolutely reliably.
As have I. We have them in-tree, even. My point is not "you cannot
write these". My point is "you come away very disappointed in the
interpreter" after writing these. It's certainly not bash/zsh. But
it's also not POSIX-sh. It's not modern hush. It's unique, slightly
different from everything else people normally have easy access to and
those quirks aren't documented really.
> > TCL has its fans. csh has it's fans. The question isn't what's the
> > best desktop shell or general scripting language, but what's the most
> > useful in our environment an use cases.
>
> Maybe you should try and do a poll of our user base which CLI they
> _want_? I doubt there will be any significant percentage voting for
> Tcl.
>
> I know of a large number of systems which offer a shell interface on
> their command line, and those who don't usually use completely
> proprietary code. I know of very few examples where Tcl is being
> used.
A good point, yes.
> > I don't know if it's right either. But drawing on my comment just now
> > and above about complex boot scripts, I also don't know if "it's sh but
> > quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's
> > TCL? I don't know that, let me hit stackoverflow and do a little
> > reading" as would be the common experience. Especially if we document
> > up-front what the quirks we have are.
>
> Point taken. But if you think this to an end, the result is: lets
> write some documentation and explain the limitations of a shell in
> U-Boot environment, and document the warts and bugs of this (or an
> updated) version of hush. This should make more users happy than
> completely new and incompatible stuff.
That would be much appreciated. And to reiterate, our current hush
isn't going away soon, or likely entirely within the decade. So this
would be something long term useful to have written.
> Frankly, I believe when you run into problems with hush in U-Boot
> (even the current version) you should lean back and think about what
> you are doing.
Agreed.
> U-Boot is a boot loader, and while it is powerful enough to do
> complex things, this is not necessarily the most clever approach.
> 15 years ago, I've written complex update scripts for U-Boot. This
> was easy enough to do, and worked perfectly. But there are so many
> limitations in a boot loader environment. We don't do this any
> more. Instead, we use an OS suitable for such tasks (Linux with
> SWUpdate).
Yes. Although there are some complex things that we can't push up
higher (see distro_bootcmd) and update systems still tend to need
something from us (or they do it in grub instead).
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 8:15 ` Wolfgang Denk
2021-07-07 13:46 ` Sean Anderson
2021-07-07 13:58 ` Tom Rini
@ 2021-07-07 14:48 ` Marek Behun
2021-07-08 5:19 ` Michael Nazzareno Trimarchi
2 siblings, 1 reply; 107+ messages in thread
From: Marek Behun @ 2021-07-07 14:48 UTC (permalink / raw)
To: Wolfgang Denk, Tom Rini, Sean Anderson
Cc: u-boot, Simon Glass, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Dear Tom, Sean, Wolfgang and others,
here are some of my opinions for this discussion
- I agree with Wolfgang that there are far better options than
a Tcl-like shell, if we want to add another language
- I also think that instead of adding another language, it is more
preferable to improve the existing one. Adding a new language will
cause more problems in the future:
- I think it can end up with OS distributions needing to write
boot scripts in both languages, because they can't be sure which
will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages
- I think we can start improving the current U-Boot's shell in ways
that are incompatible with upstream Hush.
The idea back then, as I understand it, was to minimize man-hours
invested into the CLI code, and so an existing shell was incorporated
(with many #ifdef guards). But U-Boot has since evolved so much that
it is very probable it would be more economic to simply fork from
upsteam Hush, remove all the #ifdefs and start developing features we
want in U-Boot. Is upstream Hush even maintained properly?
What is the upstream repository? Is it
https://github.com/sheumann/hush?
- even if we decide to stay with upstream Hush and just upgrade
U-Boot's Hush to upstream (since it supports functions, arithmetic
with $((...)), command substitution with $(...), these are all nice
features), it is IMO still better than adding a new language
- one of the points Sean mentioned with LIL is that when compiled, it's
size does not exceed the size of U-Boot's Hush.
If we were to add new features into U-Boot's Hush, the code size would
certainly increase.
I think we should implement these new features, and instead of adding
a new language, we should work on minimizing the code size /
resulting U-Boot image size. This is where U-Boot will gain most not
only with it's CLI, but also everywhere else. Regarding this,
- we already have LTO
- Simon worked on dtoc so that devicetrees can be compiled into C code
- we can start playing with compression
- either we can compress the whole image for machines with enough
RAM but small place for U-Boot (Nokia N900 for example has only
256 KiB space for U-Boot)
- or we can try to invent a way to decompress code when it is
needed, for machines with small RAM
Marek
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-07 14:48 ` Marek Behun
@ 2021-07-08 5:19 ` Michael Nazzareno Trimarchi
2021-07-08 15:33 ` Tom Rini
0 siblings, 1 reply; 107+ messages in thread
From: Michael Nazzareno Trimarchi @ 2021-07-08 5:19 UTC (permalink / raw)
To: Marek Behun
Cc: Wolfgang Denk, Tom Rini, Sean Anderson, U-Boot-Denx, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Hi
On Wed, Jul 7, 2021 at 4:48 PM Marek Behun <marek.behun@nic.cz> wrote:
>
> Dear Tom, Sean, Wolfgang and others,
>
> here are some of my opinions for this discussion
>
> - I agree with Wolfgang that there are far better options than
> a Tcl-like shell, if we want to add another language
>
> - I also think that instead of adding another language, it is more
> preferable to improve the existing one. Adding a new language will
> cause more problems in the future:
> - I think it can end up with OS distributions needing to write
> boot scripts in both languages, because they can't be sure which
> will be compiled into U-Boot
> - we will certainly end up with more bugs
> - userbase will fragment between the two languages
>
> - I think we can start improving the current U-Boot's shell in ways
> that are incompatible with upstream Hush.
>
> The idea back then, as I understand it, was to minimize man-hours
> invested into the CLI code, and so an existing shell was incorporated
> (with many #ifdef guards). But U-Boot has since evolved so much that
> it is very probable it would be more economic to simply fork from
> upsteam Hush, remove all the #ifdefs and start developing features we
> want in U-Boot. Is upstream Hush even maintained properly?
> What is the upstream repository? Is it
> https://github.com/sheumann/hush?
>
I think that hush is the one that is now in the busybox. I could spent
ten minutes this morning and this is my short list:
- we have several define that allow it to enabled e/o disable a lot of features
- we are talking about 11K lines compared to 3K (including comment)
- we have 25-30 configuration option on hush on busybox
- in u-boot code some of the problem was solved some time ago
- as describe is 68Kb, I think this consider all the option enables
- the code is different from what we have and what is there
I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially
solve some of the problems described in the thread
* Sean *: You have spent more on this, can you please complete it.
Out of that. Do we have some script shell unit test in uboot?
Michael
> - even if we decide to stay with upstream Hush and just upgrade
> U-Boot's Hush to upstream (since it supports functions, arithmetic
> with $((...)), command substitution with $(...), these are all nice
> features), it is IMO still better than adding a new language
>
> - one of the points Sean mentioned with LIL is that when compiled, it's
> size does not exceed the size of U-Boot's Hush.
>
> If we were to add new features into U-Boot's Hush, the code size would
> certainly increase.
>
> I think we should implement these new features, and instead of adding
> a new language, we should work on minimizing the code size /
> resulting U-Boot image size. This is where U-Boot will gain most not
> only with it's CLI, but also everywhere else. Regarding this,
> - we already have LTO
> - Simon worked on dtoc so that devicetrees can be compiled into C code
> - we can start playing with compression
> - either we can compress the whole image for machines with enough
> RAM but small place for U-Boot (Nokia N900 for example has only
> 256 KiB space for U-Boot)
> - or we can try to invent a way to decompress code when it is
> needed, for machines with small RAM
>
> Marek
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________
Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-08 5:19 ` Michael Nazzareno Trimarchi
@ 2021-07-08 15:33 ` Tom Rini
0 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-08 15:33 UTC (permalink / raw)
To: Michael Nazzareno Trimarchi
Cc: Marek Behun, Wolfgang Denk, Sean Anderson, U-Boot-Denx,
Simon Glass, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 2498 bytes --]
On Thu, Jul 08, 2021 at 07:19:05AM +0200, Michael Nazzareno Trimarchi wrote:
> Hi
>
> On Wed, Jul 7, 2021 at 4:48 PM Marek Behun <marek.behun@nic.cz> wrote:
> >
> > Dear Tom, Sean, Wolfgang and others,
> >
> > here are some of my opinions for this discussion
> >
> > - I agree with Wolfgang that there are far better options than
> > a Tcl-like shell, if we want to add another language
> >
> > - I also think that instead of adding another language, it is more
> > preferable to improve the existing one. Adding a new language will
> > cause more problems in the future:
> > - I think it can end up with OS distributions needing to write
> > boot scripts in both languages, because they can't be sure which
> > will be compiled into U-Boot
> > - we will certainly end up with more bugs
> > - userbase will fragment between the two languages
> >
> > - I think we can start improving the current U-Boot's shell in ways
> > that are incompatible with upstream Hush.
> >
> > The idea back then, as I understand it, was to minimize man-hours
> > invested into the CLI code, and so an existing shell was incorporated
> > (with many #ifdef guards). But U-Boot has since evolved so much that
> > it is very probable it would be more economic to simply fork from
> > upsteam Hush, remove all the #ifdefs and start developing features we
> > want in U-Boot. Is upstream Hush even maintained properly?
> > What is the upstream repository? Is it
> > https://github.com/sheumann/hush?
> >
>
> I think that hush is the one that is now in the busybox. I could spent
> ten minutes this morning and this is my short list:
>
> - we have several define that allow it to enabled e/o disable a lot of features
> - we are talking about 11K lines compared to 3K (including comment)
> - we have 25-30 configuration option on hush on busybox
> - in u-boot code some of the problem was solved some time ago
> - as describe is 68Kb, I think this consider all the option enables
> - the code is different from what we have and what is there
>
> I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially
> solve some of the problems described in the thread
>
> * Sean *: You have spent more on this, can you please complete it.
>
> Out of that. Do we have some script shell unit test in uboot?
To the last point, yes, we have some, but always need more, tests for
how hush behaves as part of pytest.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-06 7:44 ` Wolfgang Denk
2021-07-06 15:43 ` Tom Rini
@ 2021-07-08 4:56 ` Sean Anderson
2021-07-08 17:00 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-08 4:56 UTC (permalink / raw)
To: Wolfgang Denk, Tom Rini
Cc: u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/6/21 3:44 AM, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210705191058.GB9516@bill-the-cat> you wrote:
>> This is I think the hard question. A draw of the current shell is that
>> it it looks and acts like bash/sh/etc, for at least basic operations.
>> That's something that's comfortable to a large audience. That has
>> disadvantages when people want to start doing something complex. Sean
>> has shown that several times and he's not the only one. LIL being
>> tcl'ish is not.
>
> Tcl is a horror of a language for anything that is above trivial
> level.
Can you please elaborate on this? You've made this claim several times,
but haven't explained your reasoning for it.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-08 4:56 ` Sean Anderson
@ 2021-07-08 17:00 ` Wolfgang Denk
0 siblings, 0 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-08 17:00 UTC (permalink / raw)
To: Sean Anderson
Cc: Tom Rini, u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <43880bf0-baa6-0cb2-80fe-0fe50d43b42f@gmail.com> you wrote:
>
> > Tcl is a horror of a language for anything that is above trivial
> > level.
>
> Can you please elaborate on this? You've made this claim several times,
> but haven't explained your reasoning for it.
A long, long time ago I I was looking for an efficient way to do
regression testing for embedded systems, mainly U-Boot and Linux. I
found GNU gnats to bee too difficult to handle und started writing
my own test system (you can still find this ancient stuff here [1])
based on expect/Tcl. At that time, I could not find any other
solution, so I had to accept Tcl as programming language for this
tool. And at the beginning it usually worked just fine. Mostly.
Coming from a C programming environment I always found Tcl ...
strange. For example it's scoping rules (see [2] for an example).
Things got worse when the system grew and we had to deal with
strings or parameters which might contail quotes or other special
characters. This required fancy quoting, which depended of the
calling depth - if there was only on specific level. you might find
a solution after some (cumbersome) experimenting. But if the same
string was used in several places in the code, and different levels,
you really lost. At a certain point all our developers refused to
write new test cases for the system, because it always was the same
nightmare of fighting unforeseeable language problems.
That was when we abandoned DUTS, and some time later Heiko came up
with an initial implementation of tbot, soon to be joined by
Harald to make a production quality test system out of it [4].
IMO, Tcl has a large number of situations where it's actual
behaviours is not what you expect from it - it is good enough for
simple things, but any more complex processing of arbitrary text
data quickly drives you insane - things like quotes or braces and
other special characters.
[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] https://wiki.tcl-lang.org/page/Commonly+encountered+problems+in+Tcl
[3] https://wiki.tcl-lang.org/page/Why+can+I+not+place+unmatched+braces+in+Tcl+comments
[4] https://tbot.tools/
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The number you have dialed is imaginary. Please divide by 0 and try
again.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 02/28] cli: Add LIL shell
2021-07-02 13:33 ` Sean Anderson
2021-07-03 2:12 ` Sean Anderson
@ 2021-07-03 19:23 ` Wolfgang Denk
1 sibling, 0 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-03 19:23 UTC (permalink / raw)
To: Sean Anderson
Cc: u-boot, Tom Rini, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <c8e20c0c-bcf0-4692-6a8b-0c4ea970b656@gmail.com> you wrote:
> >
> > Do you have a list of the exact differencec between LIL and a
> > standard shell?
>
> For a partial list, see
>
> [1] https://github.com/Forty-Bot/lil/commits/master
Hm, this list of commits is not exactly helpful, I'm afraid.
Where _exactly_ should I look?
> > I wonder, if we deviate from standard shell syntax anyway, we could
> > also have a look at lua, for example?
>
> I also looked at lua (see the cover letter), but I rejected it based on
> size constraints (eLua was around the size of U-Boot itself).
I have to admit that I never tried myself to build a lua based
system, optimizing for minimal size - but I'm surprised by your
results.
> Because of how often the shell is used to debug things, I wanted the
> candidate I picked to have similar syntax to the existing shell. For
> example,
>
> load mmc ${mmcdev}:${mmcpart} $loadaddr $image
>
> is a valid command in both Hush and LIL. Compare with lua, which might
> express the above as
Yes, I'm aware of this. But then, it would add another level of
powerful scripting capabilities - and as it was already suggested
before, _replacing_ the standard shell is only one way to use lua -
another would be to use lua as a command that can be started from
the shell when needed - assuming you want to pay the price in terms
of size.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
They're usually so busy thinking about what happens next that the
only time they ever find out what is happening now is when they come
to look back on it. - Terry Pratchett, _Wyrd Sisters_
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 01/28] Add Zlib License Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 02/28] cli: Add LIL shell Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-02 8:36 ` Rasmus Villemoes
2021-07-01 6:15 ` [RFC PATCH 04/28] cli: lil: Remove most functions by default Sean Anderson
` (26 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
Apparently strdup is not portable, so LIL used its own. Use strdup.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 27 ++++++++-------------------
1 file changed, 8 insertions(+), 19 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index c8c6986fe1..294d79e86e 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -123,17 +123,6 @@ static struct lil_env **envpool;
static size_t envpoolsize, envpoolcap;
#endif
-static char *strclone(const char *s)
-{
- size_t len = strlen(s) + 1;
- char *ns = malloc(len);
-
- if (!ns)
- return NULL;
- memcpy(ns, s, len);
- return ns;
-}
-
static unsigned long hm_hash(const char *key)
{
unsigned long hash = 5381;
@@ -173,7 +162,7 @@ static void hm_put(struct hashmap *hm, const char *key, void *value)
}
cell->e = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
- cell->e[cell->c].k = strclone(key);
+ cell->e[cell->c].k = strdup(key);
cell->e[cell->c].v = value;
cell->c++;
}
@@ -606,7 +595,7 @@ static struct lil_func *add_func(struct lil *lil, const char *name)
}
cmd = calloc(1, sizeof(struct lil_func));
- cmd->name = strclone(name);
+ cmd->name = strdup(name);
ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
if (!ncmd) {
@@ -705,7 +694,7 @@ struct lil_var *lil_set_var(struct lil *lil, const char *name,
env->var = nvar;
nvar[env->vars] = calloc(1, sizeof(struct lil_var));
- nvar[env->vars]->n = strclone(name);
+ nvar[env->vars]->n = strdup(name);
nvar[env->vars]->env = env;
nvar[env->vars]->v = freeval ? val : lil_clone_value(val);
hm_put(&env->varmap, name, nvar[env->vars]);
@@ -1207,7 +1196,7 @@ void lil_set_error(struct lil *lil, const char *msg)
free(lil->err_msg);
lil->error = ERROR_FIXHEAD;
lil->err_head = 0;
- lil->err_msg = strclone(msg ? msg : "");
+ lil->err_msg = strdup(msg ? msg : "");
}
void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
@@ -1218,7 +1207,7 @@ void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
free(lil->err_msg);
lil->error = ERROR_DEFAULT;
lil->err_head = pos;
- lil->err_msg = strclone(msg ? msg : "");
+ lil->err_msg = strdup(msg ? msg : "");
}
int lil_error(struct lil *lil, const char **msg, size_t *pos)
@@ -2016,7 +2005,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
hm_put(&lil->cmdmap, oldname, 0);
hm_put(&lil->cmdmap, newname, func);
free(func->name);
- func->name = strclone(newname);
+ func->name = strdup(newname);
} else {
del_func(lil, func);
}
@@ -2728,7 +2717,7 @@ static struct lil_value *real_trim(const char *str, const char *chars, int left,
size_t len;
char *s;
- s = strclone(str + base);
+ s = strdup(str + base);
len = strlen(s);
while (len && strchr(chars, s[len - 1]))
len--;
@@ -2833,7 +2822,7 @@ static struct lil_value *fnc_repstr(struct lil *lil, size_t argc,
if (!from[0])
return NULL;
- src = strclone(lil_to_string(argv[0]));
+ src = strdup(lil_to_string(argv[0]));
srclen = strlen(src);
fromlen = strlen(from);
tolen = strlen(to);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-01 6:15 ` [RFC PATCH 03/28] cli: lil: Replace strclone with strdup Sean Anderson
@ 2021-07-02 8:36 ` Rasmus Villemoes
2021-07-02 11:38 ` Wolfgang Denk
2021-07-02 13:38 ` Sean Anderson
0 siblings, 2 replies; 107+ messages in thread
From: Rasmus Villemoes @ 2021-07-02 8:36 UTC (permalink / raw)
To: Sean Anderson, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 01/07/2021 08.15, Sean Anderson wrote:
> Apparently strdup is not portable, so LIL used its own. Use strdup.
You could reduce the churn by just making strclone "#define strclone(x)
strdup(x)", but I suppose you end up modifying the upstream code so much
that there's not really anything gained by that.
But that begs the question: What is the long-term plan for this? While
it does seem to be an improvement compared to hush, will we ever be able
to incorporate fixes&features from upstream, or will this code end up in
the same situation as hush?
Have you been in contact with upstream about this project? Perhaps some
of the things you do could go upstream - e.g. the conversion from an
array of wrongly-typed callbacks to an "ops" struct seems to be an
obvious improvement [lil_callback would have to be kept, and changed to
use a switch() statement to update the right slot, but we wouldn't have
to care because ld will remove it anyway if we don't add any users of it].
Rasmus
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-02 8:36 ` Rasmus Villemoes
@ 2021-07-02 11:38 ` Wolfgang Denk
2021-07-02 13:38 ` Sean Anderson
1 sibling, 0 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-02 11:38 UTC (permalink / raw)
To: Rasmus Villemoes
Cc: Sean Anderson, u-boot, Tom Rini, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Rasmus,
In message <bf6008b5-3daa-76fc-2d51-0de92786c76d@prevas.dk> you wrote:
>
> But that begs the question: What is the long-term plan for this? While
> it does seem to be an improvement compared to hush, will we ever be able
> to incorporate fixes&features from upstream, or will this code end up in
> the same situation as hush?
I would like to put the context of this right.
As written, "improvement compared to hush", this could be
misunderstood. What you likely mean is "compared to our ancient
(nearly 20 years old) version of hush".
Comparing to recent versions have probably much different results.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Ever try. Ever fail. No matter. Try again. Fail again. Fail better.
-- S. Beckett
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-02 8:36 ` Rasmus Villemoes
2021-07-02 11:38 ` Wolfgang Denk
@ 2021-07-02 13:38 ` Sean Anderson
2021-07-02 14:28 ` Tom Rini
` (2 more replies)
1 sibling, 3 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-02 13:38 UTC (permalink / raw)
To: Rasmus Villemoes, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/2/21 4:36 AM, Rasmus Villemoes wrote:
> On 01/07/2021 08.15, Sean Anderson wrote:
>> Apparently strdup is not portable, so LIL used its own. Use strdup.
>
> You could reduce the churn by just making strclone "#define strclone(x)
> strdup(x)", but I suppose you end up modifying the upstream code so much
> that there's not really anything gained by that.
>
> But that begs the question: What is the long-term plan for this? While
> it does seem to be an improvement compared to hush, will we ever be able
> to incorporate fixes&features from upstream, or will this code end up in
> the same situation as hush?
Well, since Hush was never updated, I don't believe LIL will be either.
I think reducing the amount of ifdefs makes the code substantially
easier to maintain. My intention is to just use LIL as a starting point
which can be modified as needed to better suit U-Boot.
The other half of this is that LIL is not particularly actively
developed. I believe the author sees his work as essentially
feature-complete, so I expect no major features which we might like to
backport.
>
> Have you been in contact with upstream about this project? Perhaps some
> of the things you do could go upstream - e.g. the conversion from an
> array of wrongly-typed callbacks to an "ops" struct seems to be an
> obvious improvement [lil_callback would have to be kept, and changed to
> use a switch() statement to update the right slot, but we wouldn't have
> to care because ld will remove it anyway if we don't add any users of it].
I have not. I'm not sure what changes I've made are compatible with the
vision he has for LIL. He's CC'd on this series, so perhaps he can
comment.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-02 13:38 ` Sean Anderson
@ 2021-07-02 14:28 ` Tom Rini
2021-07-02 22:18 ` Kostas Michalopoulos
2021-07-03 19:26 ` Wolfgang Denk
2 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-02 14:28 UTC (permalink / raw)
To: Sean Anderson
Cc: Rasmus Villemoes, u-boot, Marek Behún, Wolfgang Denk,
Simon Glass, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 1940 bytes --]
On Fri, Jul 02, 2021 at 09:38:26AM -0400, Sean Anderson wrote:
> On 7/2/21 4:36 AM, Rasmus Villemoes wrote:
> > On 01/07/2021 08.15, Sean Anderson wrote:
> > > Apparently strdup is not portable, so LIL used its own. Use strdup.
> >
> > You could reduce the churn by just making strclone "#define strclone(x)
> > strdup(x)", but I suppose you end up modifying the upstream code so much
> > that there's not really anything gained by that.
> >
> > But that begs the question: What is the long-term plan for this? While
> > it does seem to be an improvement compared to hush, will we ever be able
> > to incorporate fixes&features from upstream, or will this code end up in
> > the same situation as hush?
>
> Well, since Hush was never updated, I don't believe LIL will be either.
> I think reducing the amount of ifdefs makes the code substantially
> easier to maintain. My intention is to just use LIL as a starting point
> which can be modified as needed to better suit U-Boot.
>
> The other half of this is that LIL is not particularly actively
> developed. I believe the author sees his work as essentially
> feature-complete, so I expect no major features which we might like to
> backport.
Port it and forget it was a problem, not a bug, of our hush
implementation. For other things, I'm trying to keep us in sync
regularly, but having less luck as some stuff has already gotten well
behind and is non-trivial to resync. A digression, but for this thread
I think keeping abreast of LIL releases is important. I diff'd 20190821
to 20210502 and it was (as you note in another part of the thread)
copyright, license addition and I saw one line of code move around but
didn't check if that applied here too or not. A monthly calendar
reminder to check the site for new releases (which is what I do for
Kconfiglib) should keep us in-sync, especially given the LIL project
intentions.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-02 13:38 ` Sean Anderson
2021-07-02 14:28 ` Tom Rini
@ 2021-07-02 22:18 ` Kostas Michalopoulos
2021-07-03 2:28 ` Sean Anderson
2021-07-03 19:26 ` Wolfgang Denk
2 siblings, 1 reply; 107+ messages in thread
From: Kostas Michalopoulos @ 2021-07-02 22:18 UTC (permalink / raw)
To: Sean Anderson, Rasmus Villemoes, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt
On 7/2/2021 4:38 PM, Sean Anderson wrote:
> I have not. I'm not sure what changes I've made are compatible with the
> vision he has for LIL. He's CC'd on this series, so perhaps he can
> comment.
Yeah, sadly several of the changes i've seen in the code are not
backwards compatible and backwards compatibility is very important for
me (this is also why the callback setup is done through a lil_callback
function instead of exposing a struct: adding new callbacks in the
future wont break any existing code or application that links to the
library dynamically). I might break it before a properly versioned
release is made at some point in the (not close) future, but that is
usually unavoidable changes to fix big issues (all cases where that
happened are mentioned in the site).
In terms of future changes, i do not plan making any *big* changes - and
certainly anything i'll make it'll be backwards compatible (especially
after a versioned release) but one thing that i need to do is to improve
the interpreter's performance, which will affect its internals - two
areas i might need to change are how values are represented (everything
is a string -and things must always behave like that- but i may need to
cache already parsed non-string values) and how the code itself -e.g. in
functions- is represented.
Also i'll most likely add more functions to the library to expose and
alter internal state (the Free Pascal version of LIL already has some of
that), but that shouldn't affect much internally.
Beyond that it will be mainly bugfixes - like a use-after-free case with
"reflect this" i just fixed and uploaded :-P
Kostas
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-02 22:18 ` Kostas Michalopoulos
@ 2021-07-03 2:28 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-03 2:28 UTC (permalink / raw)
To: Kostas Michalopoulos, Rasmus Villemoes, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt
On 7/2/21 6:18 PM, Kostas Michalopoulos wrote:
> On 7/2/2021 4:38 PM, Sean Anderson wrote:
>> I have not. I'm not sure what changes I've made are compatible with the
>> vision he has for LIL. He's CC'd on this series, so perhaps he can
>> comment.
>
> Yeah, sadly several of the changes i've seen in the code are not
> backwards compatible and backwards compatibility is very important for
> me
I don't intend to keep backward compatibility. The way I see it, U-Boot
has never had a LIL shell before, so there is nothing to break :)
> (this is also why the callback setup is done through a lil_callback
> function instead of exposing a struct: adding new callbacks in the
> future wont break any existing code or application that links to the
> library dynamically).
Well, C has a very nice ABI in some repects. As long as you only append
to that struct you will not break compatibility.
> I might break it before a properly versioned release is made at some
> point in the (not close) future, but that is usually unavoidable
> changes to fix big issues (all cases where that happened are mentioned
> in the site).
>
> In terms of future changes, i do not plan making any *big* changes -
> and certainly anything i'll make it'll be backwards compatible
> (especially after a versioned release) but one thing that i need to do
> is to improve the interpreter's performance, which will affect its
> internals - two areas i might need to change are how values are
> represented (everything is a string -and things must always behave
> like that- but i may need to cache already parsed non-string values)
> and how the code itself -e.g. in functions- is represented.
One of the nice side-effects of adding a separate parser is that it will
now be very easy to get some performance gains by parsing code once and
then re-using that parse on subsequent evaluations. For example,
functions could be parsed on first use, and cached. Loop bodies could
re-used for subsequent iterations.
> Also i'll most likely add more functions to the library to expose and
> alter internal state (the Free Pascal version of LIL already has some
> of that), but that shouldn't affect much internally.
>
> Beyond that it will be mainly bugfixes - like a use-after-free case
> with "reflect this" i just fixed and uploaded :-P
As it happens, I ended up removing that because there was no longer a
"rootcode" after removing the parsing step. Perhaps at some point I will
add it back, but I do not see a pressing need for the same level of
dynamicness that you built in originally.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-02 13:38 ` Sean Anderson
2021-07-02 14:28 ` Tom Rini
2021-07-02 22:18 ` Kostas Michalopoulos
@ 2021-07-03 19:26 ` Wolfgang Denk
2021-07-05 5:07 ` Steve Bennett
2 siblings, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-03 19:26 UTC (permalink / raw)
To: Sean Anderson
Cc: Rasmus Villemoes, u-boot, Tom Rini, Marek Behún,
Simon Glass, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Dear Sean,
In message <d3a91238-db2f-edbe-ecec-ddb5dc848ed7@gmail.com> you wrote:
>
> Well, since Hush was never updated, I don't believe LIL will be either.
Let's please be exact here: Hus has never been updated _in_U-Boot_,
but it has seen a lot of changes upstream, which apparently fix all
the issues that motivated you to look for a replacement.
> I think reducing the amount of ifdefs makes the code substantially
> easier to maintain. My intention is to just use LIL as a starting point
> which can be modified as needed to better suit U-Boot.
>
> The other half of this is that LIL is not particularly actively
> developed. I believe the author sees his work as essentially
> feature-complete, so I expect no major features which we might like to
> backport.
This sounds like an advantage, indeed, but then you can also
interpret this as betting on a dead horse...
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Never worry about theory as long as the machinery does what it's
supposed to do. - R. A. Heinlein
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-03 19:26 ` Wolfgang Denk
@ 2021-07-05 5:07 ` Steve Bennett
2021-07-05 14:42 ` Sean Anderson
0 siblings, 1 reply; 107+ messages in thread
From: Steve Bennett @ 2021-07-05 5:07 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, Rasmus Villemoes, u-boot, Tom Rini,
Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 4 Jul 2021, at 5:26 am, Wolfgang Denk <wd@denx.de> wrote:
>
> Dear Sean,
>
> In message <d3a91238-db2f-edbe-ecec-ddb5dc848ed7@gmail.com> you wrote:
>>
>> Well, since Hush was never updated, I don't believe LIL will be either.
>
> Let's please be exact here: Hus has never been updated _in_U-Boot_,
> but it has seen a lot of changes upstream, which apparently fix all
> the issues that motivated you to look for a replacement.
>
>> I think reducing the amount of ifdefs makes the code substantially
>> easier to maintain. My intention is to just use LIL as a starting point
>> which can be modified as needed to better suit U-Boot.
>>
>> The other half of this is that LIL is not particularly actively
>> developed. I believe the author sees his work as essentially
>> feature-complete, so I expect no major features which we might like to
>> backport.
>
> This sounds like an advantage, indeed, but then you can also
> interpret this as betting on a dead horse...
My 2c on this.
I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate).
LIL source code has almost zero comments, poor error checking and no test suite.
I would be very hesitant to adopt it in u-boot without serious work.
I would much rather see effort put into updating hush to upstream.
My guess is that Denys would be amenable to small changes to make it easier to synchronise
with busybox in the future.
Cheers,
Steve
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-05 5:07 ` Steve Bennett
@ 2021-07-05 14:42 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
2021-07-05 17:50 ` Wolfgang Denk
0 siblings, 2 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-05 14:42 UTC (permalink / raw)
To: Steve Bennett, Wolfgang Denk
Cc: Rasmus Villemoes, u-boot, Tom Rini, Marek Behún,
Simon Glass, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
On 7/5/21 1:07 AM, Steve Bennett wrote:
> On 4 Jul 2021, at 5:26 am, Wolfgang Denk <wd@denx.de> wrote:
>>
>> Dear Sean,
>>
>> In message <d3a91238-db2f-edbe-ecec-ddb5dc848ed7@gmail.com> you wrote:
>>>
>>> Well, since Hush was never updated, I don't believe LIL will be either.
>>
>> Let's please be exact here: Hus has never been updated _in_U-Boot_,
>> but it has seen a lot of changes upstream, which apparently fix all
>> the issues that motivated you to look for a replacement.
>>
>>> I think reducing the amount of ifdefs makes the code substantially
>>> easier to maintain. My intention is to just use LIL as a starting point
>>> which can be modified as needed to better suit U-Boot.
>>>
>>> The other half of this is that LIL is not particularly actively
>>> developed. I believe the author sees his work as essentially
>>> feature-complete, so I expect no major features which we might like to
>>> backport.
>>
>> This sounds like an advantage, indeed, but then you can also
>> interpret this as betting on a dead horse...
>
> My 2c on this.
>
> I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate).
> LIL source code has almost zero comments, poor error checking and no test suite.
FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests
for LIL" based on the tests included in the LIL distibution. However, I
really would like to expand upon it.
> I would be very hesitant to adopt it in u-boot without serious work.
I think around half of the "serious work" has already been done. I have
worked on most of the core of LIL, and added error handling and
comments. I believe that most of the remaining instances of dropping
errors lie in the built-in commands.
> I would much rather see effort put into updating hush to upstream.
AIUI hush has diverged significantly from what U-Boot has. This would
not be an "update" moreso than a complete port in the style of the
current series.
> My guess is that Denys would be amenable to small changes to make it easier to synchronise
> with busybox in the future.
I don't think sh-style shells are a good match for U-Boot's execution
environment in the first place. The fundamental idea of an sh-style
shell is that the output of one command can be redirected to the input
(or arguments) of another command. This cannot be done (or rather would
be difficult to do) in U-Boot for a few reasons
* U-Boot does not support multithreading. Existing shells tend to depend
strongly on this feature of the enviromnent. Many of the changes to
U-Boot's hush are solely to deal with the lack of this feature.
* Existing commands do not read from stdin, nor do they print useful
information to stdout. Command output is designed for human
consumption and is substantially more verbose than typical unix
commands.
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
useful when working with streams are not present in U-Boot.
And of course, this feature is currently not present in U-Boot. To get
around this, commands resort to two of my least-favorite hacks: passing
in the name of a environmental variable and overloading the return
value. For an example of the first, consider
=> part uuid mmc 0:1 my_uuid
which will set my_uuid to the uuid of the selected partition. My issue
with this is threefold: every command must add new syntax to do this,
that syntax is inconsistent, and it prevents easy composition. Consider
a script which wants to iterate over partitions. Instead of doing
for p in $(part list mmc 0); do
# ...
done
it must instead do
part list mmc 0 partitions
for p in $partitions; do
# ...
done
which unnecessarily adds an extra step. This overhead accumulates with
each command which adds something like this.
The other way to return more information is to use the return value.
Consider the button command; it currently returns
0 ON, the button is pressed
1 OFF, the button is released
0 button list was shown
1 button not found
1 invalid arguments
and so there is no way to distinguish between whether the button is off,
whether the button does not exist, or whether there was a problem with
the button driver.
Both of these workarounds are natural consequences of using a sh-tyle
shell in an environment it is not suited for. If we are going to go to
the effort of porting a new language (which must be done no matter if
we use Hush or some other language), we should pick one which has better
support for single-threaded programming.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-05 14:42 ` Sean Anderson
@ 2021-07-05 15:29 ` Simon Glass
2021-07-05 15:42 ` Sean Anderson
2021-07-05 17:50 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: Steve Bennett, Wolfgang Denk, Rasmus Villemoes,
U-Boot Mailing List, Tom Rini, Marek Behún, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Hi,
On Mon, 5 Jul 2021 at 08:42, Sean Anderson <seanga2@gmail.com> wrote:
>
> On 7/5/21 1:07 AM, Steve Bennett wrote:
> > On 4 Jul 2021, at 5:26 am, Wolfgang Denk <wd@denx.de> wrote:
> >>
> >> Dear Sean,
> >>
> >> In message <d3a91238-db2f-edbe-ecec-ddb5dc848ed7@gmail.com> you wrote:
> >>>
> >>> Well, since Hush was never updated, I don't believe LIL will be either.
> >>
> >> Let's please be exact here: Hus has never been updated _in_U-Boot_,
> >> but it has seen a lot of changes upstream, which apparently fix all
> >> the issues that motivated you to look for a replacement.
> >>
> >>> I think reducing the amount of ifdefs makes the code substantially
> >>> easier to maintain. My intention is to just use LIL as a starting point
> >>> which can be modified as needed to better suit U-Boot.
> >>>
> >>> The other half of this is that LIL is not particularly actively
> >>> developed. I believe the author sees his work as essentially
> >>> feature-complete, so I expect no major features which we might like to
> >>> backport.
> >>
> >> This sounds like an advantage, indeed, but then you can also
> >> interpret this as betting on a dead horse...
> >
> > My 2c on this.
> >
> > I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate).
> > LIL source code has almost zero comments, poor error checking and no test suite.
>
> FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests
> for LIL" based on the tests included in the LIL distibution. However, I
> really would like to expand upon it.
>
> > I would be very hesitant to adopt it in u-boot without serious work.
>
> I think around half of the "serious work" has already been done. I have
> worked on most of the core of LIL, and added error handling and
> comments. I believe that most of the remaining instances of dropping
> errors lie in the built-in commands.
>
> > I would much rather see effort put into updating hush to upstream.
>
> AIUI hush has diverged significantly from what U-Boot has. This would
> not be an "update" moreso than a complete port in the style of the
> current series.
>
> > My guess is that Denys would be amenable to small changes to make it easier to synchronise
> > with busybox in the future.
>
> I don't think sh-style shells are a good match for U-Boot's execution
> environment in the first place. The fundamental idea of an sh-style
> shell is that the output of one command can be redirected to the input
> (or arguments) of another command. This cannot be done (or rather would
> be difficult to do) in U-Boot for a few reasons
>
> * U-Boot does not support multithreading. Existing shells tend to depend
> strongly on this feature of the enviromnent. Many of the changes to
> U-Boot's hush are solely to deal with the lack of this feature.
>
> * Existing commands do not read from stdin, nor do they print useful
> information to stdout. Command output is designed for human
> consumption and is substantially more verbose than typical unix
> commands.
>
> * Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
> useful when working with streams are not present in U-Boot.
>
> And of course, this feature is currently not present in U-Boot. To get
> around this, commands resort to two of my least-favorite hacks: passing
> in the name of a environmental variable and overloading the return
> value. For an example of the first, consider
>
> => part uuid mmc 0:1 my_uuid
>
> which will set my_uuid to the uuid of the selected partition. My issue
> with this is threefold: every command must add new syntax to do this,
> that syntax is inconsistent, and it prevents easy composition. Consider
> a script which wants to iterate over partitions. Instead of doing
>
> for p in $(part list mmc 0); do
> # ...
> done
>
> it must instead do
>
> part list mmc 0 partitions
> for p in $partitions; do
> # ...
> done
>
> which unnecessarily adds an extra step. This overhead accumulates with
> each command which adds something like this.
>
> The other way to return more information is to use the return value.
> Consider the button command; it currently returns
>
> 0 ON, the button is pressed
> 1 OFF, the button is released
> 0 button list was shown
> 1 button not found
> 1 invalid arguments
>
> and so there is no way to distinguish between whether the button is off,
> whether the button does not exist, or whether there was a problem with
> the button driver.
>
> Both of these workarounds are natural consequences of using a sh-tyle
> shell in an environment it is not suited for. If we are going to go to
> the effort of porting a new language (which must be done no matter if
> we use Hush or some other language), we should pick one which has better
> support for single-threaded programming.
I think these are good points, particularly the thing about
mutiltasking. In fact I think it would be better if hush could
implement things like '| grep xxx' since it would be useful in U-Boot.
But in any case, adding lil does not preclude someone coming along and
adaptive hush for U-Boot (and this time upstreaming the changes!). I
have seen discussions about updating hush for several years and no one
has done it. I took a quick look and found it was about 3x the size it
used to be and was not even sure where to start in terms of adapting
it for single-threaded use.
Re this patch, I think it should be sent upstream.
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-05 15:29 ` Simon Glass
@ 2021-07-05 15:42 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-05 15:42 UTC (permalink / raw)
To: Simon Glass
Cc: Steve Bennett, Wolfgang Denk, Rasmus Villemoes,
U-Boot Mailing List, Tom Rini, Marek Behún, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/5/21 11:29 AM, Simon Glass wrote:
> Hi,
>
> On Mon, 5 Jul 2021 at 08:42, Sean Anderson <seanga2@gmail.com> wrote:
>>
>> On 7/5/21 1:07 AM, Steve Bennett wrote:
>>> On 4 Jul 2021, at 5:26 am, Wolfgang Denk <wd@denx.de> wrote:
>>>>
>>>> Dear Sean,
>>>>
>>>> In message <d3a91238-db2f-edbe-ecec-ddb5dc848ed7@gmail.com> you wrote:
>>>>>
>>>>> Well, since Hush was never updated, I don't believe LIL will be either.
>>>>
>>>> Let's please be exact here: Hus has never been updated _in_U-Boot_,
>>>> but it has seen a lot of changes upstream, which apparently fix all
>>>> the issues that motivated you to look for a replacement.
>>>>
>>>>> I think reducing the amount of ifdefs makes the code substantially
>>>>> easier to maintain. My intention is to just use LIL as a starting point
>>>>> which can be modified as needed to better suit U-Boot.
>>>>>
>>>>> The other half of this is that LIL is not particularly actively
>>>>> developed. I believe the author sees his work as essentially
>>>>> feature-complete, so I expect no major features which we might like to
>>>>> backport.
>>>>
>>>> This sounds like an advantage, indeed, but then you can also
>>>> interpret this as betting on a dead horse...
>>>
>>> My 2c on this.
>>>
>>> I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate).
>>> LIL source code has almost zero comments, poor error checking and no test suite.
>>
>> FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests
>> for LIL" based on the tests included in the LIL distibution. However, I
>> really would like to expand upon it.
>>
>>> I would be very hesitant to adopt it in u-boot without serious work.
>>
>> I think around half of the "serious work" has already been done. I have
>> worked on most of the core of LIL, and added error handling and
>> comments. I believe that most of the remaining instances of dropping
>> errors lie in the built-in commands.
>>
>>> I would much rather see effort put into updating hush to upstream.
>>
>> AIUI hush has diverged significantly from what U-Boot has. This would
>> not be an "update" moreso than a complete port in the style of the
>> current series.
>>
>>> My guess is that Denys would be amenable to small changes to make it easier to synchronise
>>> with busybox in the future.
>>
>> I don't think sh-style shells are a good match for U-Boot's execution
>> environment in the first place. The fundamental idea of an sh-style
>> shell is that the output of one command can be redirected to the input
>> (or arguments) of another command. This cannot be done (or rather would
>> be difficult to do) in U-Boot for a few reasons
>>
>> * U-Boot does not support multithreading. Existing shells tend to depend
>> strongly on this feature of the enviromnent. Many of the changes to
>> U-Boot's hush are solely to deal with the lack of this feature.
>>
>> * Existing commands do not read from stdin, nor do they print useful
>> information to stdout. Command output is designed for human
>> consumption and is substantially more verbose than typical unix
>> commands.
>>
>> * Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
>> useful when working with streams are not present in U-Boot.
>>
>> And of course, this feature is currently not present in U-Boot. To get
>> around this, commands resort to two of my least-favorite hacks: passing
>> in the name of a environmental variable and overloading the return
>> value. For an example of the first, consider
>>
>> => part uuid mmc 0:1 my_uuid
>>
>> which will set my_uuid to the uuid of the selected partition. My issue
>> with this is threefold: every command must add new syntax to do this,
>> that syntax is inconsistent, and it prevents easy composition. Consider
>> a script which wants to iterate over partitions. Instead of doing
>>
>> for p in $(part list mmc 0); do
>> # ...
>> done
>>
>> it must instead do
>>
>> part list mmc 0 partitions
>> for p in $partitions; do
>> # ...
>> done
>>
>> which unnecessarily adds an extra step. This overhead accumulates with
>> each command which adds something like this.
>>
>> The other way to return more information is to use the return value.
>> Consider the button command; it currently returns
>>
>> 0 ON, the button is pressed
>> 1 OFF, the button is released
>> 0 button list was shown
>> 1 button not found
>> 1 invalid arguments
>>
>> and so there is no way to distinguish between whether the button is off,
>> whether the button does not exist, or whether there was a problem with
>> the button driver.
>>
>> Both of these workarounds are natural consequences of using a sh-tyle
>> shell in an environment it is not suited for. If we are going to go to
>> the effort of porting a new language (which must be done no matter if
>> we use Hush or some other language), we should pick one which has better
>> support for single-threaded programming.
>
> I think these are good points, particularly the thing about
> mutiltasking. In fact I think it would be better if hush could
> implement things like '| grep xxx' since it would be useful in U-Boot.
>
> But in any case, adding lil does not preclude someone coming along and
> adaptive hush for U-Boot (and this time upstreaming the changes!). I
> have seen discussions about updating hush for several years and no one
> has done it. I took a quick look and found it was about 3x the size it
> used to be and was not even sure where to start in terms of adapting
> it for single-threaded use.
>
> Re this patch, I think it should be sent upstream.
I believe it will not be accepted, since strdup is POSIX and not C99.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-05 14:42 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
@ 2021-07-05 17:50 ` Wolfgang Denk
2021-07-08 4:37 ` Sean Anderson
1 sibling, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-05 17:50 UTC (permalink / raw)
To: Sean Anderson
Cc: Steve Bennett, Rasmus Villemoes, u-boot, Tom Rini,
Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <5a967151-94f0-6037-2d02-0114c43b846c@gmail.com> you wrote:
>
> AIUI hush has diverged significantly from what U-Boot has. This would
> not be an "update" moreso than a complete port in the style of the
> current series.
Agreed. However, as you write this it sounds like a big problem
only. I disagree here - it is also a chance to do a few things
different than with the original port.
Please keep in mind when this original port of the hush shell was
done: it was a time when many systems came with a total of 4 MB
flash memory, and not only U-Boot, but also the Linux kernel and a
ram-disk image or such had to fit into that. At that time 40 or 60 kB
code size for just a fancy shell was immense!
Under such restrictions (and the need to complete the task with a
given budget) many things were just commented out which from today's
point of view would be nice to have.
This is not a problem of the code complexity or resources, but only
of different requirements and different resources.
> I don't think sh-style shells are a good match for U-Boot's execution
> environment in the first place. The fundamental idea of an sh-style
> shell is that the output of one command can be redirected to the input
> (or arguments) of another command. This cannot be done (or rather would
> be difficult to do) in U-Boot for a few reasons
There is an old saying:
The loser says: "It might be possible, but it is too difficult."
The winner says: "It might be difficult, but it is possible."
Apparently you take another position here than me.
First, I disagree that pipes are the "fundamental idea" of a shell.
It is a fundamental idea of the UNIX operating systems, indeed.
But please keep in mind that we are here in a boot loader, not in a
full-blown OS context.
> * U-Boot does not support multithreading. Existing shells tend to depend
> strongly on this feature of the enviromnent. Many of the changes to
> U-Boot's hush are solely to deal with the lack of this feature.
I disagree to both parts of your statement.
Multithreading (or rather, a concept of separate processes which can
eventually even run in parallel) is very nice to have, but it is not
mandatory in most cases. Command pipes can almost always be
strictly sequentialized and thus run in a single-task environment,
too. I'm not sure, but I think I even remember pipes in the "shell"
of the CPM "OS" on Z80 processors a number of decades ago...
And the fact that our current version of hush did not attempt to
keep these features was much more driven by strict memory footprint
limitations that anything else. I claim it would have been
possible, and still is.
You also don't mention (and I guess you oversee) another critical
fact: so far, U-Boot has no concept of files. The classic design
principle "everything is a file" is missing even more than the
concept of processes.
> * Existing commands do not read from stdin, nor do they print useful
> information to stdout. Command output is designed for human
> consumption and is substantially more verbose than typical unix
> commands.
This is a statement which is correct, but it does not contain any
pro or con for the discussion here.
And if we had the features, it would be easy to fix.
> * Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
> useful when working with streams are not present in U-Boot.
You are talking about OS environments here. But we are a boot
loader.
Also, this argument is not fair, as the suggested LIL does not
provide grep, sed, awk, sort, uniq etc. functionality either, or
does it?
> And of course, this feature is currently not present in U-Boot. To get
Correct, and for very good reasons.
> which will set my_uuid to the uuid of the selected partition. My issue
> with this is threefold: every command must add new syntax to do this,
> that syntax is inconsistent, and it prevents easy composition. Consider
> a script which wants to iterate over partitions. Instead of doing
>
> for p in $(part list mmc 0); do
> # ...
> done
>
> it must instead do
>
> part list mmc 0 partitions
> for p in $partitions; do
> # ...
> done
>
> which unnecessarily adds an extra step. This overhead accumulates with
> each command which adds something like this.
Apparently you fail to understand that this "extra step" is primarily
due to the fact that we are single tasking. Even if we has command
substitution (like we could have with hush) the execution sequence
would be serialized in exactly the same way under the hood - it's
just not as directly visible.
> Both of these workarounds are natural consequences of using a sh-tyle
> shell in an environment it is not suited for. If we are going to go to
No, they are not. They are much more the consequence of no strict
design guidelines for commands, so everybody implements what fits his
purposes best. A cleaner approach does not require any of the
additional features you've asked for - it would be possible as is.
> the effort of porting a new language (which must be done no matter if
> we use Hush or some other language), we should pick one which has better
> support for single-threaded programming.
In which way do you think "a new language" needs to be ported when
switching from an old to a new version of hush? It would be still a
(mostly) POSIX compatible shell, with some restrictions.
I can fully understand that you defend your proposal, but let's be
fair and stick with the facts. Thanks.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
To get something done, a committee should consist of no more than
three men, two of them absent.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-05 17:50 ` Wolfgang Denk
@ 2021-07-08 4:37 ` Sean Anderson
2021-07-08 16:13 ` Wolfgang Denk
0 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-08 4:37 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Steve Bennett, Rasmus Villemoes, u-boot, Tom Rini,
Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/5/21 1:50 PM, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <5a967151-94f0-6037-2d02-0114c43b846c@gmail.com> you wrote:
>>
>> AIUI hush has diverged significantly from what U-Boot has. This would
>> not be an "update" moreso than a complete port in the style of the
>> current series.
>
> Agreed. However, as you write this it sounds like a big problem
> only. I disagree here - it is also a chance to do a few things
> different than with the original port.
>
> Please keep in mind when this original port of the hush shell was
> done: it was a time when many systems came with a total of 4 MB
> flash memory, and not only U-Boot, but also the Linux kernel and a
> ram-disk image or such had to fit into that. At that time 40 or 60 kB
> code size for just a fancy shell was immense!
>
> Under such restrictions (and the need to complete the task with a
> given budget) many things were just commented out which from today's
> point of view would be nice to have.
>
>
> This is not a problem of the code complexity or resources, but only
> of different requirements and different resources.
>
>
>> I don't think sh-style shells are a good match for U-Boot's execution
>> environment in the first place. The fundamental idea of an sh-style
>> shell is that the output of one command can be redirected to the input
>> (or arguments) of another command. This cannot be done (or rather would
>> be difficult to do) in U-Boot for a few reasons
>
> There is an old saying:
>
> The loser says: "It might be possible, but it is too difficult."
> The winner says: "It might be difficult, but it is possible."
And yet, the winner may also say "this is too difficult to be worth it"
instead of laboring inefficiently :)
>
> Apparently you take another position here than me.
>
> First, I disagree that pipes are the "fundamental idea" of a shell.
> It is a fundamental idea of the UNIX operating systems, indeed.
Well, I think all programming languages at their core are about
transferring information from one part of the program to another.
Without pipes, how do bourne-style shells communicate? Sure you can
transfer data into functions using arguments, but what about the other
way around? Numeric return values? Global variables? I think these are
rather anemic compared to proper return values.
> But please keep in mind that we are here in a boot loader, not in a
> full-blown OS context.
>
>> * U-Boot does not support multithreading. Existing shells tend to depend
>> strongly on this feature of the enviromnent. Many of the changes to
>> U-Boot's hush are solely to deal with the lack of this feature.
>
> I disagree to both parts of your statement.
>
> Multithreading (or rather, a concept of separate processes which can
> eventually even run in parallel) is very nice to have, but it is not
> mandatory in most cases. Command pipes can almost always be
> strictly sequentialized and thus run in a single-task environment,
> too. I'm not sure, but I think I even remember pipes in the "shell"
> of the CPM "OS" on Z80 processors a number of decades ago...
The point here is that many Hush features were written with fork. For
example, to create a new scope (such as for a subshell), Hush just
fork()s. When parsing, if it encounters a construct like $(), it fork()s
and then modifies the map variable, (correctly) assuming that the
"original" map will remain unmodified. These things are all over and
make doing a port difficult; especially when they crop up as unforseen
bugs. Though I suppose the real issue here is a lack of virtual memory.
> And the fact that our current version of hush did not attempt to
> keep these features was much more driven by strict memory footprint
> limitations that anything else. I claim it would have been
> possible, and still is.
>
> You also don't mention (and I guess you oversee) another critical
> fact: so far, U-Boot has no concept of files. The classic design
> principle "everything is a file" is missing even more than the
> concept of processes.
IMO "everything is a file" is more of an API thing than a shell thing.
E.g. the idea that you can use open/read/write/close to access any
resource. But without pipes, I don't think such an abstraction is very
useful.
>> * Existing commands do not read from stdin, nor do they print useful
>> information to stdout. Command output is designed for human
>> consumption and is substantially more verbose than typical unix
>> commands.
>
> This is a statement which is correct, but it does not contain any
> pro or con for the discussion here.
In order for pipes to be useful, programs should generally follow the
Unix philosophy. In particular, many commands in U-Boot fall afoul of
rule 2:
> Expect the output of every program to become the input to another, as
> yet unknown, program. Don't clutter output with extraneous
> information. Avoid stringently columnar or binary input formats.
> Don't insist on interactive input.
Which makes it difficult to compose them.
> And if we had the features, it would be easy to fix.
I don't know about that. I think making U-Boot commands emit output
designed for machine consumption would require substantial effort.
>> * Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
>> useful when working with streams are not present in U-Boot.
>
> You are talking about OS environments here. But we are a boot
> loader.
These programs are just as much part of the shell as the builtin
commands. They allow one to take input from one program and transform it
into the appropriate input to another program. This is especially
important for extracting information from programs which were not
designed to interface well.
> Also, this argument is not fair, as the suggested LIL does not
> provide grep, sed, awk, sort, uniq etc. functionality either, or
> does it?
Of course it is fair. These programs are just as much part of the *nix
programming environment as the bourne shells which invoke them.
>
>> And of course, this feature is currently not present in U-Boot. To get
>
> Correct, and for very good reasons.
>
>> which will set my_uuid to the uuid of the selected partition. My issue
>> with this is threefold: every command must add new syntax to do this,
>> that syntax is inconsistent, and it prevents easy composition. Consider
>> a script which wants to iterate over partitions. Instead of doing
>>
>> for p in $(part list mmc 0); do
>> # ...
>> done
>>
>> it must instead do
>>
>> part list mmc 0 partitions
>> for p in $partitions; do
>> # ...
>> done
>>
>> which unnecessarily adds an extra step. This overhead accumulates with
>> each command which adds something like this.
>
> Apparently you fail to understand that this "extra step" is primarily
> due to the fact that we are single tasking. Even if we has command
> substitution (like we could have with hush) the execution sequence
> would be serialized in exactly the same way under the hood - it's
> just not as directly visible.
Please don't assume what I do and do not understand ;)
My point here is that instead of having a consistent interface for
passing data from a command to its caller, we instead have an
inconsistent one which is different for each command.
>> Both of these workarounds are natural consequences of using a sh-tyle
>> shell in an environment it is not suited for. If we are going to go to
>
> No, they are not. They are much more the consequence of no strict
> design guidelines for commands, so everybody implements what fits his
> purposes best. A cleaner approach does not require any of the
> additional features you've asked for - it would be possible as is.
So you would have every command have a new parameter at the end
specifying which environmental variable to put a return value in? IMO
such things should be done at the language level so that commands can
just call a function like set_result() and let the shell handle the
rest.
>> the effort of porting a new language (which must be done no matter if
>> we use Hush or some other language), we should pick one which has better
>> support for single-threaded programming.
>
> In which way do you think "a new language" needs to be ported when
> switching from an old to a new version of hush? It would be still a
> (mostly) POSIX compatible shell, with some restrictions.
Modern Hush is completely unrecognizable compared to what we have in
U-Boot today. Any "updating" effort would be akin to going over the
entire upstream codebase and porting it from scratch.
> I can fully understand that you defend your proposal, but let's be
> fair and stick with the facts. Thanks.
And I'd appreciate if we could presume good faith of our peers.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
2021-07-08 4:37 ` Sean Anderson
@ 2021-07-08 16:13 ` Wolfgang Denk
0 siblings, 0 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-08 16:13 UTC (permalink / raw)
To: Sean Anderson
Cc: Steve Bennett, Rasmus Villemoes, u-boot, Tom Rini,
Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <4aa7ff19-b774-0d67-b96a-9c0c9290a708@gmail.com> you wrote:
>
> Without pipes, how do bourne-style shells communicate? Sure you can
> transfer data into functions using arguments, but what about the other
> way around? Numeric return values? Global variables? I think these are
> rather anemic compared to proper return values.
Pipes are basically just a shortcut to avoid buffer space and to
make scripting easier and more elegant.
For example:
$ ps aux | grep foo | grep -v grep | awk '{print $2}' | xargs kill
can be written without use of pipes as:
$ ps aux >/tmp/$$.1 ; \
grep foo </tmp/$$.1 >/tmp/$$.2 ; \
grep -v grep </tmp/$$.2 >/tmp/$$.1 ; \
awk '{print $2}' </tmp/$$.1 >/tmp/$$.2 ; \
xargs kill </tmp/$$.2 ; \
rm -f /tmp/$$.1 /tmp/$$.2
[and of course this could be done in many other ways, too, cleaner
and more efficient]
The point is: Pipes are just a shortcut. You can do all the same
using files - of course with different performance and resource
usage.
> The point here is that many Hush features were written with fork. For
> example, to create a new scope (such as for a subshell), Hush just
> fork()s. When parsing, if it encounters a construct like $(), it fork()s
Yes, of course I am fully aware of this. This is one of the reasons
why command substitution was excluded from the inital port of hush
to U-Boot - but again: command sustitution is just a shortcut; it is
simple enough to write scripts without it, and it is certainly
possible to implent this feature in a U-Boot context as well.
The reason it was not done 18 years ago is 1) typical systems at
that time did not have the resources (especially not enough ROM for
a bigger U-Boot image), and 2) having a shell with basic scripting
capabilities was sheer extravagance in a boot loader.
Yes, the situation has changed since then; today a U-Boot image is
as big as a v2.2 Linux kernel image was then, and nobody cares.
> and then modifies the map variable, (correctly) assuming that the
> "original" map will remain unmodified. These things are all over and
> make doing a port difficult; especially when they crop up as unforseen
> bugs. Though I suppose the real issue here is a lack of virtual memory.
Not really. At the time of the port the concern was about code size
(size of the text segment).
> IMO "everything is a file" is more of an API thing than a shell thing.
> E.g. the idea that you can use open/read/write/close to access any
> resource. But without pipes, I don't think such an abstraction is very
> useful.
Pipes are just a notation to allow for a convenient use of
coroutines, even from a command line interface. They make you think
of concurrent processing, but (on single core machines - which was
all they had at the time this was invented) it was just an emulation
of concurrency on a strictly sequential system.
Nothing really fancy at all, once you have the idea. And nothing
that could not be emulated in a U-Boot environment, either.
The question is: how urgently do we need it?
My experience is that if anything is _really_ needed, someone will
show up and implement it, probably funded by some customer who needs
this feature for some product. I have yet to see any such urgent
need for pipes in U-Boot.
> > Expect the output of every program to become the input to another, as
> > yet unknown, program. Don't clutter output with extraneous
> > information. Avoid stringently columnar or binary input formats.
That's the theory, yes. In reality, things are a bit different.
Look at the output format of classic Unix commands: ls, ps, df, ...
> I don't know about that. I think making U-Boot commands emit output
> designed for machine consumption would require substantial effort.
Unix commands usually have NOT been designed for "machine
consumption", on contrary. Even network protocols, log files atc.
were intentionally designed to be humanly readable.
It's the flexibility of the tools which allows for clever usage.
In U-Boot, commands could be cleaned up for better use in pipes -
assuming we had those. But there would be no need to do this ina
single huge action - it could be done step by step, one command at
a time when somebody needs to use it in such a way.
> >> * Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely
> >> useful when working with streams are not present in U-Boot.
> >
> > You are talking about OS environments here. But we are a boot
> > loader.
>
> These programs are just as much part of the shell as the builtin
> commands.
Sorry, but they are definitely NOT part of the _shell_.
They are part of the standard Unix toolbox for basic text
processing, but for many good reasons each of these is a separate
program, and the shell knows nothing of these.
> > In which way do you think "a new language" needs to be ported when
> > switching from an old to a new version of hush? It would be still a
> > (mostly) POSIX compatible shell, with some restrictions.
>
> Modern Hush is completely unrecognizable compared to what we have in
> U-Boot today. Any "updating" effort would be akin to going over the
> entire upstream codebase and porting it from scratch.
Agreed. But this means porting some code, which still implements the
very same language (i. e. "shell"). There would be no new language in
this case - just a bigger subset, less restrictions, more options,
probably.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
God made machine language; all the rest is the work of man.
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 04/28] cli: lil: Remove most functions by default
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (2 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 03/28] cli: lil: Replace strclone with strdup Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
2021-07-01 6:15 ` [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL Sean Anderson
` (25 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This helps reduce the size impact of LIL to approximately that of hush. The
builtin functions have been chosen to roughly match the functionality
present in hush. In addition, arithmetic is removed from expr to reduce
size further.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
cmd/Kconfig | 10 ++++++
common/cli_lil.c | 83 +++++++++++++++++++++++++++---------------------
2 files changed, 57 insertions(+), 36 deletions(-)
diff --git a/cmd/Kconfig b/cmd/Kconfig
index 8bccc572af..0a7b73cb6d 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -32,6 +32,16 @@ config LIL
like `proc name {args} {body}' functions or `echo [some command]`
command substitution ("tcl scripts").
+if LIL
+
+config LIL_FULL
+ bool "Enable all LIL features"
+ help
+ This enables all LIL builtin functions, as well as expression support
+ for arithmetic and bitwise operations.
+
+endif
+
config CMDLINE_EDITING
bool "Enable command line editing"
depends on CMDLINE
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 294d79e86e..a2e5cdf35a 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -1321,7 +1321,7 @@ static void ee_unary(struct expreval *ee)
static void ee_muldiv(struct expreval *ee)
{
ee_unary(ee);
- if (ee->error)
+ if (ee->error || !IS_ENABLED(CONFIG_LIL_FULL))
return;
ee_skip_spaces(ee);
@@ -1382,8 +1382,10 @@ static void ee_muldiv(struct expreval *ee)
static void ee_addsub(struct expreval *ee)
{
ee_muldiv(ee);
- ee_skip_spaces(ee);
+ if (!IS_ENABLED(CONFIG_LIL_FULL))
+ return;
+ ee_skip_spaces(ee);
while (ee->head < ee->len && !ee->error &&
!ee_invalidpunct(ee->code[ee->head + 1]) &&
(ee->code[ee->head] == '+' || ee->code[ee->head] == '-')) {
@@ -1415,8 +1417,10 @@ static void ee_addsub(struct expreval *ee)
static void ee_shift(struct expreval *ee)
{
ee_addsub(ee);
- ee_skip_spaces(ee);
+ if (!IS_ENABLED(CONFIG_LIL_FULL))
+ return;
+ ee_skip_spaces(ee);
while (ee->head < ee->len && !ee->error &&
((ee->code[ee->head] == '<' && ee->code[ee->head + 1] == '<') ||
(ee->code[ee->head] == '>' && ee->code[ee->head + 1] == '>'))) {
@@ -1545,8 +1549,10 @@ static void ee_equals(struct expreval *ee)
static void ee_bitand(struct expreval *ee)
{
ee_equals(ee);
- ee_skip_spaces(ee);
+ if (!IS_ENABLED(CONFIG_LIL_FULL))
+ return;
+ ee_skip_spaces(ee);
while (ee->head < ee->len && !ee->error &&
(ee->code[ee->head] == '&' &&
!ee_invalidpunct(ee->code[ee->head + 1]))) {
@@ -1566,8 +1572,10 @@ static void ee_bitand(struct expreval *ee)
static void ee_bitor(struct expreval *ee)
{
ee_bitand(ee);
- ee_skip_spaces(ee);
+ if (!IS_ENABLED(CONFIG_LIL_FULL))
+ return;
+ ee_skip_spaces(ee);
while (ee->head < ee->len && !ee->error &&
(ee->code[ee->head] == '|' &&
!ee_invalidpunct(ee->code[ee->head + 1]))) {
@@ -2932,49 +2940,52 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
static void register_stdcmds(struct lil *lil)
{
- lil_register(lil, "append", fnc_append);
- lil_register(lil, "char", fnc_char);
- lil_register(lil, "charat", fnc_charat);
- lil_register(lil, "codeat", fnc_codeat);
- lil_register(lil, "concat", fnc_concat);
- lil_register(lil, "count", fnc_count);
lil_register(lil, "dec", fnc_dec);
- lil_register(lil, "downeval", fnc_downeval);
- lil_register(lil, "error", fnc_error);
lil_register(lil, "eval", fnc_eval);
lil_register(lil, "expr", fnc_expr);
- lil_register(lil, "filter", fnc_filter);
lil_register(lil, "for", fnc_for);
lil_register(lil, "foreach", fnc_foreach);
lil_register(lil, "func", fnc_func);
lil_register(lil, "if", fnc_if);
lil_register(lil, "inc", fnc_inc);
- lil_register(lil, "index", fnc_index);
- lil_register(lil, "indexof", fnc_indexof);
- lil_register(lil, "length", fnc_length);
- lil_register(lil, "list", fnc_list);
- lil_register(lil, "lmap", fnc_lmap);
lil_register(lil, "local", fnc_local);
- lil_register(lil, "ltrim", fnc_ltrim);
- lil_register(lil, "quote", fnc_quote);
- lil_register(lil, "reflect", fnc_reflect);
- lil_register(lil, "rename", fnc_rename);
- lil_register(lil, "repstr", fnc_repstr);
- lil_register(lil, "result", fnc_result);
lil_register(lil, "return", fnc_return);
- lil_register(lil, "rtrim", fnc_rtrim);
lil_register(lil, "set", fnc_set);
- lil_register(lil, "slice", fnc_slice);
- lil_register(lil, "split", fnc_split);
lil_register(lil, "strcmp", fnc_strcmp);
- lil_register(lil, "streq", fnc_streq);
- lil_register(lil, "strpos", fnc_strpos);
- lil_register(lil, "subst", fnc_subst);
- lil_register(lil, "substr", fnc_substr);
- lil_register(lil, "topeval", fnc_topeval);
- lil_register(lil, "trim", fnc_trim);
lil_register(lil, "try", fnc_try);
- lil_register(lil, "unusedname", fnc_unusedname);
- lil_register(lil, "upeval", fnc_upeval);
lil_register(lil, "while", fnc_while);
+
+ if (IS_ENABLED(CONFIG_LIL_FULL)) {
+ lil_register(lil, "append", fnc_append);
+ lil_register(lil, "char", fnc_char);
+ lil_register(lil, "charat", fnc_charat);
+ lil_register(lil, "codeat", fnc_codeat);
+ lil_register(lil, "concat", fnc_concat);
+ lil_register(lil, "count", fnc_count);
+ lil_register(lil, "downeval", fnc_downeval);
+ lil_register(lil, "error", fnc_error);
+ lil_register(lil, "filter", fnc_filter);
+ lil_register(lil, "index", fnc_index);
+ lil_register(lil, "indexof", fnc_indexof);
+ lil_register(lil, "length", fnc_length);
+ lil_register(lil, "list", fnc_list);
+ lil_register(lil, "lmap", fnc_lmap);
+ lil_register(lil, "ltrim", fnc_ltrim);
+ lil_register(lil, "quote", fnc_quote);
+ lil_register(lil, "reflect", fnc_reflect);
+ lil_register(lil, "rename", fnc_rename);
+ lil_register(lil, "repstr", fnc_repstr);
+ lil_register(lil, "result", fnc_result);
+ lil_register(lil, "rtrim", fnc_rtrim);
+ lil_register(lil, "slice", fnc_slice);
+ lil_register(lil, "split", fnc_split);
+ lil_register(lil, "streq", fnc_streq);
+ lil_register(lil, "strpos", fnc_strpos);
+ lil_register(lil, "subst", fnc_subst);
+ lil_register(lil, "substr", fnc_substr);
+ lil_register(lil, "topeval", fnc_topeval);
+ lil_register(lil, "trim", fnc_trim);
+ lil_register(lil, "unusedname", fnc_unusedname);
+ lil_register(lil, "upeval", fnc_upeval);
+ }
}
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 04/28] cli: lil: Remove most functions by default
2021-07-01 6:15 ` [RFC PATCH 04/28] cli: lil: Remove most functions by default Sean Anderson
@ 2021-07-05 15:29 ` Simon Glass
0 siblings, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>
> This helps reduce the size impact of LIL to approximately that of hush. The
> builtin functions have been chosen to roughly match the functionality
> present in hush. In addition, arithmetic is removed from expr to reduce
> size further.
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
> ---
>
> cmd/Kconfig | 10 ++++++
> common/cli_lil.c | 83 +++++++++++++++++++++++++++---------------------
> 2 files changed, 57 insertions(+), 36 deletions(-)
Reviewed-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (3 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 04/28] cli: lil: Remove most functions by default Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
2021-07-01 6:15 ` [RFC PATCH 06/28] cli: lil: Convert some defines to enums Sean Anderson
` (24 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
Several functions have different names than they do in TCL. To make things
easier for those familiar with TCL, rename them to their TCL equivalents.
At the moment, this is only done for functions not used by LIL_FULL. Some
functions need more substantive work to conform them to TCL. For example,
in TCL, there is a `string` function with a subcommand of `compare`, which
is the same as the top-level function `compare`. Several functions also
have no TCL equivalent. Do we need these?
TODO: do this for all functions
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index a2e5cdf35a..a6c77ee19c 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -1829,10 +1829,10 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
return lil_clone_value(func->code);
}
- if (!strcmp(type, "func-count"))
+ if (!strcmp(type, "proc-count"))
return lil_alloc_integer(lil->cmds);
- if (!strcmp(type, "funcs")) {
+ if (!strcmp(type, "procs")) {
struct lil_list *funcs = lil_alloc_list();
for (i = 0; i < lil->cmds; i++)
@@ -1878,7 +1878,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
return r;
}
- if (!strcmp(type, "has-func")) {
+ if (!strcmp(type, "has-proc")) {
const char *target;
if (argc == 1)
@@ -1948,7 +1948,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
return NULL;
}
-static struct lil_value *fnc_func(struct lil *lil, size_t argc,
+static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
struct lil_value **argv)
{
struct lil_value *name;
@@ -2472,8 +2472,8 @@ static struct lil_value *real_inc(struct lil *lil, const char *varname,
return pv;
}
-static struct lil_value *fnc_inc(struct lil *lil, size_t argc,
- struct lil_value **argv)
+static struct lil_value *fnc_incr(struct lil *lil, size_t argc,
+ struct lil_value **argv)
{
if (argc < 1)
return NULL;
@@ -2482,8 +2482,8 @@ static struct lil_value *fnc_inc(struct lil *lil, size_t argc,
argc > 1 ? lil_to_integer(argv[1]) : 1);
}
-static struct lil_value *fnc_dec(struct lil *lil, size_t argc,
- struct lil_value **argv)
+static struct lil_value *fnc_decr(struct lil *lil, size_t argc,
+ struct lil_value **argv)
{
if (argc < 1)
return NULL;
@@ -2786,8 +2786,8 @@ static struct lil_value *fnc_rtrim(struct lil *lil, size_t argc,
return real_trim(lil_to_string(argv[0]), chars, 0, 1);
}
-static struct lil_value *fnc_strcmp(struct lil *lil, size_t argc,
- struct lil_value **argv)
+static struct lil_value *fnc_compare(struct lil *lil, size_t argc,
+ struct lil_value **argv)
{
if (argc < 2)
return NULL;
@@ -2940,18 +2940,18 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
static void register_stdcmds(struct lil *lil)
{
- lil_register(lil, "dec", fnc_dec);
+ lil_register(lil, "decr", fnc_decr);
lil_register(lil, "eval", fnc_eval);
lil_register(lil, "expr", fnc_expr);
lil_register(lil, "for", fnc_for);
lil_register(lil, "foreach", fnc_foreach);
- lil_register(lil, "func", fnc_func);
+ lil_register(lil, "proc", fnc_proc);
lil_register(lil, "if", fnc_if);
- lil_register(lil, "inc", fnc_inc);
+ lil_register(lil, "incr", fnc_incr);
lil_register(lil, "local", fnc_local);
lil_register(lil, "return", fnc_return);
lil_register(lil, "set", fnc_set);
- lil_register(lil, "strcmp", fnc_strcmp);
+ lil_register(lil, "compare", fnc_compare);
lil_register(lil, "try", fnc_try);
lil_register(lil, "while", fnc_while);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-01 6:15 ` [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL Sean Anderson
@ 2021-07-05 15:29 ` Simon Glass
2021-07-05 15:54 ` Sean Anderson
0 siblings, 1 reply; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Hi Sean,
On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>
> Several functions have different names than they do in TCL. To make things
> easier for those familiar with TCL, rename them to their TCL equivalents.
> At the moment, this is only done for functions not used by LIL_FULL. Some
> functions need more substantive work to conform them to TCL. For example,
> in TCL, there is a `string` function with a subcommand of `compare`, which
> is the same as the top-level function `compare`. Several functions also
> have no TCL equivalent. Do we need these?
>
> TODO: do this for all functions
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
> ---
>
> common/cli_lil.c | 28 ++++++++++++++--------------
> 1 file changed, 14 insertions(+), 14 deletions(-)
Is your intent to create a fork of this in U-Boot? Could we not update
things upstream, at least as an option, to avoid carrying these
patches?
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 15:29 ` Simon Glass
@ 2021-07-05 15:54 ` Sean Anderson
2021-07-05 17:58 ` Wolfgang Denk
0 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-05 15:54 UTC (permalink / raw)
To: Simon Glass
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On 7/5/21 11:29 AM, Simon Glass wrote:
> Hi Sean,
>
> On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>>
>> Several functions have different names than they do in TCL. To make things
>> easier for those familiar with TCL, rename them to their TCL equivalents.
>> At the moment, this is only done for functions not used by LIL_FULL. Some
>> functions need more substantive work to conform them to TCL. For example,
>> in TCL, there is a `string` function with a subcommand of `compare`, which
>> is the same as the top-level function `compare`. Several functions also
>> have no TCL equivalent. Do we need these?
>>
>> TODO: do this for all functions
>>
>> Signed-off-by: Sean Anderson <seanga2@gmail.com>
>> ---
>>
>> common/cli_lil.c | 28 ++++++++++++++--------------
>> 1 file changed, 14 insertions(+), 14 deletions(-)
>
> Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC
PATCH 21/28] cli: lil: Add a distinct parsing step") would not be
accepted by upstream.
> Could we not update things upstream, at least as an option, to avoid
> carrying these patches?
For some of the smaller patches, that may be possible. However, I am not
a fan of the major amount of ifdefs that Hush has. For something as core
as a shell, I think we should be free to make changes as we see fit
without worrying about how it will affect a hypothetical backport.
For this patch in particular, I believe upstream would no accept it
because it would break backwards compatibility for existing LIL users.
However, I view compatibility with TCL as a whole more valuble than
compatibility with LIL.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 15:54 ` Sean Anderson
@ 2021-07-05 17:58 ` Wolfgang Denk
2021-07-05 18:51 ` Tom Rini
2021-07-05 19:46 ` Sean Anderson
0 siblings, 2 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-05 17:58 UTC (permalink / raw)
To: Sean Anderson
Cc: Simon Glass, U-Boot Mailing List, Tom Rini, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <d808990b-623d-d962-c7d6-e40063bc5dab@gmail.com> you wrote:
>
> > Is your intent to create a fork of this in U-Boot?
>
> Yes. I believe some of the major additions I have made (especially "[RFC
> PATCH 21/28] cli: lil: Add a distinct parsing step") would not be
> accepted by upstream.
Ouch...
> > Could we not update things upstream, at least as an option, to avoid
> > carrying these patches?
>
> For some of the smaller patches, that may be possible. However, I am not
> a fan of the major amount of ifdefs that Hush has. For something as core
> as a shell, I think we should be free to make changes as we see fit
> without worrying about how it will affect a hypothetical backport.
I'm afraind I cannot understand your thinking.
You complain that the existing port of hus has a number of severe
limitations or bugs which have long been fixed upstream, but cannot
be easily fixed in U-Boot because we essentially created an
unmaintained fork - and as a cure, you recommend to do the same
thing again, but this time intentionally and deliberately?
If you had not apparently already invested a lot of effort into this
thing I would assume you must be joking...
To me such an approach is unacceptable.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
If a train station is a place where a train stops,
then what's a workstation?
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 17:58 ` Wolfgang Denk
@ 2021-07-05 18:51 ` Tom Rini
2021-07-05 21:02 ` Simon Glass
2021-07-06 7:52 ` Wolfgang Denk
2021-07-05 19:46 ` Sean Anderson
1 sibling, 2 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-05 18:51 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, Simon Glass, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]
On Mon, Jul 05, 2021 at 07:58:18PM +0200, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <d808990b-623d-d962-c7d6-e40063bc5dab@gmail.com> you wrote:
> >
> > > Is your intent to create a fork of this in U-Boot?
> >
> > Yes. I believe some of the major additions I have made (especially "[RFC
> > PATCH 21/28] cli: lil: Add a distinct parsing step") would not be
> > accepted by upstream.
>
> Ouch...
>
> > > Could we not update things upstream, at least as an option, to avoid
> > > carrying these patches?
> >
> > For some of the smaller patches, that may be possible. However, I am not
> > a fan of the major amount of ifdefs that Hush has. For something as core
> > as a shell, I think we should be free to make changes as we see fit
> > without worrying about how it will affect a hypothetical backport.
>
> I'm afraind I cannot understand your thinking.
>
> You complain that the existing port of hus has a number of severe
> limitations or bugs which have long been fixed upstream, but cannot
> be easily fixed in U-Boot because we essentially created an
> unmaintained fork - and as a cure, you recommend to do the same
> thing again, but this time intentionally and deliberately?
>
>
> If you had not apparently already invested a lot of effort into this
> thing I would assume you must be joking...
>
> To me such an approach is unacceptable.
I think I want to try and address this. While with "hush" we have
something that's in heavy active development outside of U-Boot, with LIL
we have something that's mature and "done". Tracking an active outside
development is HARD and requires constant resync. Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed.
I see that as a positive, not a negative.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 18:51 ` Tom Rini
@ 2021-07-05 21:02 ` Simon Glass
2021-07-05 21:36 ` Tom Rini
2021-07-06 7:52 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Simon Glass @ 2021-07-05 21:02 UTC (permalink / raw)
To: Tom Rini
Cc: Wolfgang Denk, Sean Anderson, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Hi Tom,
On Mon, 5 Jul 2021 at 12:51, Tom Rini <trini@konsulko.com> wrote:
>
> On Mon, Jul 05, 2021 at 07:58:18PM +0200, Wolfgang Denk wrote:
> > Dear Sean,
> >
> > In message <d808990b-623d-d962-c7d6-e40063bc5dab@gmail.com> you wrote:
> > >
> > > > Is your intent to create a fork of this in U-Boot?
> > >
> > > Yes. I believe some of the major additions I have made (especially "[RFC
> > > PATCH 21/28] cli: lil: Add a distinct parsing step") would not be
> > > accepted by upstream.
> >
> > Ouch...
> >
> > > > Could we not update things upstream, at least as an option, to avoid
> > > > carrying these patches?
> > >
> > > For some of the smaller patches, that may be possible. However, I am not
> > > a fan of the major amount of ifdefs that Hush has. For something as core
> > > as a shell, I think we should be free to make changes as we see fit
> > > without worrying about how it will affect a hypothetical backport.
> >
> > I'm afraind I cannot understand your thinking.
> >
> > You complain that the existing port of hus has a number of severe
> > limitations or bugs which have long been fixed upstream, but cannot
> > be easily fixed in U-Boot because we essentially created an
> > unmaintained fork - and as a cure, you recommend to do the same
> > thing again, but this time intentionally and deliberately?
> >
> >
> > If you had not apparently already invested a lot of effort into this
> > thing I would assume you must be joking...
> >
> > To me such an approach is unacceptable.
>
> I think I want to try and address this. While with "hush" we have
> something that's in heavy active development outside of U-Boot, with LIL
> we have something that's mature and "done". Tracking an active outside
> development is HARD and requires constant resync. Look at the last few
> LIL releases. That could be easily re-worked in to our fork if needed.
> I see that as a positive, not a negative.
Yes I wondered about that, since hush has been in busybox, an active
project, all these years. I think this is a good point and perhaps
means that forking it should not be too much of a concern. Still, if
we can send some things upstream, we should.
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 21:02 ` Simon Glass
@ 2021-07-05 21:36 ` Tom Rini
0 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-05 21:36 UTC (permalink / raw)
To: Simon Glass
Cc: Wolfgang Denk, Sean Anderson, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 2585 bytes --]
On Mon, Jul 05, 2021 at 03:02:24PM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Mon, 5 Jul 2021 at 12:51, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Mon, Jul 05, 2021 at 07:58:18PM +0200, Wolfgang Denk wrote:
> > > Dear Sean,
> > >
> > > In message <d808990b-623d-d962-c7d6-e40063bc5dab@gmail.com> you wrote:
> > > >
> > > > > Is your intent to create a fork of this in U-Boot?
> > > >
> > > > Yes. I believe some of the major additions I have made (especially "[RFC
> > > > PATCH 21/28] cli: lil: Add a distinct parsing step") would not be
> > > > accepted by upstream.
> > >
> > > Ouch...
> > >
> > > > > Could we not update things upstream, at least as an option, to avoid
> > > > > carrying these patches?
> > > >
> > > > For some of the smaller patches, that may be possible. However, I am not
> > > > a fan of the major amount of ifdefs that Hush has. For something as core
> > > > as a shell, I think we should be free to make changes as we see fit
> > > > without worrying about how it will affect a hypothetical backport.
> > >
> > > I'm afraind I cannot understand your thinking.
> > >
> > > You complain that the existing port of hus has a number of severe
> > > limitations or bugs which have long been fixed upstream, but cannot
> > > be easily fixed in U-Boot because we essentially created an
> > > unmaintained fork - and as a cure, you recommend to do the same
> > > thing again, but this time intentionally and deliberately?
> > >
> > >
> > > If you had not apparently already invested a lot of effort into this
> > > thing I would assume you must be joking...
> > >
> > > To me such an approach is unacceptable.
> >
> > I think I want to try and address this. While with "hush" we have
> > something that's in heavy active development outside of U-Boot, with LIL
> > we have something that's mature and "done". Tracking an active outside
> > development is HARD and requires constant resync. Look at the last few
> > LIL releases. That could be easily re-worked in to our fork if needed.
> > I see that as a positive, not a negative.
>
> Yes I wondered about that, since hush has been in busybox, an active
> project, all these years. I think this is a good point and perhaps
> means that forking it should not be too much of a concern. Still, if
> we can send some things upstream, we should.
Well, that's part of the problem. If we re-port modern hush, who is
going to keep it in sync, and how often? Yes, there will be changes we
don't need, but there will be bugfixes we do as well.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 18:51 ` Tom Rini
2021-07-05 21:02 ` Simon Glass
@ 2021-07-06 7:52 ` Wolfgang Denk
2021-07-06 15:21 ` Simon Glass
2021-07-06 15:33 ` Tom Rini
1 sibling, 2 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-06 7:52 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, Simon Glass, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Dear Tom,
In message <20210705185141.GA9516@bill-the-cat> you wrote:
>
> I think I want to try and address this. While with "hush" we have
> something that's in heavy active development outside of U-Boot, with LIL
> we have something that's mature and "done".
Mature? And still without consequent error checking? And done,
i. e. this will never be fixed?
> Tracking an active outside development is HARD and requires
> constant resync.
Based on that logic we should stop supporting Linux, and stop using
DTs or file systems - all of these are under "active outside
development".
This is a bogus argiment.
> Look at the last few
> LIL releases. That could be easily re-worked in to our fork if needed.
> I see that as a positive, not a negative.
OK, this is your opinion. Mine differs.
Please consider this as a full NAK from me when when you think of it
as a _replacement_ (even an optional one) of the standard shell. If
you like, have it added as an _additional_ command, of course fully
optional and without impact on the rest of U-Boot if not
intentionally selected.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Of all the things I've lost, I miss my mind the most.
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-06 7:52 ` Wolfgang Denk
@ 2021-07-06 15:21 ` Simon Glass
2021-07-06 15:33 ` Tom Rini
1 sibling, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-06 15:21 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Tom Rini, Sean Anderson, U-Boot Mailing List, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Hi Wolfgang,
On Tue, 6 Jul 2021 at 01:53, Wolfgang Denk <wd@denx.de> wrote:
>
> Dear Tom,
>
> In message <20210705185141.GA9516@bill-the-cat> you wrote:
> >
> > I think I want to try and address this. While with "hush" we have
> > something that's in heavy active development outside of U-Boot, with LIL
> > we have something that's mature and "done".
>
> Mature? And still without consequent error checking? And done,
> i. e. this will never be fixed?
>
>
> > Tracking an active outside development is HARD and requires
> > constant resync.
>
> Based on that logic we should stop supporting Linux, and stop using
> DTs or file systems - all of these are under "active outside
> development".
>
> This is a bogus argiment.
>
>
> > Look at the last few
> > LIL releases. That could be easily re-worked in to our fork if needed.
> > I see that as a positive, not a negative.
>
> OK, this is your opinion. Mine differs.
>
> Please consider this as a full NAK from me when when you think of it
> as a _replacement_ (even an optional one) of the standard shell. If
> you like, have it added as an _additional_ command, of course fully
> optional and without impact on the rest of U-Boot if not
> intentionally selected.
That seems reasonable to me. There is certainly interest in these
threads in reporting hush, perhaps with a goal of upstreaming the
U-Boot-specific changes back to busybox. But so far no one has done
it...
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-06 7:52 ` Wolfgang Denk
2021-07-06 15:21 ` Simon Glass
@ 2021-07-06 15:33 ` Tom Rini
2021-07-06 16:00 ` Kostas Michalopoulos
2021-07-07 8:16 ` Wolfgang Denk
1 sibling, 2 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-06 15:33 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, Simon Glass, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 3084 bytes --]
On Tue, Jul 06, 2021 at 09:52:56AM +0200, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210705185141.GA9516@bill-the-cat> you wrote:
> >
> > I think I want to try and address this. While with "hush" we have
> > something that's in heavy active development outside of U-Boot, with LIL
> > we have something that's mature and "done".
>
> Mature? And still without consequent error checking? And done,
> i. e. this will never be fixed?
Intentional design by upstream, and then for the actual problem part
(error checking, test suite), Sean is saying he'll fix it, and has
started on it.
> > Tracking an active outside development is HARD and requires
> > constant resync.
>
> Based on that logic we should stop supporting Linux, and stop using
> DTs or file systems - all of these are under "active outside
> development".
>
> This is a bogus argiment.
Alright, go and update our Kbuild integration to be in sync with current
Linux. Or just v5.0. I'd suggest picking up the metadata_csum support
patch but someone did say they'd try and update that (off-list) so that
ext4 filesystem made with default options in the last 5 years (not
exaggerating) work. I'm not even going to talk about the various level
of out of sync our DTs files are but I do have some hope there will be
more re-syncs especially as it comes to more light that the DT U-Boot
uses can just be the DT the OS gets. Then tell me it's easy to keep
this stuff in sync. Because we certainly don't want to just
fork-and-forget again, right? Because that's why we have both crazy
parse bugs as well as lack of now old-and-expected features.
OK, snark aside, I'm very serious here, any "we'll just import ..."
needs to have a plan to keep it up to date, or be easy enough to do such
that I can set a monthly reminder to check for and do the update. Every
area where we don't do this is a set of problems waiting to get worse,
as we can see with the hush shell right now as it's one of the oldest
things we stopped syncing with.
> > Look at the last few
> > LIL releases. That could be easily re-worked in to our fork if needed.
> > I see that as a positive, not a negative.
>
> OK, this is your opinion. Mine differs.
>
> Please consider this as a full NAK from me when when you think of it
> as a _replacement_ (even an optional one) of the standard shell. If
> you like, have it added as an _additional_ command, of course fully
> optional and without impact on the rest of U-Boot if not
> intentionally selected.
Any new cli for U-Boot won't be a default build as a command for a long
while after it's merged. I'm not sure how much past that point it would
further need to be to become the default cli. And I'll even repeat what
I said elsewhere about how an sh syntax is a good user feature. But I
really think we want a shell environment that is not actively adding new
features is a good thing, for the default. Just how much stuff should
we be doing or need to be doing before we hand things over to the OS?
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-06 15:33 ` Tom Rini
@ 2021-07-06 16:00 ` Kostas Michalopoulos
2021-07-07 8:16 ` Wolfgang Denk
1 sibling, 0 replies; 107+ messages in thread
From: Kostas Michalopoulos @ 2021-07-06 16:00 UTC (permalink / raw)
To: Tom Rini, Wolfgang Denk
Cc: Sean Anderson, Simon Glass, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt
On 7/6/2021 6:33 PM, Tom Rini wrote:
>> Mature? And still without consequent error checking? And done,
>> i. e. this will never be fixed?
>
> Intentional design by upstream, and then for the actual problem part
> (error checking, test suite), Sean is saying he'll fix it, and has
> started on it.
To clarify, the intentional design aspect is for the language (e.g.
giving garbage to "expr"). The library does need better memory alloc
checks (this is the main thing missing because i never had to use LIL on
an environment without virtual memory... or where malloc would ever
practically fail), though IIRC there aren't any other cases where
failure could happen and isn't checked. Similarly i do plan on
implementing a proper test suite (currently there is a shell script that
runs all the examples in the source code - many examples were made to
expose previous bugs - with two different implementations which can be
used to find regressions in subsequent releases and differences between
implementations, but the examples do not exhaust the code paths the
parser could take) at some point - the latter is more likely to happen
soon than the former.
The same applies with LIL being "done" (well, mostly done) - this is
about the language, not the implementation which (as i mentioned in
another email) does need improvements, especially around performance
(which is also a reason i need to implement better automated tests).
Kostas
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-06 15:33 ` Tom Rini
2021-07-06 16:00 ` Kostas Michalopoulos
@ 2021-07-07 8:16 ` Wolfgang Denk
2021-07-07 13:58 ` Tom Rini
1 sibling, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-07 8:16 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, Simon Glass, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
Dear Tom,
In message <20210706153327.GS9516@bill-the-cat> you wrote:
>
> > Mature? And still without consequent error checking? And done,
> > i. e. this will never be fixed?
>
> Intentional design by upstream, and then for the actual problem part
> (error checking, test suite), Sean is saying he'll fix it, and has
> started on it.
Seriously - any piece of software that omits error checking
intentionally be design should be indented six feet downward and
covered with dirt. We should not even consider looking at it.
> OK, snark aside, I'm very serious here, any "we'll just import ..."
> needs to have a plan to keep it up to date, or be easy enough to do such
> that I can set a monthly reminder to check for and do the update. Every
> area where we don't do this is a set of problems waiting to get worse,
> as we can see with the hush shell right now as it's one of the oldest
> things we stopped syncing with.
Which exact _new_ problems do we see with hush right now? I can
only see old ones, that have been known (and worked around) for
nearly two decades.
The limitations and bugs have all been there since the beginning -
the limitations actually being intentional due to the typical
resource situation at that time.
> But I
> really think we want a shell environment that is not actively adding new
> features is a good thing, for the default. Just how much stuff should
> we be doing or need to be doing before we hand things over to the OS?
You are shooting yourself in the knee here.
If you think out CLI should not be adding new features, then we
should just stick with our ancient hush and neither update it nor
replace it with something else that adds not only new features, but
breaks backward compatibility, hard.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
God runs electromagnetics by wave theory on Monday, Wednesday, and
Friday, and the Devil runs them by quantum theory on Tuesday, Thurs-
day, and Saturday. -- William Bragg
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-07 8:16 ` Wolfgang Denk
@ 2021-07-07 13:58 ` Tom Rini
0 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-07 13:58 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Sean Anderson, Simon Glass, U-Boot Mailing List,
Marek Behún, Roland Gaudig, Heinrich Schuchardt,
Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 3196 bytes --]
On Wed, Jul 07, 2021 at 10:16:13AM +0200, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210706153327.GS9516@bill-the-cat> you wrote:
> >
> > > Mature? And still without consequent error checking? And done,
> > > i. e. this will never be fixed?
> >
> > Intentional design by upstream, and then for the actual problem part
> > (error checking, test suite), Sean is saying he'll fix it, and has
> > started on it.
>
> Seriously - any piece of software that omits error checking
> intentionally be design should be indented six feet downward and
> covered with dirt. We should not even consider looking at it.
>
> > OK, snark aside, I'm very serious here, any "we'll just import ..."
> > needs to have a plan to keep it up to date, or be easy enough to do such
> > that I can set a monthly reminder to check for and do the update. Every
> > area where we don't do this is a set of problems waiting to get worse,
> > as we can see with the hush shell right now as it's one of the oldest
> > things we stopped syncing with.
>
> Which exact _new_ problems do we see with hush right now? I can
> only see old ones, that have been known (and worked around) for
> nearly two decades.
>
> The limitations and bugs have all been there since the beginning -
> the limitations actually being intentional due to the typical
> resource situation at that time.
There's all of the new features that've been suggested for our current
hush, many by Sean, for which the reply has been "our hush is old, it
should be updated!".
And you didn't address my point, taking code from another project
requires dedicated maintenance to keep it up to date and we do have a
lot of in-tree examples right now of where that lack of dedicated
maintenance is a problem.
> > But I
> > really think we want a shell environment that is not actively adding new
> > features is a good thing, for the default. Just how much stuff should
> > we be doing or need to be doing before we hand things over to the OS?
>
> You are shooting yourself in the knee here.
>
> If you think out CLI should not be adding new features, then we
> should just stick with our ancient hush and neither update it nor
> replace it with something else that adds not only new features, but
> breaks backward compatibility, hard.
Honestly, adding new features to the CLI (which is NOT the same as
adding new commands, or enhancing existing commands) is very low on my
priority list. We need to get the Kconfig migration done. We need to
get DM migration done (which will help with the Kconfig one by showing
hardware no one cares for anymore, and reducing symbols). We need to
make adding new hardware easier. We need to get DTS files in-sync more
often.
There are very important developer use cases for a more flexible CLI
interpreter. But the main use case is still "boot the (redundant) OS
ASAP".
So, what is my priority on this series right here? Well, a developer
has posted a series. They believe it's useful and addressing a
long-standing problem. They want feedback. I'm trying to provide
feedback as it's to a general area of the codebase.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 17:58 ` Wolfgang Denk
2021-07-05 18:51 ` Tom Rini
@ 2021-07-05 19:46 ` Sean Anderson
2021-07-06 7:50 ` Wolfgang Denk
1 sibling, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-05 19:46 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Simon Glass, U-Boot Mailing List, Tom Rini, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On 7/5/21 1:58 PM, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <d808990b-623d-d962-c7d6-e40063bc5dab@gmail.com> you wrote:
>>
>>> Is your intent to create a fork of this in U-Boot?
>>
>> Yes. I believe some of the major additions I have made (especially "[RFC
>> PATCH 21/28] cli: lil: Add a distinct parsing step") would not be
>> accepted by upstream.
>
> Ouch...
I don't particularly mind if Kostas doesn't view these patches as good
additions to upstream. We have different goals and requirements, and so
not all changes will be compatible.
>>> Could we not update things upstream, at least as an option, to avoid
>>> carrying these patches?
>>
>> For some of the smaller patches, that may be possible. However, I am not
>> a fan of the major amount of ifdefs that Hush has. For something as core
>> as a shell, I think we should be free to make changes as we see fit
>> without worrying about how it will affect a hypothetical backport.
>
> I'm afraind I cannot understand your thinking.
>
> You complain that the existing port of hus has a number of severe
> limitations or bugs which have long been fixed upstream,
The bugs are fairly minor. The particular characteristics of Hush have
not changed. These characteristics make Hush difficult to adapt to the
limitations of U-Boot. When we cannot support the basic abstractions
expected by Hush, the shell will necessarily change for the worse.
> but cannot be easily fixed in U-Boot
Because they are core to the design of Hush (and other bourne derived
shells).
> because we essentially created an unmaintained fork
I plan to maintain this fork.
--Sean
> - and as a cure, you recommend to do the same
> thing again, but this time intentionally and deliberately?
> If you had not apparently already invested a lot of effort into this
> thing I would assume you must be joking...
>
> To me such an approach is unacceptable.
>
> Best regards,
>
> Wolfgang Denk
>
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-05 19:46 ` Sean Anderson
@ 2021-07-06 7:50 ` Wolfgang Denk
2021-07-08 4:47 ` Sean Anderson
0 siblings, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-06 7:50 UTC (permalink / raw)
To: Sean Anderson
Cc: Simon Glass, U-Boot Mailing List, Tom Rini, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <7143cb1e-4061-3034-57b9-1a12753fa642@gmail.com> you wrote:
> >
> > You complain that the existing port of hus has a number of severe
> > limitations or bugs which have long been fixed upstream,
>
> The bugs are fairly minor. The particular characteristics of Hush have
> not changed. These characteristics make Hush difficult to adapt to the
> limitations of U-Boot. When we cannot support the basic abstractions
> expected by Hush, the shell will necessarily change for the worse.
This is not true. Just have a look what hush in a recent version of
Busybox offers.
> > but cannot be easily fixed in U-Boot
>
> Because they are core to the design of Hush (and other bourne derived
> shells).
Oh, this is an interesting opinion. I doubt if a majority (or even
a significant percentage) of U-Boot users share it. If you were
right, there would be far less users of bash (or other "bourne
derived shells"). Guess which percentage of users of UNIX operating
systems is using a Tcl based command interpreter as their login
shell?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"There are three principal ways to lose money: wine, women, and engi-
neers. While the first two are more pleasant, the third is by far the
more certain." - Baron Rothschild, ca. 1800
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-06 7:50 ` Wolfgang Denk
@ 2021-07-08 4:47 ` Sean Anderson
2021-07-08 16:21 ` Wolfgang Denk
0 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-08 4:47 UTC (permalink / raw)
To: Wolfgang Denk
Cc: Simon Glass, U-Boot Mailing List, Tom Rini, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On 7/6/21 3:50 AM, Wolfgang Denk wrote:
> Dear Sean,
>
> In message <7143cb1e-4061-3034-57b9-1a12753fa642@gmail.com> you wrote:
>>>
>>> You complain that the existing port of hus has a number of severe
>>> limitations or bugs which have long been fixed upstream,
>>
>> The bugs are fairly minor. The particular characteristics of Hush have
>> not changed. These characteristics make Hush difficult to adapt to the
>> limitations of U-Boot. When we cannot support the basic abstractions
>> expected by Hush, the shell will necessarily change for the worse.
>
> This is not true. Just have a look what hush in a recent version of
> Busybox offers.
Busybox runs in a Linux environment. Many of its features rely on the
core functionality provided by Linux, which we do not provide in U-Boot.
This is what makes porting features difficult.
>
>>> but cannot be easily fixed in U-Boot
>>
>> Because they are core to the design of Hush (and other bourne derived
>> shells).
>
> Oh, this is an interesting opinion. I doubt if a majority (or even
> a significant percentage) of U-Boot users share it. If you were
> right, there would be far less users of bash (or other "bourne
> derived shells"). Guess which percentage of users of UNIX operating
> systems is using a Tcl based command interpreter as their login
> shell?
And yet, this is not the field we compete in. While bourne-style shells
can take advantage of a multi-threaded environment, embedded shells tend
to implement a much wider set of languages. See [1] for a survey of
examples.
--Sean
[1] https://github.com/dbohdan/embedded-scripting-languages
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
2021-07-08 4:47 ` Sean Anderson
@ 2021-07-08 16:21 ` Wolfgang Denk
0 siblings, 0 replies; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-08 16:21 UTC (permalink / raw)
To: Sean Anderson
Cc: Simon Glass, U-Boot Mailing List, Tom Rini, Marek Behún,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Sean,
In message <d8447d79-f1d4-8a2f-39b9-a6650c936b88@gmail.com> you wrote:
>
...
> And yet, this is not the field we compete in. While bourne-style shells
> can take advantage of a multi-threaded environment, embedded shells tend
> to implement a much wider set of languages. See [1] for a survey of
> examples.
>
> [1] https://github.com/dbohdan/embedded-scripting-languages
"Embedded scripting languages" is probably not a good match for what
we are discussing. I don't think this comparison was made for
restricted boot loader environments, but for use in general purpose
/ embedded operating systems.
Hey, they even list Forth there. Maybe somebody should port
OpenBoot, so we can have a Forth interpreter as new commandline
language? We could implement device trees in the old, traditional
way then, too :-)
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
There are bugs and then there are bugs. And then there are bugs.
- Karl Lehenbauer
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 06/28] cli: lil: Convert some defines to enums
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (4 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 07/28] cli: lil: Simplify callbacks Sean Anderson
` (23 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This converts some defines to enums. This allows them to be documented more
easily, and helps the compiler give better errors for switch statements.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 44 +++++++++++++++++++++++---------------------
include/cli_lil.h | 12 +++++++-----
2 files changed, 30 insertions(+), 26 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index a6c77ee19c..3d1e6181f9 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -24,10 +24,6 @@
* overflows and is also useful when running through an automated fuzzer like AFL */
/*#define LIL_ENABLE_RECLIMIT 10000*/
-#define ERROR_NOERROR 0
-#define ERROR_DEFAULT 1
-#define ERROR_FIXHEAD 2
-
#define CALLBACKS 8
#define HASHMAP_CELLS 256
#define HASHMAP_CELLMASK 0xFF
@@ -97,7 +93,11 @@ struct lil {
struct lil_env *rootenv;
struct lil_env *downenv;
struct lil_value *empty;
- int error;
+ enum {
+ ERROR_NOERROR = 0,
+ ERROR_DEFAULT,
+ ERROR_FIXHEAD,
+ } error;
size_t err_head;
char *err_msg;
lil_callback_proc_t callback[CALLBACKS];
@@ -108,7 +108,12 @@ struct expreval {
const char *code;
size_t len, head;
ssize_t ival;
- int error;
+ enum {
+ EERR_NO_ERROR = 0,
+ EERR_SYNTAX_ERROR,
+ EERR_DIVISION_BY_ZERO,
+ EERR_INVALID_EXPRESSION,
+ } error;
};
static struct lil_value *next_word(struct lil *lil);
@@ -645,7 +650,7 @@ int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)
}
struct lil_var *lil_set_var(struct lil *lil, const char *name,
- struct lil_value *val, int local)
+ struct lil_value *val, enum lil_setvar local)
{
struct lil_var **nvar;
struct lil_env *env =
@@ -1222,11 +1227,6 @@ int lil_error(struct lil *lil, const char **msg, size_t *pos)
return 1;
}
-#define EERR_NO_ERROR 0
-#define EERR_SYNTAX_ERROR 1
-#define EERR_DIVISION_BY_ZERO 3
-#define EERR_INVALID_EXPRESSION 4
-
static void ee_expr(struct expreval *ee);
static int ee_invalidpunct(int ch)
@@ -1673,16 +1673,18 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
ee_expr(&ee);
lil_free_value(code);
- if (ee.error) {
- switch (ee.error) {
- case EERR_DIVISION_BY_ZERO:
- lil_set_error(lil, "division by zero in expression");
- break;
- case EERR_SYNTAX_ERROR:
- lil_set_error(lil, "expression syntax error");
- break;
- }
+ switch (ee.error) {
+ case EERR_DIVISION_BY_ZERO:
+ lil_set_error(lil, "division by zero in expression");
return NULL;
+ case EERR_SYNTAX_ERROR:
+ lil_set_error(lil, "expression syntax error");
+ return NULL;
+ case EERR_INVALID_EXPRESSION:
+ lil_set_error(lil, "invalid expression");
+ return NULL;
+ case EERR_NO_ERROR:
+ break;
}
return lil_alloc_integer(ee.ival);
}
diff --git a/include/cli_lil.h b/include/cli_lil.h
index c72977ea5c..48735e0605 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -13,10 +13,12 @@
#define LIL_VERSION_STRING "0.1"
-#define LIL_SETVAR_GLOBAL 0
-#define LIL_SETVAR_LOCAL 1
-#define LIL_SETVAR_LOCAL_NEW 2
-#define LIL_SETVAR_LOCAL_ONLY 3
+enum lil_setvar {
+ LIL_SETVAR_GLOBAL = 0,
+ LIL_SETVAR_LOCAL,
+ LIL_SETVAR_LOCAL_NEW,
+ LIL_SETVAR_LOCAL_ONLY,
+};
#define LIL_CALLBACK_EXIT 0
#define LIL_CALLBACK_WRITE 1
@@ -98,7 +100,7 @@ struct lil_env *lil_push_env(struct lil *lil);
void lil_pop_env(struct lil *lil);
struct lil_var *lil_set_var(struct lil *lil, const char *name,
- struct lil_value *val, int local);
+ struct lil_value *val, enum lil_setvar local);
struct lil_value *lil_get_var(struct lil *lil, const char *name);
struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
struct lil_value *defvalue);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 07/28] cli: lil: Simplify callbacks
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (5 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 06/28] cli: lil: Convert some defines to enums Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 08/28] cli: lil: Handle commands with dots Sean Anderson
` (22 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
We only need the getvar and setvar callbacks for now (to integrate with the
environment). So remove all the other callbacks. Instead of allowing
callbacks to be set at any time, just set them once in lil_new. Lastly, get
rid of the typedefs and use a struct to pass in the callbacks instead.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 45 ++++++++++++---------------------------------
include/cli_lil.h | 34 ++++++++--------------------------
2 files changed, 20 insertions(+), 59 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 3d1e6181f9..539a4a3238 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -24,7 +24,6 @@
* overflows and is also useful when running through an automated fuzzer like AFL */
/*#define LIL_ENABLE_RECLIMIT 10000*/
-#define CALLBACKS 8
#define HASHMAP_CELLS 256
#define HASHMAP_CELLMASK 0xFF
@@ -100,7 +99,7 @@ struct lil {
} error;
size_t err_head;
char *err_msg;
- lil_callback_proc_t callback[CALLBACKS];
+ struct lil_callbacks callbacks;
size_t parse_depth;
};
@@ -667,14 +666,10 @@ struct lil_var *lil_set_var(struct lil *lil, const char *name,
var->env == lil->rootenv && var->env != env)
var = NULL;
- if (((!var && env == lil->rootenv) ||
- (var && var->env == lil->rootenv)) &&
- lil->callback[LIL_CALLBACK_SETVAR]) {
- lil_setvar_callback_proc_t proc =
- (lil_setvar_callback_proc_t)
- lil->callback[LIL_CALLBACK_SETVAR];
+ if (lil->callbacks.setvar && ((!var && env == lil->rootenv) ||
+ (var && var->env == lil->rootenv))) {
struct lil_value *newval = val;
- int r = proc(lil, name, &newval);
+ int r = lil->callbacks.setvar(lil, name, &newval);
if (r < 0) {
return NULL;
@@ -717,14 +712,10 @@ struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
struct lil_var *var = lil_find_var(lil, lil->env, name);
struct lil_value *retval = var ? var->v : defvalue;
- if (lil->callback[LIL_CALLBACK_GETVAR] &&
- (!var || var->env == lil->rootenv)) {
- lil_getvar_callback_proc_t proc =
- (lil_getvar_callback_proc_t)
- lil->callback[LIL_CALLBACK_GETVAR];
+ if (lil->callbacks.getvar && (!var || var->env == lil->rootenv)) {
struct lil_value *newretval = retval;
- if (proc(lil, name, &newretval))
+ if (lil->callbacks.getvar(lil, name, &newretval))
retval = newretval;
}
return retval;
@@ -748,10 +739,15 @@ void lil_pop_env(struct lil *lil)
}
}
-struct lil *lil_new(void)
+struct lil *lil_new(const struct lil_callbacks *callbacks)
{
struct lil *lil = calloc(1, sizeof(struct lil));
+ if (!lil)
+ return NULL;
+
+ if (callbacks)
+ memcpy(&lil->callbacks, callbacks, sizeof(lil->callbacks));
lil->rootenv = lil->env = lil_alloc_env(NULL);
lil->empty = alloc_value(NULL);
hm_init(&lil->cmdmap);
@@ -1148,15 +1144,6 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
}
cleanup:
- if (lil->error && lil->callback[LIL_CALLBACK_ERROR] &&
- lil->parse_depth == 1) {
- lil_error_callback_proc_t proc =
- (lil_error_callback_proc_t)
- lil->callback[LIL_CALLBACK_ERROR];
-
- proc(lil, lil->err_head, lil->err_msg);
- }
-
if (words)
lil_free_list(words);
lil->code = save_code;
@@ -1185,14 +1172,6 @@ struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
return lil_parse(lil, val->d, val->l, funclevel);
}
-void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc)
-{
- if (cb < 0 || cb > CALLBACKS)
- return;
-
- lil->callback[cb] = proc;
-}
-
void lil_set_error(struct lil *lil, const char *msg)
{
if (lil->error)
diff --git a/include/cli_lil.h b/include/cli_lil.h
index 48735e0605..b8df94a766 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -20,15 +20,6 @@ enum lil_setvar {
LIL_SETVAR_LOCAL_ONLY,
};
-#define LIL_CALLBACK_EXIT 0
-#define LIL_CALLBACK_WRITE 1
-#define LIL_CALLBACK_READ 2
-#define LIL_CALLBACK_STORE 3
-#define LIL_CALLBACK_SOURCE 4
-#define LIL_CALLBACK_ERROR 5
-#define LIL_CALLBACK_SETVAR 6
-#define LIL_CALLBACK_GETVAR 7
-
#include <stdint.h>
#include <inttypes.h>
@@ -40,22 +31,15 @@ struct lil_list;
struct lil;
typedef struct lil_value *(*lil_func_proc_t)(struct lil *lil, size_t argc,
struct lil_value **argv);
-typedef void (*lil_exit_callback_proc_t)(struct lil *lil,
- struct lil_value *arg);
-typedef void (*lil_write_callback_proc_t)(struct lil *lil, const char *msg);
-typedef char *(*lil_read_callback_proc_t)(struct lil *lil, const char *name);
-typedef char *(*lil_source_callback_proc_t)(struct lil *lil, const char *name);
-typedef void (*lil_store_callback_proc_t)(struct lil *lil, const char *name,
- const char *data);
-typedef void (*lil_error_callback_proc_t)(struct lil *lil, size_t pos,
- const char *msg);
-typedef int (*lil_setvar_callback_proc_t)(struct lil *lil, const char *name,
- struct lil_value **value);
-typedef int (*lil_getvar_callback_proc_t)(struct lil *lil, const char *name,
- struct lil_value **value);
-typedef void (*lil_callback_proc_t)(void);
-struct lil *lil_new(void);
+struct lil_callbacks {
+ int (*setvar)(struct lil *lil, const char *name,
+ struct lil_value **value);
+ int (*getvar)(struct lil *lil, const char *name,
+ struct lil_value **value);
+};
+
+struct lil *lil_new(const struct lil_callbacks *callbacks);
void lil_free(struct lil *lil);
int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc);
@@ -65,8 +49,6 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
int funclevel);
-void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc);
-
void lil_set_error(struct lil *lil, const char *msg);
void lil_set_error_at(struct lil *lil, size_t pos, const char *msg);
int lil_error(struct lil *lil, const char **msg, size_t *pos);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 08/28] cli: lil: Handle commands with dots
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (6 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 07/28] cli: lil: Simplify callbacks Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 09/28] cli: lil: Use error codes Sean Anderson
` (21 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
Some commands (such as md, mw, etc) use dot suffixes as options to the
command. Therefore, we must ignore anything after a dot when finding
commands.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 539a4a3238..dd020e8452 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -579,7 +579,17 @@ static struct lil_var *lil_find_var(struct lil *lil, struct lil_env *env,
static struct lil_func *lil_find_cmd(struct lil *lil, const char *name)
{
- return hm_get(&lil->cmdmap, name);
+ struct lil_func *r;
+ char *dot = strchr(name, '.');
+
+ /* Some U-Boot commands have dots in their names */
+ if (dot)
+ *dot = '\0';
+ r = hm_get(&lil->cmdmap, name);
+
+ if (dot)
+ *dot = '.';
+ return r;
}
static struct lil_func *add_func(struct lil *lil, const char *name)
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 09/28] cli: lil: Use error codes
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (7 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 08/28] cli: lil: Handle commands with dots Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 10/28] cli: lil: Add printf-style format helper for errors Sean Anderson
` (20 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds numeric error codes to be used for errors. This makes it easier
for code to differentiate between different classes of errors. Some error
codes are not used yet, and will be added in future commits. The position
argument has been removed for a few reasons:
- It doesn't make sense for some errors, such as running out of memory
- It was often not very helpful, since it pointed at the end of the command
- Determining the location of an error is more difficult when the parsing
and evaluating steps are done separately.
In addition, lil.error is changed to lil.err for consistency.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 92 ++++++++++++++++++-----------------------------
include/cli_lil.h | 36 +++++++++++++++++--
2 files changed, 68 insertions(+), 60 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index dd020e8452..1bdf0e5265 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -92,12 +92,7 @@ struct lil {
struct lil_env *rootenv;
struct lil_env *downenv;
struct lil_value *empty;
- enum {
- ERROR_NOERROR = 0,
- ERROR_DEFAULT,
- ERROR_FIXHEAD,
- } error;
- size_t err_head;
+ enum lil_error err;
char *err_msg;
struct lil_callbacks callbacks;
size_t parse_depth;
@@ -117,6 +112,7 @@ struct expreval {
static struct lil_value *next_word(struct lil *lil);
static void register_stdcmds(struct lil *lil);
+static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
#ifdef LIL_ENABLE_POOLS
static struct lil_value **pool;
@@ -974,7 +970,7 @@ static struct lil_list *substitute(struct lil *lil)
struct lil_list *words = lil_alloc_list();
lil_skip_spaces(lil);
- while (lil->head < lil->clen && !ateol(lil) && !lil->error) {
+ while (lil->head < lil->clen && !ateol(lil) && !lil->err) {
struct lil_value *w = alloc_value(NULL);
do {
@@ -993,7 +989,7 @@ static struct lil_list *substitute(struct lil *lil)
lil_free_value(wp);
} while (lil->head < lil->clen &&
!eolchar(lil->code[lil->head]) &&
- !isspace(lil->code[lil->head]) && !lil->error);
+ !isspace(lil->code[lil->head]) && !lil->err);
lil_skip_spaces(lil);
lil_list_append(words, w);
@@ -1042,13 +1038,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
struct lil_value *r;
if (cmd->proc) {
- size_t shead = lil->head;
-
r = cmd->proc(lil, words->c - 1, words->v + 1);
- if (lil->error == ERROR_FIXHEAD) {
- lil->error = ERROR_DEFAULT;
- lil->err_head = shead;
- }
} else {
lil_push_env(lil);
lil->env->func = cmd;
@@ -1102,18 +1092,18 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
lil->parse_depth++;
#ifdef LIL_ENABLE_RECLIMIT
if (lil->parse_depth > LIL_ENABLE_RECLIMIT) {
- lil_set_error(lil, "Too many recursive calls");
+ lil_set_error(lil, LIL_ERR_DEPTH, "Too many recursive calls");
goto cleanup;
}
#endif
if (lil->parse_depth == 1)
- lil->error = 0;
+ lil->err = LIL_ERR_NONE;
if (funclevel)
lil->env->breakrun = 0;
- while (lil->head < lil->clen && !lil->error) {
+ while (lil->head < lil->clen && !lil->err) {
if (words)
lil_free_list(words);
@@ -1122,7 +1112,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
val = NULL;
words = substitute(lil);
- if (!words || lil->error)
+ if (!words || lil->err)
goto cleanup;
if (words->c) {
@@ -1136,7 +1126,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
snprintf(msg, sizeof(msg),
"unknown function %s",
words->v[0]->d);
- lil_set_error_at(lil, lil->head, msg);
+ lil_set_error(lil, LIL_ERR_NOCMD, msg);
goto cleanup;
}
} else {
@@ -1182,38 +1172,25 @@ struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
return lil_parse(lil, val->d, val->l, funclevel);
}
-void lil_set_error(struct lil *lil, const char *msg)
+static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg)
{
- if (lil->error)
- return;
-
+ assert(!lil->err);
free(lil->err_msg);
- lil->error = ERROR_FIXHEAD;
- lil->err_head = 0;
- lil->err_msg = strdup(msg ? msg : "");
+ lil->err = err;
+ lil->err_msg = strdup(msg);
}
-void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
+enum lil_error lil_error(struct lil *lil, const char **msg)
{
- if (lil->error)
- return;
+ enum lil_error err = lil->err;
- free(lil->err_msg);
- lil->error = ERROR_DEFAULT;
- lil->err_head = pos;
- lil->err_msg = strdup(msg ? msg : "");
-}
-
-int lil_error(struct lil *lil, const char **msg, size_t *pos)
-{
- if (!lil->error)
- return 0;
+ if (!err)
+ return LIL_ERR_NONE;
*msg = lil->err_msg;
- *pos = lil->err_head;
- lil->error = ERROR_NOERROR;
+ lil->err = LIL_ERR_NONE;
- return 1;
+ return err;
}
static void ee_expr(struct expreval *ee);
@@ -1642,7 +1619,7 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
struct expreval ee;
code = lil_subst_to_value(lil, code);
- if (lil->error)
+ if (lil->err)
return NULL;
ee.code = lil_to_string(code);
@@ -1664,13 +1641,13 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
lil_free_value(code);
switch (ee.error) {
case EERR_DIVISION_BY_ZERO:
- lil_set_error(lil, "division by zero in expression");
+ lil_set_error(lil, LIL_ERR_DOM, "division by zero");
return NULL;
case EERR_SYNTAX_ERROR:
- lil_set_error(lil, "expression syntax error");
+ lil_set_error(lil, LIL_ERR_SYNTAX, "expression syntax");
return NULL;
case EERR_INVALID_EXPRESSION:
- lil_set_error(lil, "invalid expression");
+ lil_set_error(lil, LIL_ERR_SYNTAX, "invalid expression");
return NULL;
case EERR_NO_ERROR:
break;
@@ -1994,7 +1971,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
char *msg = malloc(24 + strlen(oldname));
sprintf(msg, "unknown function '%s'", oldname);
- lil_set_error_at(lil, lil->head, msg);
+ lil_set_error(lil, LIL_ERR_NOCMD, msg);
free(msg);
return NULL;
}
@@ -2397,7 +2374,7 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
else
lil_free_value(rv);
- if (lil->env->breakrun || lil->error)
+ if (lil->env->breakrun || lil->err)
break;
}
@@ -2499,7 +2476,7 @@ static struct lil_value *fnc_if(struct lil *lil, size_t argc,
return NULL;
val = lil_eval_expr(lil, argv[base]);
- if (!val || lil->error)
+ if (!val || lil->err)
return NULL;
v = lil_to_boolean(val);
@@ -2530,9 +2507,9 @@ static struct lil_value *fnc_while(struct lil *lil, size_t argc,
if (argc < (size_t)base + 2)
return NULL;
- while (!lil->error && !lil->env->breakrun) {
+ while (!lil->err && !lil->env->breakrun) {
val = lil_eval_expr(lil, argv[base]);
- if (!val || lil->error)
+ if (!val || lil->err)
return NULL;
v = lil_to_boolean(val);
@@ -2562,9 +2539,9 @@ static struct lil_value *fnc_for(struct lil *lil, size_t argc,
return NULL;
lil_free_value(lil_parse_value(lil, argv[0], 0));
- while (!lil->error && !lil->env->breakrun) {
+ while (!lil->err && !lil->env->breakrun) {
val = lil_eval_expr(lil, argv[1]);
- if (!val || lil->error)
+ if (!val || lil->err)
return NULL;
if (!lil_to_boolean(val)) {
@@ -2889,12 +2866,12 @@ static struct lil_value *fnc_try(struct lil *lil, size_t argc,
if (argc < 1)
return NULL;
- if (lil->error)
+ if (lil->err)
return NULL;
r = lil_parse_value(lil, argv[0], 0);
- if (lil->error) {
- lil->error = ERROR_NOERROR;
+ if (lil->err) {
+ lil->err = LIL_ERR_NONE;
lil_free_value(r);
if (argc > 1)
@@ -2908,7 +2885,8 @@ static struct lil_value *fnc_try(struct lil *lil, size_t argc,
static struct lil_value *fnc_error(struct lil *lil, size_t argc,
struct lil_value **argv)
{
- lil_set_error(lil, argc > 0 ? lil_to_string(argv[0]) : NULL);
+ lil_set_error(lil, LIL_ERR_USER,
+ argc > 0 ? lil_to_string(argv[0]) : NULL);
return NULL;
}
diff --git a/include/cli_lil.h b/include/cli_lil.h
index b8df94a766..cdaa79fd15 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -39,6 +39,38 @@ struct lil_callbacks {
struct lil_value **value);
};
+/**
+ * enum lil_error - Error codes returned by @lil_error
+ * LIL_ERR_NONE: No error
+ * LIL_ERR_OOM: Out of memory
+ * LIL_ERR_CASE: Missing case statement; generally a bug in the program or the
+ * result of memory corruption.
+ * LIL_ERR_DEPTH: Too much recursion while parsing or evaluating
+ * LIL_ERR_SYNTAX: Unrecoverable syntax error
+ * LIL_ERR_EOF: Unexpected end of input; may be recoverable by prompting the
+ * user for more input.
+ * LIL_ERR_INTR: Interrupted by Ctrl-C
+ * LIL_ERR_NOCMD: No such command
+ * LIL_ERR_NOVAR: No such variable
+ * LIL_ERR_ARGC: Incorrect number of arguments
+ * LIL_ERR_DOM: Numerical argument out of range (typically division by zero)
+ * LIL_ERR_USER: Error set by user
+ */
+enum lil_error {
+ LIL_ERR_NONE = 0,
+ LIL_ERR_OOM,
+ LIL_ERR_CASE,
+ LIL_ERR_DEPTH,
+ LIL_ERR_SYNTAX,
+ LIL_ERR_EOF,
+ LIL_ERR_INTR,
+ LIL_ERR_NOCMD,
+ LIL_ERR_NOVAR,
+ LIL_ERR_ARGC,
+ LIL_ERR_DOM,
+ LIL_ERR_USER,
+};
+
struct lil *lil_new(const struct lil_callbacks *callbacks);
void lil_free(struct lil *lil);
@@ -49,9 +81,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
int funclevel);
-void lil_set_error(struct lil *lil, const char *msg);
-void lil_set_error_at(struct lil *lil, size_t pos, const char *msg);
-int lil_error(struct lil *lil, const char **msg, size_t *pos);
+enum lil_error lil_error(struct lil *lil, const char **msg);
const char *lil_to_string(struct lil_value *val);
ssize_t lil_to_integer(struct lil_value *val);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 10/28] cli: lil: Add printf-style format helper for errors
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (8 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 09/28] cli: lil: Use error codes Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 11/28] cli: lil: Add several helper functions " Sean Anderson
` (19 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds a helper for dynamically-created error messages.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 43 ++++++++++++++++++++++++++++++++-----------
1 file changed, 32 insertions(+), 11 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 1bdf0e5265..f29c8065d8 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -113,6 +113,9 @@ struct expreval {
static struct lil_value *next_word(struct lil *lil);
static void register_stdcmds(struct lil *lil);
static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
+static void lil_set_errorf(struct lil *lil, enum lil_error err,
+ const char *fmt, ...)
+ __attribute((format(__printf__, 3, 4)));
#ifdef LIL_ENABLE_POOLS
static struct lil_value **pool;
@@ -1121,12 +1124,9 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
if (!cmd) {
if (words->v[0]->l) {
- char msg[64];
-
- snprintf(msg, sizeof(msg),
- "unknown function %s",
- words->v[0]->d);
- lil_set_error(lil, LIL_ERR_NOCMD, msg);
+ lil_set_errorf(lil, LIL_ERR_NOCMD,
+ "unknown function %s",
+ words->v[0]->d);
goto cleanup;
}
} else {
@@ -1180,6 +1180,30 @@ static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg)
lil->err_msg = strdup(msg);
}
+static void lil_set_errorf(struct lil *lil, enum lil_error err,
+ const char *fmt, ...)
+{
+ va_list args, saveargs;
+ size_t n;
+
+ assert(!lil->err);
+ free(lil->err_msg);
+ lil->err = err;
+
+ va_start(args, fmt);
+ va_copy(saveargs, args);
+ n = vsnprintf(NULL, 0, fmt, args) + 1;
+ va_end(args);
+
+ lil->err_msg = calloc(1, n);
+ if (!lil->err_msg) {
+ lil->err = LIL_ERR_OOM;
+ return;
+ }
+ vsnprintf(lil->err_msg, n, fmt, saveargs);
+ va_end(saveargs);
+}
+
enum lil_error lil_error(struct lil *lil, const char **msg)
{
enum lil_error err = lil->err;
@@ -1968,11 +1992,8 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
newname = lil_to_string(argv[1]);
func = lil_find_cmd(lil, oldname);
if (!func) {
- char *msg = malloc(24 + strlen(oldname));
-
- sprintf(msg, "unknown function '%s'", oldname);
- lil_set_error(lil, LIL_ERR_NOCMD, msg);
- free(msg);
+ lil_set_errorf(lil, LIL_ERR_NOCMD, "unknown function %s",
+ oldname);
return NULL;
}
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 11/28] cli: lil: Add several helper functions for errors
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (9 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 10/28] cli: lil: Add printf-style format helper for errors Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 12/28] cli: lil: Check for ctrl-c Sean Anderson
` (18 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds several helper functions for common error types. This helps keep
down code duplication, and also ensures that each of these errors uses the
same message.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 38 +++++++++++++++++++++++++++++++++-----
1 file changed, 33 insertions(+), 5 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index f29c8065d8..c19a25b2bf 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -116,6 +116,9 @@ static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
static void lil_set_errorf(struct lil *lil, enum lil_error err,
const char *fmt, ...)
__attribute((format(__printf__, 3, 4)));
+static void lil_set_error_oom(struct lil *lil);
+static void lil_set_error_nocmd(struct lil *lil, const char *cmdname);
+static void lil_set_error_intr(struct lil *lil);
#ifdef LIL_ENABLE_POOLS
static struct lil_value **pool;
@@ -1124,9 +1127,8 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
if (!cmd) {
if (words->v[0]->l) {
- lil_set_errorf(lil, LIL_ERR_NOCMD,
- "unknown function %s",
- words->v[0]->d);
+ lil_set_error_nocmd(lil,
+ words->v[0]->d);
goto cleanup;
}
} else {
@@ -1204,6 +1206,33 @@ static void lil_set_errorf(struct lil *lil, enum lil_error err,
va_end(saveargs);
}
+/*
+ * The next several functions provide helpers for throwing some formulaic
+ * errors. Their purpose is to ensure that the same wording is used everywhere
+ * the error is used. This reduces data size.
+ */
+
+static void lil_set_error_oom(struct lil *lil)
+{
+ lil_set_error(lil, LIL_ERR_OOM, NULL);
+}
+
+static void lil_set_error_argc(struct lil *lil, size_t expected)
+{
+ lil_set_errorf(lil, LIL_ERR_ARGC, "%s: expected %zu arguments",
+ lil->env->proc ?: lil->env->func->name, expected);
+}
+
+static void lil_set_error_nocmd(struct lil *lil, const char *cmdname)
+{
+ lil_set_errorf(lil, LIL_ERR_NOCMD, "no such command %s", cmdname);
+}
+
+static void lil_set_error_intr(struct lil *lil)
+{
+ lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+}
+
enum lil_error lil_error(struct lil *lil, const char **msg)
{
enum lil_error err = lil->err;
@@ -1992,8 +2021,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
newname = lil_to_string(argv[1]);
func = lil_find_cmd(lil, oldname);
if (!func) {
- lil_set_errorf(lil, LIL_ERR_NOCMD, "unknown function %s",
- oldname);
+ lil_set_error_nocmd(lil, oldname);
return NULL;
}
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 12/28] cli: lil: Check for ctrl-c
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (10 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 11/28] cli: lil: Add several helper functions " Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
2021-07-01 6:15 ` [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot Sean Anderson
` (17 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
Check for ctrl-c in lil_parse. This works out to around every time a
function or command is called. We also check at the beginning of
lil_eval_expr so that constructs like
while {1} {}
get interrupted. Since there are no non-trivial commands in that example,
lil_parse never gets to its ctrlc check. However, we do need to evaluate
the loop expression, so that's a good place to put a check.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index c19a25b2bf..50e314a643 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -10,6 +10,7 @@
#include <common.h>
#include <cli_lil.h>
+#include <console.h>
#include <ctype.h>
#include <stdlib.h>
#include <stdio.h>
@@ -1117,6 +1118,11 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
lil_free_value(val);
val = NULL;
+ if (ctrlc()) {
+ lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+ goto cleanup;
+ }
+
words = substitute(lil);
if (!words || lil->err)
goto cleanup;
@@ -1671,6 +1677,11 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
{
struct expreval ee;
+ if (ctrlc()) {
+ lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+ return NULL;
+ }
+
code = lil_subst_to_value(lil, code);
if (lil->err)
return NULL;
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 12/28] cli: lil: Check for ctrl-c
2021-07-01 6:15 ` [RFC PATCH 12/28] cli: lil: Check for ctrl-c Sean Anderson
@ 2021-07-05 15:29 ` Simon Glass
0 siblings, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>
> Check for ctrl-c in lil_parse. This works out to around every time a
> function or command is called. We also check at the beginning of
> lil_eval_expr so that constructs like
>
> while {1} {}
>
> get interrupted. Since there are no non-trivial commands in that example,
> lil_parse never gets to its ctrlc check. However, we do need to evaluate
> the loop expression, so that's a good place to put a check.
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
> ---
>
> common/cli_lil.c | 11 +++++++++++
> 1 file changed, 11 insertions(+)
Reviewed-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (11 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 12/28] cli: lil: Check for ctrl-c Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-02 8:18 ` Rasmus Villemoes
2021-07-05 15:29 ` Simon Glass
2021-07-01 6:15 ` [RFC PATCH 14/28] cli: lil: Document structures Sean Anderson
` (16 subsequent siblings)
29 siblings, 2 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands
are not supporteed. Neither are partial commands a la Hush's secondary
prompt. Setting and getting environmental variables is done through
callbacks to assist with testing.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
cmd/Kconfig | 12 +++++--
common/cli.c | 84 +++++++++++++++++++++++++++++++++++++++---------
common/cli_lil.c | 32 ++++++++++++++++++
3 files changed, 111 insertions(+), 17 deletions(-)
diff --git a/cmd/Kconfig b/cmd/Kconfig
index 0a7b73cb6d..b61a7557a9 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -11,9 +11,14 @@ config CMDLINE
Depending on the number of commands enabled, this can add
substantially to the size of U-Boot.
+if CMDLINE
+
+choice
+ prompt "Shell"
+ default HUSH_PARSER
+
config HUSH_PARSER
bool "Use hush shell"
- depends on CMDLINE
help
This option enables the "hush" shell (from Busybox) as command line
interpreter, thus enabling powerful command line syntax like
@@ -25,13 +30,14 @@ config HUSH_PARSER
config LIL
bool "Use LIL shell"
- depends on CMDLINE
help
This options enables the "Little Interpreted Language" (LIL) shell as
command line interpreter, thus enabling powerful command line syntax
like `proc name {args} {body}' functions or `echo [some command]`
command substitution ("tcl scripts").
+endchoice
+
if LIL
config LIL_FULL
@@ -42,6 +48,8 @@ config LIL_FULL
endif
+endif
+
config CMDLINE_EDITING
bool "Enable command line editing"
depends on CMDLINE
diff --git a/common/cli.c b/common/cli.c
index 048eacb9ef..ad5d76d563 100644
--- a/common/cli.c
+++ b/common/cli.c
@@ -12,6 +12,7 @@
#include <bootstage.h>
#include <cli.h>
#include <cli_hush.h>
+#include <cli_lil.h>
#include <command.h>
#include <console.h>
#include <env.h>
@@ -22,6 +23,53 @@
DECLARE_GLOBAL_DATA_PTR;
+#ifdef CONFIG_LIL
+static struct lil *lil;
+
+static int env_setvar(struct lil *lil, const char *name,
+ struct lil_value **value)
+{
+ if (env_set(name, lil_to_string(*value)))
+ return -1;
+ return 0;
+}
+
+static int env_getvar(struct lil *lil, const char *name,
+ struct lil_value **value)
+{
+ *value = lil_alloc_string(env_get(name));
+ return 1;
+}
+
+static const struct lil_callbacks env_callbacks = {
+ .setvar = env_setvar,
+ .getvar = env_getvar,
+};
+
+static int lil_run(const char *cmd)
+{
+ int err;
+ struct lil_value *result = lil_parse(lil, cmd, 0, 0);
+ const char *err_msg, *strres = lil_to_string(result);
+
+ /* The result may be very big, so use puts */
+ if (strres && strres[0]) {
+ puts(strres);
+ putc('\n');
+ }
+ lil_free_value(result);
+
+ err = lil_error(lil, &err_msg);
+ if (err) {
+ if (err_msg)
+ printf("error: %s\n", err_msg);
+ else
+ printf("error: %d\n", err);
+ }
+ return !!err;
+}
+#endif
+
#ifdef CONFIG_CMDLINE
/*
* Run a command using the selected parser.
@@ -32,7 +80,15 @@ DECLARE_GLOBAL_DATA_PTR;
*/
int run_command(const char *cmd, int flag)
{
-#if !CONFIG_IS_ENABLED(HUSH_PARSER)
+#ifdef CONFIG_HUSH_PARSER
+ int hush_flags = FLAG_PARSE_SEMICOLON | FLAG_EXIT_FROM_LOOP;
+
+ if (flag & CMD_FLAG_ENV)
+ hush_flags |= FLAG_CONT_ON_NEWLINE;
+ return parse_string_outer(cmd, hush_flags);
+#elif defined(CONFIG_LIL)
+ return lil_run(cmd);
+#else
/*
* cli_run_command can return 0 or 1 for success, so clean up
* its result.
@@ -41,12 +97,6 @@ int run_command(const char *cmd, int flag)
return 1;
return 0;
-#else
- int hush_flags = FLAG_PARSE_SEMICOLON | FLAG_EXIT_FROM_LOOP;
-
- if (flag & CMD_FLAG_ENV)
- hush_flags |= FLAG_CONT_ON_NEWLINE;
- return parse_string_outer(cmd, hush_flags);
#endif
}
@@ -59,9 +109,7 @@ int run_command(const char *cmd, int flag)
*/
int run_command_repeatable(const char *cmd, int flag)
{
-#ifndef CONFIG_HUSH_PARSER
- return cli_simple_run_command(cmd, flag);
-#else
+#ifdef CONFIG_HUSH_PARSER
/*
* parse_string_outer() returns 1 for failure, so clean up
* its result.
@@ -71,6 +119,10 @@ int run_command_repeatable(const char *cmd, int flag)
return -1;
return 0;
+#elif defined(CONFIG_LIL)
+ return run_command(cmd, flag);
+#else
+ return cli_simple_run_command(cmd, flag);
#endif
}
#else
@@ -90,7 +142,7 @@ int run_command_list(const char *cmd, int len, int flag)
if (len == -1) {
len = strlen(cmd);
-#ifdef CONFIG_HUSH_PARSER
+#if defined(CONFIG_HUSH_PARSER) || defined(CONFIG_LIL)
/* hush will never change our string */
need_buff = 0;
#else
@@ -107,7 +159,9 @@ int run_command_list(const char *cmd, int len, int flag)
}
#ifdef CONFIG_HUSH_PARSER
rcode = parse_string_outer(buff, FLAG_PARSE_SEMICOLON);
-#else
+#elif defined(CONFIG_LIL)
+ rcode = lil_run(buff);
+#elif defined(CONFIG_CMDLINE)
/*
* This function will overwrite any \n it sees with a \0, which
* is why it can't work with a const char *. Here we are making
@@ -115,11 +169,9 @@ int run_command_list(const char *cmd, int len, int flag)
* doing a malloc() which is actually required only in a case that
* is pretty rare.
*/
-#ifdef CONFIG_CMDLINE
rcode = cli_simple_run_command_list(buff, flag);
#else
rcode = board_run_command(buff);
-#endif
#endif
if (need_buff)
free(buff);
@@ -241,9 +293,11 @@ void cli_init(void)
{
#ifdef CONFIG_HUSH_PARSER
u_boot_hush_start();
+#elif defined(CONFIG_LIL)
+ lil = lil_new(&env_callbacks);
#endif
-#if defined(CONFIG_HUSH_INIT_VAR)
+#ifdef CONFIG_HUSH_INIT_VAR
hush_init_var();
#endif
}
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 50e314a643..66ee62bf33 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -11,6 +11,7 @@
#include <common.h>
#include <cli_lil.h>
#include <console.h>
+#include <command.h>
#include <ctype.h>
#include <stdlib.h>
#include <stdio.h>
@@ -59,6 +60,7 @@ struct lil_var {
struct lil_env {
struct lil_env *parent;
struct lil_func *func;
+ const char *proc;
struct lil_var **var;
size_t vars;
struct hashmap varmap;
@@ -1045,7 +1047,9 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
struct lil_value *r;
if (cmd->proc) {
+ lil->env->proc = words->v[0]->d;
r = cmd->proc(lil, words->c - 1, words->v + 1);
+ lil->env->proc = NULL;
} else {
lil_push_env(lil);
lil->env->func = cmd;
@@ -2967,8 +2971,33 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
return NULL;
}
+static struct lil_value *fnc_builtin(struct lil *lil, size_t argc,
+ struct lil_value **lil_argv)
+{
+ int err, repeatable;
+ size_t i;
+ /*
+ * We need space for the function name, and the last argv must be NULL
+ */
+ char **argv = calloc(sizeof(char *), argc + 2);
+
+ argv[0] = (char *)lil->env->proc;
+ for (i = 0; i < argc; i++)
+ argv[i + 1] = (char *)lil_to_string(lil_argv[i]);
+
+ err = cmd_process(0, argc + 1, argv, &repeatable, NULL);
+ if (err)
+ lil_set_errorf(lil, LIL_ERR_USER, "%s failed", argv[0]);
+ free(argv);
+
+ return 0;
+}
+
static void register_stdcmds(struct lil *lil)
{
+ struct cmd_tbl *cmdtp, *start = ll_entry_start(struct cmd_tbl, cmd);
+ const int len = ll_entry_count(struct cmd_tbl, cmd);
+
lil_register(lil, "decr", fnc_decr);
lil_register(lil, "eval", fnc_eval);
lil_register(lil, "expr", fnc_expr);
@@ -2984,6 +3013,9 @@ static void register_stdcmds(struct lil *lil)
lil_register(lil, "try", fnc_try);
lil_register(lil, "while", fnc_while);
+ for (cmdtp = start; cmdtp != start + len; cmdtp++)
+ lil_register(lil, cmdtp->name, fnc_builtin);
+
if (IS_ENABLED(CONFIG_LIL_FULL)) {
lil_register(lil, "append", fnc_append);
lil_register(lil, "char", fnc_char);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
2021-07-01 6:15 ` [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot Sean Anderson
@ 2021-07-02 8:18 ` Rasmus Villemoes
2021-07-02 13:40 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
1 sibling, 1 reply; 107+ messages in thread
From: Rasmus Villemoes @ 2021-07-02 8:18 UTC (permalink / raw)
To: Sean Anderson, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 01/07/2021 08.15, Sean Anderson wrote:
> This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands
> are not supporteed. Neither are partial commands a la Hush's secondary
> prompt. Setting and getting environmental variables is done through
> callbacks to assist with testing.
This all looks very interesting, thanks for doing this!
Re this patch, is there some way LIL and HUSH could coexist, with HUSH
then being the initial shell, and one could then enter a "lil shell"
with the command "lil", just as I can start a busybox shell from bash by
saying "busybox sh". That could make it a bit easier for playing around
with initially.
I have no idea how hard that is to do with Kconfig. Perhaps make LIL
selectable by itself, then have a hidden symbol LIL_SHELL which is
"default y if LIL && !HUSH_PARSER" or something like that.
Rasmus
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
2021-07-02 8:18 ` Rasmus Villemoes
@ 2021-07-02 13:40 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-02 13:40 UTC (permalink / raw)
To: Rasmus Villemoes, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/2/21 4:18 AM, Rasmus Villemoes wrote:
> On 01/07/2021 08.15, Sean Anderson wrote:
>> This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands
>> are not supporteed. Neither are partial commands a la Hush's secondary
>> prompt. Setting and getting environmental variables is done through
>> callbacks to assist with testing.
>
> This all looks very interesting, thanks for doing this!
>
> Re this patch, is there some way LIL and HUSH could coexist, with HUSH
> then being the initial shell, and one could then enter a "lil shell"
> with the command "lil", just as I can start a busybox shell from bash by
> saying "busybox sh". That could make it a bit easier for playing around
> with initially.
>
> I have no idea how hard that is to do with Kconfig. Perhaps make LIL
> selectable by itself, then have a hidden symbol LIL_SHELL which is
> "default y if LIL && !HUSH_PARSER" or something like that.
Yeah, I wanted to do this anyway for unit testing at least.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
2021-07-01 6:15 ` [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot Sean Anderson
2021-07-02 8:18 ` Rasmus Villemoes
@ 2021-07-05 15:29 ` Simon Glass
1 sibling, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Hi Sean,
On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>
> This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands
> are not supporteed. Neither are partial commands a la Hush's secondary
> prompt. Setting and getting environmental variables is done through
> callbacks to assist with testing.
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
> ---
>
> cmd/Kconfig | 12 +++++--
> common/cli.c | 84 +++++++++++++++++++++++++++++++++++++++---------
> common/cli_lil.c | 32 ++++++++++++++++++
> 3 files changed, 111 insertions(+), 17 deletions(-)
Can you try using if() instead of #ifdef in here?
Regards,
Simon
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 14/28] cli: lil: Document structures
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (12 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 15/28] cli: lil: Convert LIL_ENABLE_POOLS to Kconfig Sean Anderson
` (15 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This documents the major structures of LIL.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++
include/cli_lil.h | 57 ++++++++++++++++++++++++++
2 files changed, 157 insertions(+)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 66ee62bf33..7fbae1964a 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -29,20 +29,44 @@
#define HASHMAP_CELLS 256
#define HASHMAP_CELLMASK 0xFF
+/**
+ * struct hashentry - An entry in a cell
+ * @k: The key
+ * @v: The value
+ */
struct hashentry {
char *k;
void *v;
};
+/**
+ * struct hashcell - A list of entries with the same hash
+ * @e: An array of entries
+ * @c: The capacity of this cell
+ */
struct hashcell {
struct hashentry *e;
size_t c;
};
+/**
+ * struct hashmap - An array-based hash map
+ * @cell: An array of cells
+ *
+ * This hash map uses separate chaining with an array for each cell. We cannot
+ * use the existing hash map functions (hsearch_r et al.) because they assume
+ * that the value is a string.
+ */
struct hashmap {
struct hashcell cell[HASHMAP_CELLS];
};
+/**
+ * struct lil_value - A string and its length
+ * @l: The length of the value.
+ * @c: The capacity of this value (maximum of @l before requiring reallocation).
+ * @d: The contents of the value, as a nul-terminated string.
+ */
struct lil_value {
size_t l;
#ifdef LIL_ENABLE_POOLS
@@ -51,12 +75,34 @@ struct lil_value {
char *d;
};
+/**
+ * struct lil_var - A variable
+ * @n: The name of this variable
+ * @env: The environment containing this variable
+ * @v: The value of this variable
+ */
struct lil_var {
char *n;
struct lil_env *env;
struct lil_value *v;
};
+/**
+ * struct lil_env - A function call's execution environment
+ * @parent: The parent of this environment. This is %NULL for the root
+ * environment.
+ * @func: The currently-executing function
+ * @proc: The name of the currently-executing built-in procedure
+ * @var: A list of the variables in this environment
+ * @vars: The number of variables in @var
+ * @varmap: A hash map mapping variable names to pointers to variables in @var
+ * @retval: The value set by "return" or "result"
+ * @retval_set: Whether @retval has been set
+ * @breakrun: Whether to immediately break out the current run of execution.
+ * This is set by "return"
+ *
+ * Variables are inherited from @parent.
+ */
struct lil_env {
struct lil_env *parent;
struct lil_func *func;
@@ -69,12 +115,28 @@ struct lil_env {
int breakrun;
};
+/**
+ * struct list - A list of values
+ * @v: A list of pointers to &struct lil_value
+ * @c: The number of values in this list
+ * @cap: The space allocated for @v
+ */
struct lil_list {
struct lil_value **v;
size_t c;
size_t cap;
};
+/**
+ * struct lil_func - A function which may be evaluated with a list of arguments
+ * @name: The name of the function
+ * @code: A value containing LIL code to be evaluated
+ * @argnames: The names of variables to assign to the passed-in arguments
+ * @proc: A C function to use to evaluate this function
+ *
+ * @code and @argnames are only used to evaluate the function if @proc is %NULL.
+ * If @argnames is empty, then the arguments are assigned to a variable "args".
+ */
struct lil_func {
char *name;
struct lil_value *code;
@@ -82,6 +144,31 @@ struct lil_func {
lil_func_proc_t proc;
};
+/**
+ * struct lil - The current state of the interpreter
+ * @code: The code which is being interpreted
+ * @rootcode: The top-level code (e.g. the code for the initial call to
+ * lil_parse())
+ * @clen: The length of @code
+ * @head: The first uninterpreted part of this code, as an index of @code
+ * @ignoreeol: Whether to treat newlines as whitespace or command terminators
+ * @cmd: A list of the current commands
+ * @cmds: The number of commands in @cmd
+ * @cmdmap: A hash map mapping command names to pointers to commands in @cmd
+ * @env: The current environment to evaluate commands in
+ * @rootenv: The top-level "root" environment
+ * @downenv: The original environment after a call to "topeval" or "upeval"
+ * @empty: An empty value, allocated once
+ * @ERROR_NOERROR: There is no error.
+ * @ERROR_DEFAULT: There was an error.
+ * @ERROR_FIXHEAD: There was an error, but @err_head needs to be fixed.
+ * @ERROR_UNBALANCED: An opening quote or bracket lacks a balancing closing
+ * quote or bracket.
+ * @error: The current error status
+ * @err_head: The offset in @code which caused the @error
+ * @err_msg: An optional string describing the current @error
+ * @parse_depth: The depth of recursive function calls
+ */
struct lil {
const char *code; /* need save on parse */
const char *rootcode;
@@ -101,6 +188,19 @@ struct lil {
size_t parse_depth;
};
+/**
+ * struct expreval - A (mathematical) expression being evaluated
+ * @code: The complete code for this expression
+ * @len: The length of @code
+ * @head: The first unevaluated part of this expression, as an index of @code
+ * @ival: The integer value of this expression
+ * @EERR_NO_ERROR: There is no error
+ * @EERR_SYNTAX_ERROR: Syntax error. For now this is just mismatched
+ * parentheses.
+ * @EERR_DIVISION_BY_ZERO: Attempted division by zero
+ * @EERR_INVALID_EXPRESSION: A non-number was present
+ * @error: The error of this expression (if any)
+ */
struct expreval {
const char *code;
size_t len, head;
diff --git a/include/cli_lil.h b/include/cli_lil.h
index cdaa79fd15..91e79c12f4 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -13,10 +13,35 @@
#define LIL_VERSION_STRING "0.1"
+/**
+ * enum lil_setvar - The strategy to use when creating new variables
+ */
enum lil_setvar {
+ /**
+ * @LIL_SETVAR_GLOBAL: Set in the root environment
+ */
LIL_SETVAR_GLOBAL = 0,
+ /**
+ * @LIL_SETVAR_LOCAL: Set, starting with the local environment
+ *
+ * Search for a variable. If one is found, overwrite it. Otherwise,
+ * create a new variable in the local environment.
+ */
LIL_SETVAR_LOCAL,
+ /**
+ * @LIL_SETVAR_LOCAL_NEW: Create in the local environment
+ *
+ * Create a new variable in the local environment. This never overrides
+ * existing variables (even if one exists in the local environment).
+ */
LIL_SETVAR_LOCAL_NEW,
+ /**
+ * @LIL_SETVAR_LOCAL_ONLY: Set in a local environment only
+ *
+ * Search for a variable in the local environment. If one is found,
+ * overwrite it. Otherwise, create a new variable in the local
+ * environment.
+ */
LIL_SETVAR_LOCAL_ONLY,
};
@@ -32,9 +57,41 @@ struct lil;
typedef struct lil_value *(*lil_func_proc_t)(struct lil *lil, size_t argc,
struct lil_value **argv);
+/**
+ * struct lil_callbacks - Functions called by LIL to allow overriding behavior
+ */
struct lil_callbacks {
+ /**
+ * @setvar: Called when a non-existent global variable is assigned
+ *
+ * @lil: The LIL interpreter
+ *
+ * @name: The name of the variable
+ *
+ * @value: A pointer to the value which would be assigned. This may be
+ * modified to assign a different value.
+ *
+ * This can be used to override or cancel the assignment of a variable
+ *
+ * @Return: A negative value to prevent the assignment, %0 to assign the
+ * original value, or a positive value to assign @value.
+ */
int (*setvar)(struct lil *lil, const char *name,
struct lil_value **value);
+ /**
+ * @getvar: Called when a global variable is read
+ *
+ * @lil: The LIL interpreter
+ *
+ * @name The name of the variable
+ *
+ * @value: The pointer to the value which would be read. This may be
+ * modified to read a different value.
+ *
+ * @Return: A non-zero value to read the value pointed to by @value
+ * instead of the original value, or anything else to use the
+ * original value
+ */
int (*getvar)(struct lil *lil, const char *name,
struct lil_value **value);
};
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 15/28] cli: lil: Convert LIL_ENABLE_POOLS to Kconfig
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (13 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 14/28] cli: lil: Document structures Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:15 ` [RFC PATCH 16/28] cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig Sean Anderson
` (14 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds a Kconfig option to enable memory pools for some commonly-created
and destroyed LIL structures. There is still some significant code
duplication, so perhaps these functions can be refactored further to all
use common pool functions.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
cmd/Kconfig | 7 ++
common/cli_lil.c | 275 +++++++++++++++++++++++------------------------
2 files changed, 143 insertions(+), 139 deletions(-)
diff --git a/cmd/Kconfig b/cmd/Kconfig
index b61a7557a9..28a387b380 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -46,6 +46,13 @@ config LIL_FULL
This enables all LIL builtin functions, as well as expression support
for arithmetic and bitwise operations.
+config LIL_POOLS
+ bool "Use memory pools for LIL structures"
+ help
+ Enable pools for reusing values, lists, and environments. This will
+ use more memory but will cause considerably less memory fragmentation
+ and improve the script execution performance.
+
endif
endif
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 7fbae1964a..750a085f63 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -17,11 +17,6 @@
#include <stdio.h>
#include <string.h>
-/* Enable pools for reusing values, lists and environments. This will use more memory and
- * will rely on the runtime/OS to free the pools once the program ends, but will cause
- * considerably less memory fragmentation and improve the script execution performance. */
-/*#define LIL_ENABLE_POOLS*/
-
/* Enable limiting recursive calls to lil_parse - this can be used to avoid call stack
* overflows and is also useful when running through an automated fuzzer like AFL */
/*#define LIL_ENABLE_RECLIMIT 10000*/
@@ -69,7 +64,7 @@ struct hashmap {
*/
struct lil_value {
size_t l;
-#ifdef LIL_ENABLE_POOLS
+#ifdef CONFIG_LIL_POOLS
size_t c;
#endif
char *d;
@@ -223,14 +218,12 @@ static void lil_set_error_oom(struct lil *lil);
static void lil_set_error_nocmd(struct lil *lil, const char *cmdname);
static void lil_set_error_intr(struct lil *lil);
-#ifdef LIL_ENABLE_POOLS
-static struct lil_value **pool;
-static int poolsize, poolcap;
-static struct lil_list **listpool;
-static size_t listpoolsize, listpoolcap;
-static struct lil_env **envpool;
-static size_t envpoolsize, envpoolcap;
-#endif
+static __maybe_unused struct lil_value **pool;
+static __maybe_unused int poolsize, poolcap;
+static __maybe_unused struct lil_list **listpool;
+static __maybe_unused size_t listpoolsize, listpoolcap;
+static __maybe_unused struct lil_env **envpool;
+static __maybe_unused size_t envpoolsize, envpoolcap;
static unsigned long hm_hash(const char *key)
{
@@ -298,7 +291,7 @@ static int hm_has(struct hashmap *hm, const char *key)
return 0;
}
-#ifdef LIL_ENABLE_POOLS
+#ifdef CONFIG_LIL_POOLS
static struct lil_value *alloc_from_pool(void)
{
if (poolsize > 0) {
@@ -327,39 +320,48 @@ static void ensure_capacity(struct lil_value *val, size_t cap)
val->d = realloc(val->d, val->c);
}
}
+#else
+static struct lil_value *alloc_from_pool(void)
+{
+ return NULL;
+}
+
+static void release_to_pool(struct lil_value *val) { }
+static void ensure_capacity(struct lil_value *val, size_t cap) { }
#endif
static struct lil_value *alloc_value_len(const char *str, size_t len)
{
-#ifdef LIL_ENABLE_POOLS
- struct lil_value *val = alloc_from_pool();
-#else
- struct lil_value *val = calloc(1, sizeof(struct lil_value));
-#endif
+ struct lil_value *val;
+ if (IS_ENABLED(CONFIG_LIL_POOLS))
+ val = alloc_from_pool();
+ else
+ val = calloc(1, sizeof(struct lil_value));
if (!val)
return NULL;
+
if (str) {
val->l = len;
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, len + 1);
-#else
- val->d = malloc(len + 1);
- if (!val->d) {
- free(val);
- return NULL;
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, len + 1);
+ } else {
+ val->d = malloc(len + 1);
+ if (!val->d) {
+ free(val);
+ return NULL;
+ }
}
-#endif
memcpy(val->d, str, len);
val->d[len] = 0;
} else {
val->l = 0;
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, 1);
- val->d[0] = '\0';
-#else
- val->d = NULL;
-#endif
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, 1);
+ val->d[0] = '\0';
+ } else {
+ val->d = NULL;
+ }
}
return val;
}
@@ -375,76 +377,72 @@ struct lil_value *lil_clone_value(struct lil_value *src)
if (!src)
return NULL;
-#ifdef LIL_ENABLE_POOLS
- val = alloc_from_pool();
-#else
- val = calloc(1, sizeof(struct lil_value));
-#endif
+
+ if (IS_ENABLED(CONFIG_LIL_POOLS))
+ val = alloc_from_pool();
+ else
+ val = calloc(1, sizeof(struct lil_value));
if (!val)
return NULL;
val->l = src->l;
if (src->l) {
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, val->l + 1);
-#else
- val->d = malloc(val->l + 1);
- if (!val->d) {
- free(val);
- return NULL;
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, val->l + 1);
+ } else {
+ val->d = malloc(val->l + 1);
+ if (!val->d) {
+ free(val);
+ return NULL;
+ }
}
-#endif
memcpy(val->d, src->d, val->l + 1);
} else {
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, 1);
- val->d[0] = '\0';
-#else
- val->d = NULL;
-#endif
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, 1);
+ val->d[0] = '\0';
+ } else {
+ val->d = NULL;
+ }
}
return val;
}
int lil_append_char(struct lil_value *val, char ch)
{
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, val->l + 2);
- val->d[val->l++] = ch;
- val->d[val->l] = '\0';
-#else
- char *new = realloc(val->d, val->l + 2);
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, val->l + 2);
+ val->d[val->l++] = ch;
+ val->d[val->l] = '\0';
+ } else {
+ char *new = realloc(val->d, val->l + 2);
- if (!new)
- return 0;
+ if (!new)
+ return 0;
- new[val->l++] = ch;
- new[val->l] = 0;
- val->d = new;
-#endif
+ new[val->l++] = ch;
+ new[val->l] = 0;
+ val->d = new;
+ }
return 1;
}
int lil_append_string_len(struct lil_value *val, const char *s, size_t len)
{
-#ifndef LIL_ENABLE_POOLS
- char *new;
-#endif
-
if (!s || !s[0])
return 1;
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, val->l + len + 1);
- memcpy(val->d + val->l, s, len + 1);
-#else
- new = realloc(val->d, val->l + len + 1);
- if (!new)
- return 0;
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, val->l + len + 1);
+ memcpy(val->d + val->l, s, len + 1);
+ } else {
+ char *new = realloc(val->d, val->l + len + 1);
- memcpy(new + val->l, s, len + 1);
- val->d = new;
-#endif
+ if (!new)
+ return 0;
+ memcpy(new + val->l, s, len + 1);
+ val->d = new;
+ }
val->l += len;
return 1;
}
@@ -456,24 +454,20 @@ int lil_append_string(struct lil_value *val, const char *s)
int lil_append_val(struct lil_value *val, struct lil_value *v)
{
-#ifndef LIL_ENABLE_POOLS
- char *new;
-#endif
-
if (!v || !v->l)
return 1;
-#ifdef LIL_ENABLE_POOLS
- ensure_capacity(val, val->l + v->l + 1);
- memcpy(val->d + val->l, v->d, v->l + 1);
-#else
- new = realloc(val->d, val->l + v->l + 1);
- if (!new)
- return 0;
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ ensure_capacity(val, val->l + v->l + 1);
+ memcpy(val->d + val->l, v->d, v->l + 1);
+ } else {
+ char *new = realloc(val->d, val->l + v->l + 1);
- memcpy(new + val->l, v->d, v->l + 1);
- val->d = new;
-#endif
+ if (!new)
+ return 0;
+ memcpy(new + val->l, v->d, v->l + 1);
+ val->d = new;
+ }
val->l += v->l;
return 1;
}
@@ -483,22 +477,21 @@ void lil_free_value(struct lil_value *val)
if (!val)
return;
-#ifdef LIL_ENABLE_POOLS
- release_to_pool(val);
-#else
- free(val->d);
- free(val);
-#endif
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ release_to_pool(val);
+ } else {
+ free(val->d);
+ free(val);
+ }
}
struct lil_list *lil_alloc_list(void)
{
struct lil_list *list;
-#ifdef LIL_ENABLE_POOLS
- if (listpoolsize > 0)
+ if (IS_ENABLED(CONFIG_LIL_POOLS) && listpoolsize > 0)
return listpool[--listpoolsize];
-#endif
+
list = calloc(1, sizeof(struct lil_list));
list->v = NULL;
return list;
@@ -514,19 +507,21 @@ void lil_free_list(struct lil_list *list)
for (i = 0; i < list->c; i++)
lil_free_value(list->v[i]);
-#ifdef LIL_ENABLE_POOLS
- list->c = 0;
- if (listpoolsize == listpoolcap) {
- listpoolcap =
- listpoolcap ? (listpoolcap + listpoolcap / 2) : 32;
- listpool = realloc(listpool,
- sizeof(struct lil_list *) * listpoolcap);
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ list->c = 0;
+ if (listpoolsize == listpoolcap) {
+ if (listpoolcap)
+ listpoolcap += listpoolcap / 2;
+ else
+ listpoolcap = 32;
+ listpool = realloc(listpool,
+ sizeof(*listpool) * listpoolcap);
+ }
+ listpool[listpoolsize++] = list;
+ } else {
+ free(list->v);
+ free(list);
}
- listpool[listpoolsize++] = list;
-#else
- free(list->v);
- free(list);
-#endif
}
void lil_list_append(struct lil_list *list, struct lil_value *val)
@@ -603,8 +598,7 @@ struct lil_env *lil_alloc_env(struct lil_env *parent)
{
struct lil_env *env;
-#ifdef LIL_ENABLE_POOLS
- if (envpoolsize > 0) {
+ if (IS_ENABLED(CONFIG_LIL_POOLS) && envpoolsize > 0) {
size_t i, j;
env = envpool[--envpoolsize];
@@ -622,7 +616,7 @@ struct lil_env *lil_alloc_env(struct lil_env *parent)
}
return env;
}
-#endif
+
env = calloc(1, sizeof(struct lil_env));
env->parent = parent;
return env;
@@ -636,30 +630,33 @@ void lil_free_env(struct lil_env *env)
return;
lil_free_value(env->retval);
-#ifdef LIL_ENABLE_POOLS
- for (i = 0; i < env->vars; i++) {
- free(env->var[i]->n);
- lil_free_value(env->var[i]->v);
- free(env->var[i]);
- }
- free(env->var);
+ if (IS_ENABLED(CONFIG_LIL_POOLS)) {
+ for (i = 0; i < env->vars; i++) {
+ free(env->var[i]->n);
+ lil_free_value(env->var[i]->v);
+ free(env->var[i]);
+ }
+ free(env->var);
- if (envpoolsize == envpoolcap) {
- envpoolcap = envpoolcap ? (envpoolcap + envpoolcap / 2) : 64;
- envpool =
- realloc(envpool, sizeof(struct lil_env *) * envpoolcap);
+ if (envpoolsize == envpoolcap) {
+ if (envpoolcap)
+ envpoolcap += envpoolcap / 2;
+ else
+ envpoolcap = 64;
+ envpool = realloc(envpool,
+ sizeof(*envpool) * envpoolcap);
+ }
+ envpool[envpoolsize++] = env;
+ } else {
+ hm_destroy(&env->varmap);
+ for (i = 0; i < env->vars; i++) {
+ free(env->var[i]->n);
+ lil_free_value(env->var[i]->v);
+ free(env->var[i]);
+ }
+ free(env->var);
+ free(env);
}
- envpool[envpoolsize++] = env;
-#else
- hm_destroy(&env->varmap);
- for (i = 0; i < env->vars; i++) {
- free(env->var[i]->n);
- lil_free_value(env->var[i]->v);
- free(env->var[i]);
- }
- free(env->var);
- free(env);
-#endif
}
static struct lil_var *lil_find_local_var(struct lil *lil, struct lil_env *env,
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 16/28] cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (14 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 15/28] cli: lil: Convert LIL_ENABLE_POOLS to Kconfig Sean Anderson
@ 2021-07-01 6:15 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 17/28] test: Add tests for LIL Sean Anderson
` (13 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:15 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds a configuration option to set the recursion limit. I've set it to
a (conservative) 1000 by default. In addition, there is an option to turn
it off for a very minor space savings and performance increase.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
Do we need this? Perhaps it should default to 0.
cmd/Kconfig | 8 ++++++++
common/cli_lil.c | 10 ++--------
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/cmd/Kconfig b/cmd/Kconfig
index 28a387b380..7c8962cfc2 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -53,6 +53,14 @@ config LIL_POOLS
use more memory but will cause considerably less memory fragmentation
and improve the script execution performance.
+config LIL_RECLIMIT
+ int "LIL function recursion limit"
+ default 1000
+ help
+ Enable limiting recursive calls to lil_parse - this can be used to
+ avoid call stack overflows and is also useful when running through an
+ automated fuzzer like AFL. Set to 0 to disable the recursion limit.
+
endif
endif
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 750a085f63..6c05531441 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -17,10 +17,6 @@
#include <stdio.h>
#include <string.h>
-/* Enable limiting recursive calls to lil_parse - this can be used to avoid call stack
- * overflows and is also useful when running through an automated fuzzer like AFL */
-/*#define LIL_ENABLE_RECLIMIT 10000*/
-
#define HASHMAP_CELLS 256
#define HASHMAP_CELLMASK 0xFF
@@ -1198,12 +1194,10 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
lil_skip_spaces(lil);
lil->parse_depth++;
-#ifdef LIL_ENABLE_RECLIMIT
- if (lil->parse_depth > LIL_ENABLE_RECLIMIT) {
- lil_set_error(lil, LIL_ERR_DEPTH, "Too many recursive calls");
+ if (CONFIG_LIL_RECLIMIT && lil->parse_depth > CONFIG_LIL_RECLIMIT) {
+ lil_set_error(lil, LIL_ERR_DEPTH, "recursion limit reached");
goto cleanup;
}
-#endif
if (lil->parse_depth == 1)
lil->err = LIL_ERR_NONE;
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 17/28] test: Add tests for LIL
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (15 preceding siblings ...)
2021-07-01 6:15 ` [RFC PATCH 16/28] cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-05 15:29 ` Simon Glass
2021-07-01 6:16 ` [RFC PATCH 18/28] cli: lil: Remove duplicate function bodies Sean Anderson
` (12 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This tests several aspects of the parser. These tests are primarily adapted
from the *.lil code examples included with upstream LIL. These tests should
probably get their own category, especially if I add additional styles of
tests.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
Yes, I know checkpatch complains about the quoted string being split, but
that warning is intended for user-visible strings.
MAINTAINERS | 1 +
test/cmd/Makefile | 1 +
test/cmd/lil.c | 339 ++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 341 insertions(+)
create mode 100644 test/cmd/lil.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 0184de5f93..3bf460127b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -771,6 +771,7 @@ M: Sean Anderson <seanga2@gmail.com>
S: Maintained
F: common/cli_lil.c
F: include/cli_lil.h
+F: test/cmd/lil.c
LOGGING
M: Simon Glass <sjg@chromium.org>
diff --git a/test/cmd/Makefile b/test/cmd/Makefile
index 2cfe43a6bd..4f7440cb44 100644
--- a/test/cmd/Makefile
+++ b/test/cmd/Makefile
@@ -5,6 +5,7 @@
ifdef CONFIG_HUSH_PARSER
obj-$(CONFIG_CONSOLE_RECORD) += test_echo.o
endif
+obj-$(CONFIG_LIL_FULL) += lil.o
obj-y += mem.o
obj-$(CONFIG_CMD_ADDRMAP) += addrmap.o
obj-$(CONFIG_CMD_MEM_SEARCH) += mem_search.o
diff --git a/test/cmd/lil.c b/test/cmd/lil.c
new file mode 100644
index 0000000000..896b2fed15
--- /dev/null
+++ b/test/cmd/lil.c
@@ -0,0 +1,339 @@
+// SPDX-License-Identifier: GPL-2.0+ AND Zlib
+/*
+ * Copyright (C) 2021 Sean Anderson <seanga2@gmail.com>
+ * Copyright (C) 2010-2021 Kostas Michalopoulos
+ *
+ * This file contains code which originated from the LIL project, licensed under
+ * Zlib. All modifications are licensed under GPL-2.0+
+ */
+
+#include <common.h>
+#include <asm/global_data.h>
+#include <cli_lil.h>
+#include <test/lib.h>
+#include <test/test.h>
+#include <test/ut.h>
+
+const char helpers[] =
+ "proc assert {cond} {"
+ "if not [upeval expr [set cond]] {"
+ "error [set cond]"
+ "}"
+ "};"
+ "proc assert_err {cmd} {"
+ "set ok 1;"
+ "try {upeval $cmd; set ok 0} {};"
+ "assert {$ok};"
+ "};"
+ "proc asserteq {expr1 expr2} {"
+ "set val1 [upeval 'expr \"$expr1\"'];"
+ "set val2 [upeval 'expr \"$expr2\"'];"
+ "if {$val1 != $val2} {"
+ "error '$expr1 == ${expr2}: "
+ "Expected ${val1}, got $val2'"
+ "}"
+ "};"
+ "proc asserteq_str {expr1 expr2} {"
+ "set val1 [upeval 'subst \"$expr1\"'];"
+ "set val2 [upeval 'subst \"$expr2\"'];"
+ "if not [streq $val1 $val2] {"
+ "error '$expr1 == ${expr2}: "
+ "Expected ${val1}, got $val2'"
+ "}"
+ "};"
+ "proc asserteq_list {xs ys} {"
+ "set xlen [count $xs];"
+ "set ylen [count $ys];"
+ "if not {$xlen == $ylen} {"
+ "error '\\[count ${xs}\\] == \\[count ${ys}\\]: "
+ "Expected ${xlen}, got $ylen'\n"
+ "};"
+ "for {set i 0} {$i < $xlen} {incr i} {"
+ "set x [index $xs $i];"
+ "set y [index $ys $i];"
+ "if not {[streq $x $y]} {"
+ "error '$xs == ${ys}: "
+ "Expected $x at ${i}, got $y'"
+ "}"
+ "}"
+ "}";
+
+static const struct {
+ const char *name;
+ const char *cmd;
+} lil_tests[] = {
+ {"and",
+ "proc and args {"
+ "foreach [slice $args 1] {"
+ "upeval 'downeval \\'set v \\'\\[${i}\\]';"
+ "if not $v { return 0 }"
+ "};"
+ "return 1"
+ "};"
+ "set a 0;"
+ "set final [and {set a 3} {return 0} {set a 32}];"
+ "asserteq 0 {$final};"
+ "assert 3 {$a};"
+ },
+ {"assert",
+ "assert 1;"
+ "assert_err {assert 0};"
+ "asserteq 1 1;"
+ "assert_err {asserteq 1 0};"
+ "asserteq_str {string one} {string one};"
+ "assert_err {asserteq_str {string one} {string two}};"
+ "asserteq_list [list 1 2 3] [list 1 2 3];"
+ "assert_err {asserteq_list [list 1 2] [list 1 2 3]};"
+ "assert_err {asserteq_list [list 1 2 3] [list 1 2]};"
+ "assert_err {asserteq_list [list 1 2 3] [list 1 2 4]};"
+ },
+ {"downeval",
+ "proc grab-some-list {} {"
+ "set items {};"
+ "upeval {"
+ "foreach $some-list {"
+ "downeval 'append items $i'"
+ "}"
+ "};"
+ "return $items"
+ "};"
+ "set some-list [list foo bar baz blah moo boo];"
+ "asserteq_list $some-list [grab-some-list]"
+ },
+ {"expr",
+ "asserteq 7 {1 + ( 2 * 3 )};"
+ "asserteq 7 {1+(2*3)};"
+ "asserteq -6 {1+ ~(2*3)};"
+ "asserteq -6 {1 + ~( 2 * 3 )};"
+ "asserteq -6 {1 +~ (2*3 )};"
+ "asserteq -6 {~(2*3)+1};"
+ "asserteq 0 {1*!(2+2)};"
+ "asserteq -1 {~!(!{})};"
+ "asserteq 1 {1 +~*(2*3)};"
+ "asserteq 1 {'hello'};"
+ "asserteq 0 {0};"
+ "asserteq 0 {{}};"
+ "asserteq 1 {()};"
+ "asserteq 1 {( )};"
+ "asserteq_str '' {[expr]};"
+ },
+ {"factorial",
+ "proc fact {n} {"
+ "if {$n} {"
+ "expr {$n * [fact [expr {$n - 1}]]}"
+ "} {"
+ "return 1"
+ "}"
+ "};"
+ "asserteq 1 {[fact 0]};"
+ "asserteq 1 {[fact 1]};"
+ "asserteq 6 {[fact 3]};"
+ "asserteq 3628800 {[fact 10]};"
+ "asserteq 2432902008176640000 {[fact 20]}"
+ },
+ {"filter",
+ "set short_procs [filter [reflect procs] {[length $x] < 5}];"
+ "foreach $short_procs {assert {[length $i] < 5}}"
+ },
+ {"funcs",
+ "proc lapply {list proc} {"
+ "set ret {};"
+ "foreach $list {"
+ "append ret [$proc $i];"
+ "};"
+ "return $ret"
+ "};"
+ "set list [list {bad's day} {good's day} eh??];"
+ "asserteq_list [lapply $list split] [list "
+ "[list {bad's} day] "
+ "[list {good's} day] "
+ "[list eh??]"
+ "];"
+ "asserteq_list [lapply $list length] [list 9 10 4];"
+ "asserteq_list [lapply $list [proc {a} {"
+ "return [index [split $a] 0]"
+ "}]] [list {bad's} {good's} eh??]"
+ },
+ {"lists",
+ "set l [list foo bar baz bad];"
+ "asserteq_str baz {[index $l 2]};"
+ "append l 'Hello, world!';"
+ "asserteq_list $l [list foo bar baz bad 'Hello, world!'];"
+ "set l [subst $l];"
+ "asserteq_list $l [list foo bar baz bad Hello, world!];"
+ "lmap $l foox barx bamia;"
+ "asserteq_str foo {$foox};"
+ "asserteq_str bar {$barx};"
+ "asserteq_str baz {$bamia};"
+ "set l {one # linebreaks are ignored in list parsing mode\n"
+ "\n"
+ "two;three # a semicolon still counts as line break\n"
+ " # (which in list mode is treated as a\n"
+ " # separator for list entries)\n"
+ "# of course a semicolon inside quotes is treated like normal\n"
+ "three';'and';a;half'\n"
+ "# like in code mode, a semicolon will stop the comment; four\n"
+ "\n"
+ "# below we have a quote, square brackets for inline\n"
+ "# expansions are still taken into consideration\n"
+ "[quote {this line will be ignored completely\n"
+ " as will this line and instead be replaced\n"
+ " with the 'five' below since while in code\n"
+ " mode (that is, inside the brackets here)\n"
+ " linebreaks are still processed}\n"
+ " quote five]\n"
+ "\n"
+ "# The curly brackets are also processed so the next three\n"
+ "# lines will show up as three separate lines\n"
+ "{six\n"
+ "seven\n"
+ "eight}}\n"
+ "asserteq_list $l [list one two three 'three;and;a;half' four "
+ "five 'six\\nseven\\neight'];"
+ },
+ {"local",
+ "proc bits-for {x} {"
+ "local y bits;"
+ "set y 0 bits 0;"
+ "while {$y <= $x} {"
+ "incr bits;"
+ "set y [expr 1 << $bits]"
+ "};"
+ "return $bits"
+ "};"
+ "set y 1001;"
+ "set bits [bits-for $y];"
+ "set x 45;"
+ "set bitsx [bits-for $x];"
+ "asserteq 1001 {$y};"
+ "asserteq 10 {$bits};"
+ "asserteq 45 {$x};"
+ "asserteq 6 {$bitsx}"
+ },
+ {"multiline comment",
+ "# this line will not be executed, but the following will\n"
+ "set ok1 1\n"
+ "## This is a multiline comment\n"
+ " which, as the name implies,\n"
+ " spans multiple lines.\n"
+ "set ok2 1\n"
+ " the code above wouldn't execute,\n"
+ " but this will --> ##set ok3 1\n"
+ "### more than two #s will not count as multiline comments\n"
+ "set ok4 1\n"
+ "# Note that semicolons can be used as linebreaks so\n"
+ "# this code will be executed: ; set ok5 1\n"
+ "##\n"
+ " ...however inside multiline comments semicolons do not\n"
+ " stop the comment section (pretty much like linebreaks)\n"
+ " and this code will not be executed: ; set ok6 1\n"
+ "##\n"
+ "# Also note that unlike in regular code, semicolons cannot\n"
+ "# be escaped in single-line comments, e.g.: ; set ok7 1\n"
+ "asserteq_str 1 {$ok1};"
+ "assert {![reflect has-var ok2]}"
+ "asserteq_str 1 {$ok3};"
+ "asserteq_str 1 {$ok4};"
+ "asserteq_str 1 {$ok5};"
+ "assert {![reflect has-var ok6]}"
+ "asserteq_str 1 {$ok7};"
+ },
+ {"multiline code",
+ "asserteq_list [list hello \\\n"
+ " world] [list hello world]"
+ },
+ {"return",
+ "proc uses_return {} {"
+ "return 1;"
+ "return 0;"
+ "};"
+ "proc doesnt_use_return {} {"
+ "quote 1;"
+ "};"
+ "proc uses_result {} {"
+ "result 1;"
+ "quote 0;"
+ "};"
+ "assert {[uses_return]};"
+ "assert {[doesnt_use_return]};"
+ "assert {[uses_result]}"
+ },
+ {"strings",
+ "set a 'This is a string';"
+ "set b 'This is another string';"
+ "asserteq 16 {[length $a]};"
+ "asserteq 22 {[length $b]};"
+ "asserteq_str a {[charat $a [expr [length $a] / 2]]};"
+ "asserteq_str t {[charat $b [expr [length $b] / 2]]};"
+ "asserteq 97 {[codeat $a [expr [length $a] / 2]]};"
+ "asserteq 116 {[codeat $b [expr [length $b] / 2]]};"
+ "asserteq 10 {[strpos $a string]};"
+ "asserteq 16 {[strpos $b string]};"
+ "asserteq -78 {[compare $a $b]};"
+ "assert {![streq $a $b]};"
+ "asserteq_str 'This is a foo' {[repstr $a string foo]};"
+ "asserteq_str 'This is another foo' {[repstr $b string foo]};"
+ "asserteq_list [split $a] [list This is a string];"
+ "asserteq_list [split $b] [list This is another string];"
+ },
+ {"topeval",
+ "proc does-something {} {"
+ "topeval {"
+ "asserteq 10 {$x};"
+ "set x 42;"
+ "downeval {set y [expr $x * 10]}"
+ "};"
+ "asserteq 420 {$y}"
+ "};"
+ "proc calls-something {} {"
+ "local x;"
+ "set x 33;"
+ "does-something;"
+ "asserteq 33 {$x};"
+ "asserteq 420 {$y}"
+ "};"
+ "set x 10;"
+ "set y 20;"
+ "calls-something;"
+ "asserteq 42 {$x};"
+ "asserteq 420 {$y}"
+ },
+ {"trim",
+ "set str ' Hello, world! ';"
+ "asserteq_str 'Hello, world!' {[trim $str]};"
+ "asserteq_str 'Hello, world! ' {[ltrim $str]};"
+ "asserteq_str ' Hello, world!' {[rtrim $str]};"
+ "asserteq_str 'Hello world' {[foreach [split $str] {"
+ "quote [trim $i {,!}]"
+ "}]};"
+ "asserteq_str 'Hello world' {[filter [split $str {,! }] {"
+ "[length $x] > 0"
+ "}]};"
+ },
+};
+
+static int lib_test_lil(struct unit_test_state *uts)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(lil_tests); i++) {
+ const char *err_msg;
+ enum lil_error err;
+ struct lil *lil = lil_new(NULL);
+
+ lil_free_value(lil_parse(lil, helpers, sizeof(helpers) - 1, 0));
+ ut_asserteq(LIL_ERR_NONE, lil_error(lil, &err_msg));
+ lil_free_value(lil_parse(lil, lil_tests[i].cmd, 0, 0));
+ err = lil_error(lil, &err_msg);
+ if (err) {
+ ut_failf(uts, __FILE__, __LINE__, __func__,
+ lil_tests[i].name, "err=%d: %s", err, err_msg);
+ lil_free(lil);
+ return CMD_RET_FAILURE;
+ };
+ lil_free(lil);
+ }
+
+ return 0;
+}
+LIB_TEST(lib_test_lil, 0);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 17/28] test: Add tests for LIL
2021-07-01 6:16 ` [RFC PATCH 17/28] test: Add tests for LIL Sean Anderson
@ 2021-07-05 15:29 ` Simon Glass
0 siblings, 0 replies; 107+ messages in thread
From: Simon Glass @ 2021-07-05 15:29 UTC (permalink / raw)
To: Sean Anderson
Cc: U-Boot Mailing List, Tom Rini, Marek Behún, Wolfgang Denk,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On Thu, 1 Jul 2021 at 00:16, Sean Anderson <seanga2@gmail.com> wrote:
>
> This tests several aspects of the parser. These tests are primarily adapted
> from the *.lil code examples included with upstream LIL. These tests should
> probably get their own category, especially if I add additional styles of
> tests.
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
> ---
> Yes, I know checkpatch complains about the quoted string being split, but
> that warning is intended for user-visible strings.
>
> MAINTAINERS | 1 +
> test/cmd/Makefile | 1 +
> test/cmd/lil.c | 339 ++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 341 insertions(+)
> create mode 100644 test/cmd/lil.c
>
Reviewed-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 18/28] cli: lil: Remove duplicate function bodies
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (16 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 17/28] test: Add tests for LIL Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 19/28] cli: lil: Add "symbol" structure Sean Anderson
` (11 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
lil_append_val is just lil_append_string with the string and length taken
from a struct lil_value. Use lil_append_stringh_len to implement both. Do
the same for lil_clone_value.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 51 ++----------------------------------------------
1 file changed, 2 insertions(+), 49 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 6c05531441..5875fbd46b 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -369,39 +369,7 @@ static struct lil_value *alloc_value(const char *str)
struct lil_value *lil_clone_value(struct lil_value *src)
{
- struct lil_value *val;
-
- if (!src)
- return NULL;
-
- if (IS_ENABLED(CONFIG_LIL_POOLS))
- val = alloc_from_pool();
- else
- val = calloc(1, sizeof(struct lil_value));
- if (!val)
- return NULL;
-
- val->l = src->l;
- if (src->l) {
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, val->l + 1);
- } else {
- val->d = malloc(val->l + 1);
- if (!val->d) {
- free(val);
- return NULL;
- }
- }
- memcpy(val->d, src->d, val->l + 1);
- } else {
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, 1);
- val->d[0] = '\0';
- } else {
- val->d = NULL;
- }
- }
- return val;
+ return alloc_value_len(src->d, src->l);
}
int lil_append_char(struct lil_value *val, char ch)
@@ -450,22 +418,7 @@ int lil_append_string(struct lil_value *val, const char *s)
int lil_append_val(struct lil_value *val, struct lil_value *v)
{
- if (!v || !v->l)
- return 1;
-
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, val->l + v->l + 1);
- memcpy(val->d + val->l, v->d, v->l + 1);
- } else {
- char *new = realloc(val->d, val->l + v->l + 1);
-
- if (!new)
- return 0;
- memcpy(new + val->l, v->d, v->l + 1);
- val->d = new;
- }
- val->l += v->l;
- return 1;
+ return lil_append_string_len(val, v->d, v->l);
}
void lil_free_value(struct lil_value *val)
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 19/28] cli: lil: Add "symbol" structure
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (17 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 18/28] cli: lil: Remove duplicate function bodies Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 20/28] cli: lil: Add config to enable debug output Sean Anderson
` (10 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
We need a generic structure to hold symbols parsed by the parser. We would
also like to re-use existing code as much as possible without rewriting
everything. To do this, we hijack the allocators for lil_list and lil_value
and have them allocate enough space for a lil_symbol.
While we're at it, we can make lil_list hold lil_symbols instead of
lil_values. To keep all the old users sane, we just cast back to lil_value
before retrieving the value (with an assert to make sure we're not sending
back something else). Unfortunately, many functions were accessing the
list vector directly, so convert them.
This commit also fixes pools not behaving properly when running out of
memory. This should likely be refactored in the future so everything uses
one set of allocator/free routines to avoid code duplication.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 317 +++++++++++++++++++++++++++++-----------------
include/cli_lil.h | 8 +-
2 files changed, 206 insertions(+), 119 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 5875fbd46b..06fd37c383 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -106,18 +106,45 @@ struct lil_env {
int breakrun;
};
+struct lil_symbol;
+
/**
- * struct list - A list of values
- * @v: A list of pointers to &struct lil_value
- * @c: The number of values in this list
+ * struct list - A list of symbols
+ * @v: A list of pointers to symbols
+ * @c: The number of symbols in this list
* @cap: The space allocated for @v
*/
struct lil_list {
- struct lil_value **v;
+ struct lil_symbol **v;
size_t c;
size_t cap;
};
+/**
+ * struct lil_symbol - A symbol parsed by the parser
+ * @LIL_SYMBOL_VALUE: A plain old string and length
+ * @LIL_SYMBOL_VARIABLE: A name of a variable to be substituted
+ * @LIL_SYMBOL_LIST: A list of symbols
+ * @value: A literal value or name of variable
+ * @list: A list of commands in the script
+ * @word: Another word to be evaluated
+ * @type: The type of word
+ */
+struct lil_symbol {
+ union {
+ struct lil_value value;
+ struct lil_symbol *symbol;
+ struct lil_list list;
+ };
+ enum {
+ LIL_SYMBOL_VALUE = 0,
+ LIL_SYMBOL_LIST,
+ LIL_SYMBOL_VARIABLE,
+ LIL_SYMBOL_COMMAND,
+ LIL_SYMBOL_SCRIPT,
+ } type;
+};
+
/**
* struct lil_func - A function which may be evaluated with a list of arguments
* @name: The name of the function
@@ -287,24 +314,50 @@ static int hm_has(struct hashmap *hm, const char *key)
return 0;
}
-#ifdef CONFIG_LIL_POOLS
+static void lil_free_symbol(struct lil_symbol *sym)
+{
+ switch (sym->type) {
+ case LIL_SYMBOL_VALUE:
+ lil_free_value(&sym->value);
+ return;
+ case LIL_SYMBOL_VARIABLE:
+ lil_free_symbol(sym->symbol);
+ free(sym);
+ return;
+ case LIL_SYMBOL_LIST:
+ case LIL_SYMBOL_COMMAND:
+ case LIL_SYMBOL_SCRIPT:
+ lil_free_list(&sym->list);
+ return;
+ }
+ log_debug("unknown type %d\n", sym->type);
+ assert(0);
+}
+
+#if IS_ENABLED(CONFIG_LIL_POOLS)
static struct lil_value *alloc_from_pool(void)
{
- if (poolsize > 0) {
- poolsize--;
- return pool[poolsize];
- } else {
- struct lil_value *val = calloc(1, sizeof(struct lil_value));
-
- return val;
- }
+ if (poolsize > 0)
+ return pool[--poolsize];
+ else
+ return calloc(1, sizeof(struct lil_symbol));
}
static void release_to_pool(struct lil_value *val)
{
if (poolsize == poolcap) {
- poolcap = poolcap ? (poolcap + poolcap / 2) : 64;
- pool = realloc(pool, sizeof(struct lil_value *) * poolcap);
+ size_t npoolcap = poolcap ? (poolcap + poolcap / 2) : 64;
+ struct lil_value **npool =
+ realloc(pool, sizeof(struct lil_symbol *) * npoolcap);
+
+ if (!npool) {
+ free(val->d);
+ free(val);
+ return;
+ }
+
+ poolcap = npoolcap;
+ pool = npool;
}
pool[poolsize++] = val;
}
@@ -319,45 +372,48 @@ static void ensure_capacity(struct lil_value *val, size_t cap)
#else
static struct lil_value *alloc_from_pool(void)
{
- return NULL;
+ return calloc(1, sizeof(struct lil_symbol));
}
-static void release_to_pool(struct lil_value *val) { }
-static void ensure_capacity(struct lil_value *val, size_t cap) { }
+static void release_to_pool(struct lil_value *val)
+{
+ free(val->d);
+ free(val);
+}
+static void ensure_capacity(struct lil_value *val, size_t cap)
+{
+ val->d = realloc(val->d, cap);
+}
#endif
+static struct lil_symbol *value_to_symbol(struct lil_value *val)
+{
+ return container_of(val, struct lil_symbol, value);
+}
+
static struct lil_value *alloc_value_len(const char *str, size_t len)
{
struct lil_value *val;
- if (IS_ENABLED(CONFIG_LIL_POOLS))
- val = alloc_from_pool();
- else
- val = calloc(1, sizeof(struct lil_value));
+ val = alloc_from_pool();
if (!val)
return NULL;
+ value_to_symbol(val)->type = LIL_SYMBOL_VALUE;
if (str) {
val->l = len;
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, len + 1);
- } else {
- val->d = malloc(len + 1);
- if (!val->d) {
- free(val);
- return NULL;
- }
+ ensure_capacity(val, len + 1);
+ if (!val->d) {
+ release_to_pool(val);
+ return NULL;
}
memcpy(val->d, str, len);
val->d[len] = 0;
} else {
val->l = 0;
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, 1);
+ ensure_capacity(val, 1);
+ if (val->d)
val->d[0] = '\0';
- } else {
- val->d = NULL;
- }
}
return val;
}
@@ -372,53 +428,41 @@ struct lil_value *lil_clone_value(struct lil_value *src)
return alloc_value_len(src->d, src->l);
}
-int lil_append_char(struct lil_value *val, char ch)
+enum lil_error lil_append_char(struct lil_value *val, char ch)
{
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, val->l + 2);
- val->d[val->l++] = ch;
- val->d[val->l] = '\0';
- } else {
- char *new = realloc(val->d, val->l + 2);
+ ensure_capacity(val, val->l + 2);
+ if (!val->d)
+ return LIL_ERR_OOM;
- if (!new)
- return 0;
-
- new[val->l++] = ch;
- new[val->l] = 0;
- val->d = new;
- }
- return 1;
+ val->d[val->l++] = ch;
+ val->d[val->l] = '\0';
+ return LIL_ERR_NONE;
}
-int lil_append_string_len(struct lil_value *val, const char *s, size_t len)
+static enum lil_error lil_append_string_len(struct lil_value *val,
+ const char *s, size_t len)
{
if (!s || !s[0])
- return 1;
+ return LIL_ERR_NONE;
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- ensure_capacity(val, val->l + len + 1);
- memcpy(val->d + val->l, s, len + 1);
- } else {
- char *new = realloc(val->d, val->l + len + 1);
-
- if (!new)
- return 0;
- memcpy(new + val->l, s, len + 1);
- val->d = new;
- }
+ ensure_capacity(val, val->l + len + 1);
+ if (!val->d)
+ return LIL_ERR_OOM;
+ memcpy(val->d + val->l, s, len + 1);
val->l += len;
- return 1;
+ return LIL_ERR_NONE;
}
-int lil_append_string(struct lil_value *val, const char *s)
+enum lil_error lil_append_string(struct lil_value *val, const char *s)
{
return lil_append_string_len(val, s, strlen(s));
}
-int lil_append_val(struct lil_value *val, struct lil_value *v)
+enum lil_error lil_append_val(struct lil_value *val, struct lil_value *v)
{
- return lil_append_string_len(val, v->d, v->l);
+ if (v)
+ return lil_append_string_len(val, v->d, v->l);
+ return LIL_ERR_NONE;
}
void lil_free_value(struct lil_value *val)
@@ -426,23 +470,28 @@ void lil_free_value(struct lil_value *val)
if (!val)
return;
- if (IS_ENABLED(CONFIG_LIL_POOLS)) {
- release_to_pool(val);
- } else {
- free(val->d);
- free(val);
- }
+ release_to_pool(val);
+}
+
+static struct lil_symbol *list_to_symbol(struct lil_list *list)
+{
+ return container_of(list, struct lil_symbol, list);
}
struct lil_list *lil_alloc_list(void)
{
struct lil_list *list;
- if (IS_ENABLED(CONFIG_LIL_POOLS) && listpoolsize > 0)
- return listpool[--listpoolsize];
+ if (IS_ENABLED(CONFIG_LIL_POOLS) && listpoolsize > 0) {
+ list = listpool[--listpoolsize];
+ } else {
+ list = calloc(1, sizeof(struct lil_symbol));
+ if (!list)
+ return list;
+ list->v = NULL;
+ }
- list = calloc(1, sizeof(struct lil_list));
- list->v = NULL;
+ list_to_symbol(list)->type = LIL_SYMBOL_LIST;
return list;
}
@@ -454,49 +503,61 @@ void lil_free_list(struct lil_list *list)
return;
for (i = 0; i < list->c; i++)
- lil_free_value(list->v[i]);
+ lil_free_symbol(list->v[i]);
if (IS_ENABLED(CONFIG_LIL_POOLS)) {
list->c = 0;
if (listpoolsize == listpoolcap) {
+ int ncap;
+ struct lil_list **npool;
+
if (listpoolcap)
- listpoolcap += listpoolcap / 2;
+ ncap += listpoolcap / 2;
else
- listpoolcap = 32;
- listpool = realloc(listpool,
- sizeof(*listpool) * listpoolcap);
+ ncap = 32;
+ npool = realloc(listpool, sizeof(*npool) * ncap);
+ if (!npool)
+ goto free;
+
+ listpoolcap = ncap;
+ listpool = npool;
}
listpool[listpoolsize++] = list;
} else {
+free:
free(list->v);
free(list);
}
}
-void lil_list_append(struct lil_list *list, struct lil_value *val)
+int lil_list_append(struct lil_list *list, void *item)
{
if (list->c == list->cap) {
size_t cap = list->cap ? (list->cap + list->cap / 2) : 32;
- struct lil_value **nv =
- realloc(list->v, sizeof(struct lil_value *) * cap);
+ struct lil_symbol **nv = realloc(list->v, sizeof(void *) * cap);
if (!nv)
- return;
+ return -ENOMEM;
list->cap = cap;
list->v = nv;
}
- list->v[list->c++] = val;
+ list->v[list->c++] = item;
+ return 0;
}
-size_t lil_list_size(struct lil_list *list)
+static struct lil_symbol *lil_list_gets(struct lil_list *list, size_t index)
{
- return list->c;
+ assert(index < list->c);
+ return list->v[index];
}
struct lil_value *lil_list_get(struct lil_list *list, size_t index)
{
- return index >= list->c ? NULL : list->v[index];
+ struct lil_symbol *sym = lil_list_gets(list, index);
+
+ assert(sym->type == LIL_SYMBOL_VALUE);
+ return &sym->value;
}
static int needs_escape(const char *str)
@@ -519,25 +580,26 @@ struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape)
size_t i, j;
for (i = 0; i < list->c; i++) {
+ struct lil_value *item = lil_list_get(list, i);
int escape =
- do_escape ? needs_escape(lil_to_string(list->v[i])) : 0;
+ do_escape ? needs_escape(lil_to_string(item)) : 0;
if (i)
lil_append_char(val, ' ');
if (escape) {
lil_append_char(val, '{');
- for (j = 0; j < list->v[i]->l; j++) {
- if (list->v[i]->d[j] == '{')
+ for (j = 0; j < item->l; j++) {
+ if (item->d[j] == '{')
lil_append_string(val, "}\"\\o\"{");
- else if (list->v[i]->d[j] == '}')
+ else if (item->d[j] == '}')
lil_append_string(val, "}\"\\c\"{");
else
- lil_append_char(val, list->v[i]->d[j]);
+ lil_append_char(val, item->d[j]);
}
lil_append_char(val, '}');
} else {
- lil_append_val(val, list->v[i]);
+ lil_append_val(val, item);
}
}
return val;
@@ -588,15 +650,23 @@ void lil_free_env(struct lil_env *env)
free(env->var);
if (envpoolsize == envpoolcap) {
+ int ncap;
+ struct lil_env **npool;
+
if (envpoolcap)
envpoolcap += envpoolcap / 2;
else
envpoolcap = 64;
- envpool = realloc(envpool,
- sizeof(*envpool) * envpoolcap);
+ npool = realloc(envpool, sizeof(*npool) * ncap);
+ if (!npool)
+ goto free;
+
+ envpoolcap = ncap;
+ envpool = npool;
}
envpool[envpoolsize++] = env;
} else {
+free:
hm_destroy(&env->varmap);
for (i = 0; i < env->vars; i++) {
free(env->var[i]->n);
@@ -660,6 +730,8 @@ static struct lil_func *add_func(struct lil *lil, const char *name)
}
cmd = calloc(1, sizeof(struct lil_func));
+ if (!cmd)
+ return NULL;
cmd->name = strdup(name);
ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
@@ -1093,15 +1165,17 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
struct lil_value *r;
if (cmd->proc) {
- lil->env->proc = words->v[0]->d;
- r = cmd->proc(lil, words->c - 1, words->v + 1);
+ lil->env->proc = lil_to_string(lil_list_get(words, 0));
+ r = cmd->proc(lil, words->c - 1,
+ (struct lil_value **)words->v + 1);
lil->env->proc = NULL;
} else {
lil_push_env(lil);
lil->env->func = cmd;
if (cmd->argnames->c == 1 &&
- !strcmp(lil_to_string(cmd->argnames->v[0]), "args")) {
+ !strcmp(lil_to_string(lil_list_get(cmd->argnames, 0)),
+ "args")) {
struct lil_value *args = lil_list_to_value(words, 1);
lil_set_var(lil, "args", args, LIL_SETVAR_LOCAL_NEW);
@@ -1111,14 +1185,15 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
for (i = 0; i < cmd->argnames->c; i++) {
struct lil_value *val;
+ struct lil_value *name =
+ lil_list_get(cmd->argnames, i);
if (i < words->c - 1)
- val = words->v[i + 1];
+ val = lil_list_get(words, i + 1);
else
val = lil->empty;
- lil_set_var(lil,
- lil_to_string(cmd->argnames->v[i]),
+ lil_set_var(lil, lil_to_string(name),
val, LIL_SETVAR_LOCAL_NEW);
}
}
@@ -1176,13 +1251,13 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
goto cleanup;
if (words->c) {
- struct lil_func *cmd =
- lil_find_cmd(lil, lil_to_string(words->v[0]));
+ const char *cmdname =
+ lil_to_string(lil_list_get(words, 0));
+ struct lil_func *cmd = lil_find_cmd(lil, cmdname);
if (!cmd) {
- if (words->v[0]->l) {
- lil_set_error_nocmd(lil,
- words->v[0]->d);
+ if (cmdname[0]) {
+ lil_set_error_nocmd(lil, cmdname);
goto cleanup;
}
} else {
@@ -2042,6 +2117,9 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
name = lil_clone_value(argv[0]);
fargs = lil_subst_to_list(lil, argv[1]);
cmd = add_func(lil, lil_to_string(argv[0]));
+ if (!cmd)
+ return NULL;
+
cmd->argnames = fargs;
cmd->code = lil_clone_value(argv[2]);
} else {
@@ -2052,11 +2130,17 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
fargs = lil_subst_to_list(lil, tmp);
lil_free_value(tmp);
cmd = add_func(lil, lil_to_string(name));
+ if (!cmd)
+ return NULL;
+
cmd->argnames = fargs;
cmd->code = lil_clone_value(argv[0]);
} else {
fargs = lil_subst_to_list(lil, argv[0]);
cmd = add_func(lil, lil_to_string(name));
+ if (!cmd)
+ return NULL;
+
cmd->argnames = fargs;
cmd->code = lil_clone_value(argv[1]);
}
@@ -2278,7 +2362,7 @@ static struct lil_value *fnc_index(struct lil *lil, size_t argc,
if (index >= list->c)
r = NULL;
else
- r = lil_clone_value(list->v[index]);
+ r = lil_clone_value(lil_list_get(list, index));
lil_free_list(list);
return r;
}
@@ -2295,7 +2379,7 @@ static struct lil_value *fnc_indexof(struct lil *lil, size_t argc,
list = lil_subst_to_list(lil, argv[0]);
for (index = 0; index < list->c; index++) {
- if (!strcmp(lil_to_string(list->v[index]),
+ if (!strcmp(lil_to_string(lil_list_get(list, index)),
lil_to_string(argv[1]))) {
r = lil_alloc_integer(index);
break;
@@ -2364,7 +2448,7 @@ static struct lil_value *fnc_slice(struct lil *lil, size_t argc,
slice = lil_alloc_list();
for (i = (size_t)from; i < (size_t)to; i++)
- lil_list_append(slice, lil_clone_value(list->v[i]));
+ lil_list_append(slice, lil_clone_value(lil_list_get(list, i)));
lil_free_list(list);
r = lil_list_to_value(slice, 1);
@@ -2395,10 +2479,12 @@ static struct lil_value *fnc_filter(struct lil *lil, size_t argc,
list = lil_subst_to_list(lil, argv[base]);
filtered = lil_alloc_list();
for (i = 0; i < list->c && !lil->env->breakrun; i++) {
- lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+ lil_set_var(lil, varname, lil_list_get(list, i),
+ LIL_SETVAR_LOCAL_ONLY);
r = lil_eval_expr(lil, argv[base + 1]);
if (lil_to_boolean(r))
- lil_list_append(filtered, lil_clone_value(list->v[i]));
+ lil_list_append(filtered,
+ lil_clone_value(lil_list_get(list, i)));
lil_free_value(r);
}
lil_free_list(list);
@@ -2475,7 +2561,8 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
for (i = 0; i < list->c; i++) {
struct lil_value *rv;
- lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+ lil_set_var(lil, varname, lil_list_get(list, i),
+ LIL_SETVAR_LOCAL_ONLY);
rv = lil_parse_value(lil, argv[codeidx], 0);
if (rv->l)
lil_list_append(rlist, rv);
diff --git a/include/cli_lil.h b/include/cli_lil.h
index 91e79c12f4..40c822401e 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -149,13 +149,13 @@ struct lil_value *lil_alloc_integer(ssize_t num);
void lil_free_value(struct lil_value *val);
struct lil_value *lil_clone_value(struct lil_value *src);
-int lil_append_char(struct lil_value *val, char ch);
-int lil_append_string(struct lil_value *val, const char *s);
-int lil_append_val(struct lil_value *val, struct lil_value *v);
+enum lil_error lil_append_char(struct lil_value *val, char ch);
+enum lil_error lil_append_string(struct lil_value *val, const char *s);
+enum lil_error lil_append_val(struct lil_value *val, struct lil_value *v);
struct lil_list *lil_alloc_list(void);
void lil_free_list(struct lil_list *list);
-void lil_list_append(struct lil_list *list, struct lil_value *val);
+int lil_list_append(struct lil_list *list, void *item);
size_t lil_list_size(struct lil_list *list);
struct lil_value *lil_list_get(struct lil_list *list, size_t index);
struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 20/28] cli: lil: Add config to enable debug output
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (18 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 19/28] cli: lil: Add "symbol" structure Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 21/28] cli: lil: Add a distinct parsing step Sean Anderson
` (9 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This provides an easy way to enable assertions and debug messages. It will
also be used to enable tracing features in future patches.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
cmd/Kconfig | 6 ++++++
common/Makefile | 3 +++
2 files changed, 9 insertions(+)
diff --git a/cmd/Kconfig b/cmd/Kconfig
index 7c8962cfc2..bba72bbdc2 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -61,6 +61,12 @@ config LIL_RECLIMIT
avoid call stack overflows and is also useful when running through an
automated fuzzer like AFL. Set to 0 to disable the recursion limit.
+config LIL_DEBUG
+ bool "Enable LIL debugging"
+ help
+ This enables debug prints, assertions, and other tracing features in
+ LIL. If you are not working on LIL, say 'n' here.
+
endif
endif
diff --git a/common/Makefile b/common/Makefile
index dce04b305e..558e1932fe 100644
--- a/common/Makefile
+++ b/common/Makefile
@@ -11,6 +11,9 @@ obj-y += exports.o
obj-$(CONFIG_HASH) += hash.o
obj-$(CONFIG_HUSH_PARSER) += cli_hush.o
obj-$(CONFIG_LIL) += cli_lil.o
+ifneq ($(CONFIG_LIL_DEBUG),)
+CFLAGS_cli_lil.o += -DDEBUG
+endif
obj-$(CONFIG_AUTOBOOT) += autoboot.o
# This option is not just y/n - it can have a numeric value
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 21/28] cli: lil: Add a distinct parsing step
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (19 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 20/28] cli: lil: Add config to enable debug output Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 22/28] env: Add a priv pointer to hwalk_r Sean Anderson
` (8 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This adds a parser to LIL (as separate from the interpreter). This is
necessary to detect syntax errors before evaluating anything. Before this,
running a script like
echo some message; echo syntax error}
would result in "some message" being printed before the error was
discovered. This is not only rather surprising, but also makes things like
Hush's secondary prompt impossible to implement. In addition, the original
parser would accept almost any input, and silently return NULL if it
encountered problems. This made it difficult to determine if a command had
been mis-parsed, since an empty command would just evaluate to "".
The grammar is not the same as LIL originally. Several ideas have been
taken from TCL proper as well. In order to simplify the parser, it has been
rewritten to be LL(1), except for line continuations which are LL(2). In
particular, multi-line comments and command/variable subtitutions partially
through unquoted words (e.g. a$b) have been removed. Some other characters
such as unescaped, unmatched }s are now syntax errors. On the other hand,
some things such as escaped characters in unquoted words have been added
back (as seen in TCL). Unlike TCL, comments may be placed almost anywhere.
The exact grammar is subject to change, but I have tried to make it as sane
as I can get it.
The grammar has been documented in (extended) EBNF. The names of the
nonterminals are the same as are used in the dodekalogue [1]. Each
nonterminal foo has a function parse_foo() which recognizes it.
[1] https://www.tcl.tk/man/tcl8.6/TclCmd/Tcl.htm
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
cmd/Kconfig | 4 +-
common/cli.c | 2 +-
common/cli_lil.c | 1880 ++++++++++++++++++++++++++++++++++++---------
include/cli_lil.h | 11 +-
test/cmd/lil.c | 73 +-
5 files changed, 1527 insertions(+), 443 deletions(-)
diff --git a/cmd/Kconfig b/cmd/Kconfig
index bba72bbdc2..7ff8e4a7e5 100644
--- a/cmd/Kconfig
+++ b/cmd/Kconfig
@@ -43,8 +43,8 @@ if LIL
config LIL_FULL
bool "Enable all LIL features"
help
- This enables all LIL builtin functions, as well as expression support
- for arithmetic and bitwise operations.
+ This enables all LIL builtin functions, expression support for
+ arithmetic and bitwise operations, and expanded error messages.
config LIL_POOLS
bool "Use memory pools for LIL structures"
diff --git a/common/cli.c b/common/cli.c
index ad5d76d563..391fee0ec7 100644
--- a/common/cli.c
+++ b/common/cli.c
@@ -49,7 +49,7 @@ static const struct lil_callbacks env_callbacks = {
static int lil_run(const char *cmd)
{
int err;
- struct lil_value *result = lil_parse(lil, cmd, 0, 0);
+ struct lil_value *result = lil_parse_eval(lil, cmd, 0, true);
const char *err_msg, *strres = lil_to_string(result);
/* The result may be very big, so use puts */
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 06fd37c383..2ed96ebc2d 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -16,6 +16,7 @@
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
+#include <linux/err.h>
#define HASHMAP_CELLS 256
#define HASHMAP_CELLMASK 0xFF
@@ -121,14 +122,27 @@ struct lil_list {
};
/**
- * struct lil_symbol - A symbol parsed by the parser
+ * enum lil_symbol_type - The type of data in a symbol
* @LIL_SYMBOL_VALUE: A plain old string and length
* @LIL_SYMBOL_VARIABLE: A name of a variable to be substituted
* @LIL_SYMBOL_LIST: A list of symbols
- * @value: A literal value or name of variable
- * @list: A list of commands in the script
- * @word: Another word to be evaluated
- * @type: The type of word
+ * @LIL_SYMBOL_COMMAND: A command to be ran
+ * @LIL_SYMBOL_SCRIPT: A script to be run
+ */
+enum lil_symbol_type {
+ LIL_SYMBOL_VALUE = 0,
+ LIL_SYMBOL_LIST,
+ LIL_SYMBOL_VARIABLE,
+ LIL_SYMBOL_COMMAND,
+ LIL_SYMBOL_SCRIPT,
+};
+
+/**
+ * struct lil_symbol - A symbol parsed by the parser
+ * @value: A literal value
+ * @list: A list of commands, words, or symbols
+ * @symbol: Another symbol to be evaluated
+ * @type: The type of symbol
*/
struct lil_symbol {
union {
@@ -136,13 +150,7 @@ struct lil_symbol {
struct lil_symbol *symbol;
struct lil_list list;
};
- enum {
- LIL_SYMBOL_VALUE = 0,
- LIL_SYMBOL_LIST,
- LIL_SYMBOL_VARIABLE,
- LIL_SYMBOL_COMMAND,
- LIL_SYMBOL_SCRIPT,
- } type;
+ enum lil_symbol_type type;
};
/**
@@ -162,14 +170,64 @@ struct lil_func {
lil_func_proc_t proc;
};
+/**
+ * struct lil_position - A position within a script
+ * @head: The absolute offset
+ * @line: The current line (as delineated by newlines)
+ * @column: The column within the current line
+ */
+struct lil_position {
+ size_t head;
+ size_t line;
+ size_t column;
+};
+
+/**
+ * struct lil_parser_error - Errors encountered while parsing a script
+ * @PERR_NONE: There is no error.
+ * @PERR_OOM: We ran out of memory.
+ * @PERR_EXPECTED: The parser was expecting a specific character but got
+ * something else or ran out of input.
+ * @PERR_UNEXPECTED: The parser encountered an unexpected character or ran out
+ * of input.
+ * @type: The type of error
+ * @func: The name of the function which caused the error
+ * @expected: The character we expected to find
+ * @matching: The position of the character which made us expect @expected
+ *
+ * @expected is only valid when @type is %PERR_EXPECTED. @matching is valid when
+ * @type is %PERR_EXPECTED or when @type is %PERR_UNEXPECTED and the parser is
+ * at the end of the file.
+ */
+struct lil_parser_error {
+ struct lil_position matching;
+ const char *func;
+ enum {
+ PERR_NONE = 0,
+ PERR_OOM,
+ PERR_EXPECTED,
+ PERR_UNEXPECTED,
+ } type;
+ char expected;
+};
+
+/**
+ * struct lil_parser - State used when parsing a script
+ * @code: The script which is being parsed
+ * @len: The length of the script
+ * @pos: Our current position within @code
+ * @err: The current error (if any)
+ */
+struct lil_parser {
+ const char *code;
+ size_t len;
+ size_t depth;
+ struct lil_position pos;
+ struct lil_parser_error err;
+};
+
/**
* struct lil - The current state of the interpreter
- * @code: The code which is being interpreted
- * @rootcode: The top-level code (e.g. the code for the initial call to
- * lil_parse())
- * @clen: The length of @code
- * @head: The first uninterpreted part of this code, as an index of @code
- * @ignoreeol: Whether to treat newlines as whitespace or command terminators
* @cmd: A list of the current commands
* @cmds: The number of commands in @cmd
* @cmdmap: A hash map mapping command names to pointers to commands in @cmd
@@ -185,14 +243,9 @@ struct lil_func {
* @error: The current error status
* @err_head: The offset in @code which caused the @error
* @err_msg: An optional string describing the current @error
- * @parse_depth: The depth of recursive function calls
+ * @depth: The depth of recursive function calls
*/
struct lil {
- const char *code; /* need save on parse */
- const char *rootcode;
- size_t clen; /* need save on parse */
- size_t head; /* need save on parse */
- int ignoreeol;
struct lil_func **cmd;
size_t cmds;
struct hashmap cmdmap;
@@ -203,7 +256,7 @@ struct lil {
enum lil_error err;
char *err_msg;
struct lil_callbacks callbacks;
- size_t parse_depth;
+ size_t depth;
};
/**
@@ -231,7 +284,6 @@ struct expreval {
} error;
};
-static struct lil_value *next_word(struct lil *lil);
static void register_stdcmds(struct lil *lil);
static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
static void lil_set_errorf(struct lil *lil, enum lil_error err,
@@ -560,51 +612,6 @@ struct lil_value *lil_list_get(struct lil_list *list, size_t index)
return &sym->value;
}
-static int needs_escape(const char *str)
-{
- size_t i;
-
- if (!str || !str[0])
- return 1;
-
- for (i = 0; str[i]; i++)
- if (ispunct(str[i]) || isspace(str[i]))
- return 1;
-
- return 0;
-}
-
-struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape)
-{
- struct lil_value *val = alloc_value(NULL);
- size_t i, j;
-
- for (i = 0; i < list->c; i++) {
- struct lil_value *item = lil_list_get(list, i);
- int escape =
- do_escape ? needs_escape(lil_to_string(item)) : 0;
-
- if (i)
- lil_append_char(val, ' ');
-
- if (escape) {
- lil_append_char(val, '{');
- for (j = 0; j < item->l; j++) {
- if (item->d[j] == '{')
- lil_append_string(val, "}\"\\o\"{");
- else if (item->d[j] == '}')
- lil_append_string(val, "}\"\\c\"{");
- else
- lil_append_char(val, item->d[j]);
- }
- lil_append_char(val, '}');
- } else {
- lil_append_val(val, item);
- }
- }
- return val;
-}
-
struct lil_env *lil_alloc_env(struct lil_env *parent)
{
struct lil_env *env;
@@ -888,275 +895,1167 @@ struct lil *lil_new(const struct lil_callbacks *callbacks)
return lil;
}
-static int islilspecial(char ch)
+/**
+ * DOC: Syntax
+ *
+ * Syntax is EBNF, except that [], {}, ?, *, and + have been borrowed from
+ * regular expressions and , is optional. In addition, a - b matches strings
+ * which matches a but which do not match b (set difference).
+ *
+ * ::
+ *
+ * script = whitespace* command? ( terminator whitespace* command? )* ;
+ *
+ * command = word ( ( whitespace - continuation ) whitespace* word )*
+ * whitespace* ;
+ *
+ * word = single-quote | double-quote | brace | bracket | dollar
+ * | ( escape-sequence | continuation |
+ * ( character - word-special - space ) )+ ;
+ *
+ * single-quote = "'" ( subcommand | dollar | escape-sequence | continuation
+ * | ( character - "'" ) )* "'" ;
+ * double-quote = '"' ( subcommand | dollar | escape-sequence | continuation
+ * | ( character - '"' ) )* '"' ;
+ * escape-sequence = '\\' ( character - '\n' ) ;
+ * brace = '{' ( brace | '\\' character | comment
+ * | ( character - '}' - '#' ) )* '}' ;
+ * bracket = '[' script ']' ;
+ * dollar = '$' word ;
+ *
+ * whitespace = space | continuation ;
+ * continuation = '\\\n' ;
+ * terminator = '\n' | ';' | comment ;
+ * comment = '#' ( character - '\n' )* '\n' ;
+ *
+ * space = ' ' | '\f' | '\r' | '\t' | '\v' ;
+ * word-special = '$' | '{' | '}' | '[' | ']' | '"' | "'" | '\\' |
+ * | terminator ;
+ * character = [\x00-\xff] ;
+ *
+ * In addition to the full syntax above, many commands expect arguments
+ * formatted as lists. This syntax is similar to the above, with the following
+ * exceptions:
+ *
+ * - Lists are words separated by whitespace.
+ * - Neither ``$`` nor ``[`` substitutions are performed.
+ * - ``#`` and ``;`` have no special meaning.
+ * - ``\n`` is considered whitespace.
+ *
+ * ::
+ *
+ * list = space* word? ( space+ word )* space* ;
+ *
+ * word = single-quote | double-quote | brace
+ * | ( escape-sequence | continuation | ( character - space ) )+ ;
+ *
+ * single-quote = "'" ( escape-sequence | continuation
+ * | ( character - "'" ) )* "'" ;
+ * double-quote = '"' ( escape-sequence | continuation
+ * | ( character - '"' ) )* '"' ;
+ * escape-sequence = '\\' ( character - '\n' ) ;
+ * brace = '{' ( brace | '\\' character | ( character - '}' ) )* '}' ;
+ *
+ * continuation = '\\\n' ;
+ * space = ' ' | '\f' | '\n' | '\r' | '\t' | '\v' ;
+ * character = [\x00-\xff] ;
+ *
+ * Because of the similarity of these grammars, they may be parsed using the
+ * same functions. Where differences occur, they are selected by a boolean
+ * parameter.
+ *
+ * In general, each parse function must determine two things: what symbol to try
+ * and parse when there are multiple possible choices, and where the end of the
+ * symbol is. To choose from symbols, we consider the FIRST set of the possible
+ * symbols. The FIRST set is the set of terminals which may begin a symbol. For
+ * example, in the grammar
+ *
+ * ::
+ *
+ * number = sign? integer fraction? ;
+ * integer = digit+ ;
+ * fraction = '.' digit+ ;
+ * sign = '+' | '-' ;
+ * digit = [0-9] ;
+ *
+ * the FIRST sets are
+ *
+ * ::
+ *
+ * FIRST(sign) = '+' | '-' ;
+ * FIRST(digit) = [0-9] ;
+ * FIRST(fraction) = '.' ;
+ * FIRST(integer) = FIRST(digit) ;
+ * = [0-9] ;
+ * FIRST(number) = FIRST(sign) | FIRST(integer) ;
+ * = '+' | '-' | [0-9] ;
+ *
+ * A parser, when deciding whether to parse a sign or an integer, may observe
+ * whether the number begins with FIRST(sign) or FIRST(integer). To prevent
+ * backtracking, the FIRST sets of all symbols which must be picked from must
+ * be disjoint. When this is not the case (like for escape-sequence and
+ * continuation), the analogous SECOND set may be used.
+ *
+ * The FOLLOW set of a symbol is the union of all FIRST sets which may come
+ * after it. For the above grammar, the FOLLOW sets are
+ *
+ * ::
+ *
+ * FOLLOW(number) = ;
+ * FOLLOW(sign) = FIRST(integer) ;
+ * = [0-9] ;
+ * FOLLOW(integer) = FIRST(fraction) | ;
+ * = '.' | ;
+ * FOLLOW(fraction) = ;
+ * FOLLOW(digit) = FOLLOW(integer) | FOLLOW(fraction) ;
+ * = '.' | ;
+ *
+ * The parser, when deciding whether it is done parsing an integer, may consider
+ * whether the current character matches FOLLOW(integer). To prevent
+ * backtracking, the FOLLOW sets of each symbol must not contain characters
+ * which may be present at the end of the symbol. In general, the FOLLOW set of
+ * a symbol is interesting if the symbol has a trailing repeating portion which
+ * is ended only by the next symbol.
+ */
+
+static struct lil_symbol *parse_word(struct lil_parser *p, bool islist);
+static struct lil_list *parse_script(struct lil_parser *p);
+
+/**
+ * eof() - Whether we have reached the end of input
+ * @p: The parser
+ *
+ * Return: %true if there are no more characters left
+ */
+static bool eof(struct lil_parser *p)
{
- return ch == '$' || ch == '{' || ch == '}' || ch == '[' || ch == ']' ||
- ch == '"' || ch == '\'' || ch == ';';
+ return p->pos.head >= p->len;
}
-static int eolchar(char ch)
+/**
+ * peek() - Peek at the next character to parse
+ * @p: The parser
+ *
+ * eof() for @p must be %false.
+ *
+ * Return: The character which would be returned by pop().
+ */
+static char peek(struct lil_parser *p)
{
- return ch == '\n' || ch == '\r' || ch == ';';
+ return p->code[p->pos.head];
}
-static int ateol(struct lil *lil)
+/**
+ * peek2() - Peek two characters ahead
+ * @p: The parser
+ *
+ * NB: Unlike peek(), peek2() checks for eof().
+ *
+ * Return: The character which would be returned by pop()ing twice, or %-1 if no
+ * such character is present (due to the end of input).
+ */
+static char peek2(struct lil_parser *p)
{
- return !(lil->ignoreeol) && eolchar(lil->code[lil->head]);
+ return p->pos.head + 1 < p->len ? p->code[p->pos.head + 1] : -1;
}
-static void lil_skip_spaces(struct lil *lil)
+/**
+ * pop() - Advance the parser by one character
+ * @p: The parser
+ *
+ * Return: The character which is next in the input
+ */
+static char pop(struct lil_parser *p)
{
- while (lil->head < lil->clen) {
- if (lil->code[lil->head] == '#') {
- if (lil->code[lil->head + 1] == '#' &&
- lil->code[lil->head + 2] != '#') {
- lil->head += 2;
- while (lil->head < lil->clen) {
- if ((lil->code[lil->head] == '#') &&
- (lil->code[lil->head + 1] == '#') &&
- (lil->code[lil->head + 2] != '#')) {
- lil->head += 2;
- break;
- }
- lil->head++;
- }
- } else {
- while (lil->head < lil->clen &&
- !eolchar(lil->code[lil->head]))
- lil->head++;
+ char ret = p->code[p->pos.head++];
+
+#if IS_ENABLED(CONFIG_LIL_FULL)
+ p->pos.column++;
+ if (ret == '\n') {
+ p->pos.line++;
+ p->pos.column = 1;
+ }
+#endif
+ return ret;
+}
+
+#if IS_ENABLED(CONFIG_LIL_DEBUG)
+#define set_err_func(p, _func) (p)->err.func = _func
+#else
+#define set_err_func(p, _func)
+#endif
+
+#define err_oom(p) do { \
+ (p)->err.type = PERR_OOM; \
+ set_err_func(p, __func__); \
+} while (0)
+
+#define err_unexpected_1(p) do { \
+ assert(!eof(p)); \
+ (p)->err.type = PERR_UNEXPECTED; \
+ set_err_func(p, __func__); \
+} while (0)
+
+#if IS_ENABLED(CONFIG_LIL_FULL)
+#define err_expected(p, c, _matching) do { \
+ (p)->err.type = PERR_EXPECTED; \
+ set_err_func(p, __func__); \
+ (p)->err.expected = (c); \
+ (p)->err.matching = (_matching); \
+} while (0)
+
+#define err_unexpected_2(p, _matching) do { \
+ assert(eof(p)); \
+ (p)->err.type = PERR_UNEXPECTED; \
+ set_err_func(p, __func__); \
+ (p)->err.matching = (_matching); \
+} while (0)
+#else /* CONFIG_LIL_FULL */
+#define UNUSED(x) (void)(x)
+
+#define err_expected(p, c, _matching) do { \
+ (p)->err.type = PERR_EXPECTED; \
+ set_err_func(p, __func__); \
+ UNUSED(_matching); \
+} while (0)
+
+#define err_unexpected_2(p, _matching) do { \
+ assert(eof(p)); \
+ (p)->err.type = PERR_UNEXPECTED; \
+ set_err_func(p, __func__); \
+ UNUSED(_matching); \
+} while (0)
+#endif
+
+#define err_unexpected(p, ...) \
+ __concat(err_unexpected_, __count_args(p, ##__VA_ARGS__)) (p, ##__VA_ARGS__)
+
+/**
+ * expect() - Expect a specific character next
+ * @p: The parser
+ * @sym: The symbol to return on success; free()'d on error
+ * @expected: The character to expect
+ * @matching: The position of the thing we expected to match
+ *
+ * Return: @w, or %NULL on error
+ */
+static struct lil_symbol *_expect(struct lil_parser *p, struct lil_symbol *sym,
+ char expected, struct lil_position *matching,
+ const char *func)
+{
+ char got;
+
+ if (!sym)
+ return sym;
+
+ if (eof(p))
+ goto err;
+
+ got = pop(p);
+ if (got != expected)
+ goto err;
+ return sym;
+
+err:
+ lil_free_symbol(sym);
+ err_expected(p, expected, *matching);
+ set_err_func(p, func);
+ return NULL;
+}
+
+#define expect(p, sym, expected, matching) \
+ _expect(p, sym, expected, matching, __func__)
+
+#define CASE_SPACE \
+ case '\f': \
+ case '\r': \
+ case '\t': \
+ case '\v': \
+ case ' ' \
+
+#define CASE_WHITESPACE \
+ CASE_SPACE: \
+ case '\\'
+
+#define CASE_TERMINATOR \
+ case '\n': \
+ case ';': \
+ case '#'
+
+/**
+ * parse_continuation() - Parse a line continuation
+ * @p: The parser
+ *
+ * FIRST(continuation) = '\\' ;
+ * SECOND(continuation) = '\n' ;
+ */
+static void parse_continuation(struct lil_parser *p)
+{
+ assert(pop(p) == '\\');
+ assert(pop(p) == '\n');
+}
+
+/**
+ * parse_whitespace() - Parse a single unit of whitespace, if present
+ * @p: The parser
+ *
+ * FIRST(whitespace) = space | '\\' ;
+ */
+static void parse_whitespace(struct lil_parser *p)
+{
+ switch (peek(p)) {
+ CASE_SPACE:
+ pop(p);
+ return;
+ case '\\':
+ parse_continuation(p);
+ return;
+ default:
+ return;
+ }
+}
+
+/**
+ * parse_dollar() - Parse a variable reference
+ * @p: The parser
+ *
+ * FIRST(variable) = '$' ;
+ * FOLLOW(variable) = FOLLOW(word) ;
+ *
+ * Return: A symbol containing the name of the variable, or %NULL on error
+ */
+static struct lil_symbol *parse_dollar(struct lil_parser *p)
+{
+ struct lil_symbol *sym = calloc(1, sizeof(struct lil_symbol));
+
+ if (!sym) {
+ err_oom(p);
+ return NULL;
+ }
+
+ assert(pop(p) == '$');
+ sym->type = LIL_SYMBOL_VARIABLE;
+ sym->symbol = parse_word(p, false);
+ if (!sym->symbol) {
+ lil_free_symbol(sym);
+ return NULL;
+ }
+ return sym;
+}
+
+/**
+ * parse_bracket() - Parse a subscript enclosed in brackets
+ * @p: The parser
+ *
+ * FIRST(bracket) = '[' ;
+ * FOLLOW(bracket) = FOLLOW(word) ;
+ *
+ * Return: A symbol containing the script, or %NULL on error
+ */
+static struct lil_symbol *parse_bracket(struct lil_parser *p)
+{
+ struct lil_position savepos = p->pos;
+
+ assert(pop(p) == '[');
+ return expect(p, list_to_symbol(parse_script(p)), ']', &savepos);
+}
+
+/**
+ * parse_comment() - Parse a comment
+ * @p: The parser
+ *
+ * FIRST(comment) = '#' ;
+ */
+static void parse_comment(struct lil_parser *p)
+{
+ assert(pop(p) == '#');
+ while (!eof(p)) {
+ switch (pop(p)) {
+ case '\n':
+ return;
+ }
+ }
+}
+
+/**
+ * parse_brace() - Parse a value enclosed in braces
+ * @p: The parser
+ * @islist: If we are parsing a list
+ *
+ * This function is used to parse braces in scripts and lists. If we are parsing
+ * a list, then comments are not parsed.
+ *
+ * FIRST(brace) = '{' ;
+ * FOLLOW(brace) = FOLLOW(word) ;
+ *
+ * Return: A symbol containing a value, or %NULL on error
+ */
+static struct lil_symbol *parse_brace(struct lil_parser *p, bool islist)
+{
+ struct lil_value *val = alloc_value(NULL);
+ struct lil_position savepos = p->pos;
+
+ if (!val)
+ goto oom;
+
+ assert(pop(p) == '{');
+ while (!eof(p)) {
+ switch (peek(p)) {
+ case '{': {
+ bool fail;
+ struct lil_symbol *brace = parse_brace(p, islist);
+
+ if (!brace)
+ goto err;
+
+ fail = lil_append_char(val, '{') ||
+ lil_append_val(val, &brace->value) ||
+ lil_append_char(val, '}');
+
+ lil_free_symbol(brace);
+ if (fail)
+ goto oom;
+ break;
+ }
+ case '#':
+ if (islist)
+ goto character;
+ parse_comment(p);
+ break;
+ case '\\': {
+ struct lil_position escapepos = p->pos;
+
+ if (lil_append_char(val, pop(p)))
+ goto oom;
+
+ if (eof(p)) {
+ err_unexpected(p, escapepos);
+ goto err;
}
- } else if (lil->code[lil->head] == '\\' &&
- eolchar(lil->code[lil->head + 1])) {
- lil->head++;
- while (lil->head < lil->clen &&
- eolchar(lil->code[lil->head]))
- lil->head++;
- } else if (eolchar(lil->code[lil->head])) {
- if (lil->ignoreeol)
- lil->head++;
+ }
+ fallthrough;
+ default:
+character:
+ if (lil_append_char(val, pop(p)))
+ goto oom;
+ break;
+ case '}':
+ pop(p);
+ return value_to_symbol(val);
+ }
+ }
+ err_expected(p, '}', savepos);
+ goto err;
+
+oom:
+ err_oom(p);
+err:
+ lil_free_value(val);
+ return NULL;
+}
+
+/**
+ * parse_escape() - Parse an escape sequence
+ * @p: The parser
+ *
+ * FIRST(escape) = '\\' ;
+ * SECOND(escape) = character - '\n' ;
+ *
+ * Return: The character parsed. If there was an error, then @p->err.type will
+ * be set.
+ */
+static char parse_escape(struct lil_parser *p)
+{
+ char c;
+ struct lil_position savepos = p->pos;
+
+ assert(pop(p) == '\\');
+ if (eof(p)) {
+ err_unexpected(p, savepos);
+ return -1;
+ }
+
+ c = pop(p);
+ switch (c) {
+ case 'a':
+ return '\a';
+ case 'b':
+ return '\b';
+ case 'f':
+ return '\f';
+ case 'n':
+ return '\n';
+ case 't':
+ return '\t';
+ case 'r':
+ return '\r';
+ case 'v':
+ return '\v';
+ case '\n':
+ assert(0);
+ fallthrough;
+ default:
+ return c;
+ }
+}
+
+/**
+ * parse_quote() - Parse a value in quotes
+ * @p: The parser
+ * @q: The quote character, either ``'`` or ``"``. The special value %-1 may
+ * also be used to specify that no enclosing quotes are expected.
+ * @islist: If we are parsing a list
+ *
+ * This function is used both for parsing scripts and lists. When used in
+ * scripts, we parse a list of symbols which must be later evaluated and
+ * concatenated. When parsing a list, we just parse a value because lists do not
+ * contain $ or [ substitutions.
+ *
+ * When @q is %-1, then no enclosing quotes are parsed. This may be used to
+ * parse a string into a form which may have substitutions performed on it.
+ *
+ * FIRST(quote) = "'" | '"' ;
+ * FOLLOW(quote) = FOLLOW(word) ;
+ *
+ * Return: A list of symbols, a value (if @islist), or %NULL on error.
+ */
+static struct lil_symbol *parse_quote(struct lil_parser *p, char q, bool islist)
+{
+ struct lil_position savepos = p->pos;
+ struct lil_list *list = islist ? NULL : lil_alloc_list();
+ struct lil_value *val = alloc_value(NULL);
+
+ if ((!islist && !list) || !val)
+ goto oom;
+
+ if (q != -1) {
+ assert(q == '\'' || q == '"');
+ assert(pop(p) == q);
+ }
+
+ while (!eof(p)) {
+ char c = peek(p);
+
+ switch (c) {
+ case '$':
+ case '[': {
+ struct lil_symbol *sym;
+
+ if (islist)
+ goto character;
+
+ if (val->l) {
+ if (lil_list_append(list, val))
+ goto oom;
+ val = alloc_value(NULL);
+ if (!val)
+ goto oom;
+ }
+
+ if (c == '[')
+ sym = parse_bracket(p);
else
- break;
- } else if (isspace(lil->code[lil->head])) {
- lil->head++;
- } else {
+ sym = parse_dollar(p);
+
+ if (!sym)
+ goto err;
+
+ if (lil_list_append(list, sym))
+ goto oom;
+
+ break;
+ }
+ case '\'':
+ case '"':
+ if (c == q) {
+ pop(p);
+ goto out;
+ }
+ goto character;
+ case '\\':
+ if (peek2(p) == '\n') {
+ parse_continuation(p);
+ continue;
+ }
+
+ c = parse_escape(p);
+ if (p->err.type)
+ goto err;
+ goto character_post_pop;
+ default:
+character:
+ c = pop(p);
+character_post_pop:
+ if (lil_append_char(val, c))
+ goto oom;
break;
}
}
+
+ if (q == -1)
+ goto out;
+ err_expected(p, q, savepos);
+ goto err;
+
+out:
+ if (islist)
+ return value_to_symbol(val);
+ else if (lil_list_append(list, val))
+ goto oom;
+ return list_to_symbol(list);
+
+oom:
+ err_oom(p);
+err:
+ lil_free_list(list);
+ lil_free_value(val);
+ return NULL;
}
-static struct lil_value *get_bracketpart(struct lil *lil)
+/**
+ * parse_word() - Parse a word
+ * @p: The parser
+ *
+ * This function used to parse words for both scripts and lists. For lists, $
+ * and [ subtitution is not performed. In addition, there are less illegal
+ * characters (since we no longer need to worry about some cases of nesting).
+ * Because of this, the FIRST and FOLLOW sets for parsing scripts are:
+ *
+ * FIRST(word) = character - ']' - '\\' - '}' - terminator - FIRST(whitespace) ;
+ * FOLLOW(word) = ''' | '"' | ']' | FIRST(terminator) | space | ;
+ *
+ * and the sets when parsing lists are:
+ *
+ * FIRST(word) = character - space ;
+ * FOLLOW(word) = space | ;
+ *
+ * Return: A symbol for one word, or %NULL on error. If we are parsing a list,
+ * this symbol will always have type %LIL_SYMBOL_VALUE.
+ */
+static struct lil_symbol *parse_word(struct lil_parser *p, bool islist)
{
- size_t cnt = 1;
- struct lil_value *val, *cmd = alloc_value(NULL);
- int save_eol = lil->ignoreeol;
+ struct lil_value *word;
- lil->ignoreeol = 0;
- lil->head++;
- while (lil->head < lil->clen) {
- if (lil->code[lil->head] == '[') {
- lil->head++;
- cnt++;
- lil_append_char(cmd, '[');
- } else if (lil->code[lil->head] == ']') {
- lil->head++;
- if (--cnt == 0)
- break;
- else
- lil_append_char(cmd, ']');
- } else {
- lil_append_char(cmd, lil->code[lil->head++]);
- }
+ switch (peek(p)) {
+ case '\'':
+ case '"':
+ return parse_quote(p, peek(p), islist);
+ case '{':
+ return parse_brace(p, islist);
+ case '[':
+ if (islist)
+ break;
+ return parse_bracket(p);
+ case '$':
+ if (islist)
+ break;
+ return parse_dollar(p);
+ case '\\':
+ if (peek2(p) == '\n')
+ goto terminator;
+ break;
+ case ']':
+ case '}':
+ case ';':
+ case '#':
+ if (islist)
+ break;
+ fallthrough;
+ case '\n':
+ CASE_SPACE:
+terminator:
+ err_unexpected(p);
+ return NULL;
}
- val = lil_parse_value(lil, cmd, 0);
- lil_free_value(cmd);
- lil->ignoreeol = save_eol;
- return val;
-}
+ word = alloc_value(NULL);
+ if (!word)
+ goto oom;
-static struct lil_value *get_dollarpart(struct lil *lil)
-{
- struct lil_value *val, *name, *tmp;
+ do {
+ char c = peek(p);
- lil->head++;
- name = next_word(lil);
- tmp = alloc_value("set ");
- lil_append_val(tmp, name);
- lil_free_value(name);
-
- val = lil_parse_value(lil, tmp, 0);
- lil_free_value(tmp);
- return val;
-}
-
-static struct lil_value *next_word(struct lil *lil)
-{
- struct lil_value *val;
- size_t start;
-
- lil_skip_spaces(lil);
- if (lil->code[lil->head] == '$') {
- val = get_dollarpart(lil);
- } else if (lil->code[lil->head] == '{') {
- size_t cnt = 1;
-
- lil->head++;
- val = alloc_value(NULL);
- while (lil->head < lil->clen) {
- if (lil->code[lil->head] == '{') {
- lil->head++;
- cnt++;
- lil_append_char(val, '{');
- } else if (lil->code[lil->head] == '}') {
- lil->head++;
- if (--cnt == 0)
- break;
- else
- lil_append_char(val, '}');
- } else {
- lil_append_char(val, lil->code[lil->head++]);
+ switch (c) {
+ case ']':
+ case '\'':
+ case '"':
+ case ';':
+ case '#':
+ if (islist)
+ goto character;
+ fallthrough;
+ case '\n':
+ CASE_SPACE:
+ return value_to_symbol(word);
+ case '{':
+ case '}':
+ case '[':
+ case '$':
+ if (islist)
+ goto character;
+ err_unexpected(p);
+ return NULL;
+ case '\\':
+ if (peek2(p) == '\n') {
+ parse_continuation(p);
+ continue;
}
+
+ c = parse_escape(p);
+ if (p->err.type)
+ goto err;
+ goto character_post_pop;
+ default:
+character:
+ c = pop(p);
+character_post_pop:
+ if (lil_append_char(word, c))
+ goto oom;
}
- } else if (lil->code[lil->head] == '[') {
- val = get_bracketpart(lil);
- } else if (lil->code[lil->head] == '"' ||
- lil->code[lil->head] == '\'') {
- char sc = lil->code[lil->head++];
+ } while (!eof(p));
- val = alloc_value(NULL);
- while (lil->head < lil->clen) {
- if (lil->code[lil->head] == '[' ||
- lil->code[lil->head] == '$') {
- struct lil_value *tmp =
- lil->code[lil->head] == '$' ?
- get_dollarpart(lil) :
- get_bracketpart(lil);
+ return value_to_symbol(word);
- lil_append_val(val, tmp);
- lil_free_value(tmp);
- lil->head--; /* avoid skipping the char below */
- } else if (lil->code[lil->head] == '\\') {
- lil->head++;
- switch (lil->code[lil->head]) {
- case 'b':
- lil_append_char(val, '\b');
- break;
- case 't':
- lil_append_char(val, '\t');
- break;
- case 'n':
- lil_append_char(val, '\n');
- break;
- case 'v':
- lil_append_char(val, '\v');
- break;
- case 'f':
- lil_append_char(val, '\f');
- break;
- case 'r':
- lil_append_char(val, '\r');
- break;
- case '0':
- lil_append_char(val, 0);
- break;
- case 'a':
- lil_append_char(val, '\a');
- break;
- case 'c':
- lil_append_char(val, '}');
- break;
- case 'o':
- lil_append_char(val, '{');
- break;
- default:
- lil_append_char(val,
- lil->code[lil->head]);
- break;
+oom:
+ err_oom(p);
+err:
+ lil_free_value(word);
+ return NULL;
+}
+
+/**
+ * parse_command() - Parse a command
+ * @p: The parser
+ *
+ * FIRST(command) = FIRST(word) ;
+ * FOLLOW(command) = ']' | FIRST(terminator) | ;
+ *
+ * Return: A list of words which compose the command, or %NULL on error
+ */
+static struct lil_list *parse_command(struct lil_parser *p)
+{
+ struct lil_symbol *word;
+ struct lil_list *command = lil_alloc_list();
+
+ if (!command)
+ goto oom;
+ list_to_symbol(command)->type = LIL_SYMBOL_COMMAND;
+
+ do {
+ word = parse_word(p, false);
+ if (!word)
+ goto err;
+
+ if (lil_list_append(command, word))
+ goto oom;
+
+ if (eof(p))
+ return command;
+
+ switch (peek(p)) {
+ CASE_WHITESPACE:
+ do {
+ switch (peek(p)) {
+ case '\\':
+ if (peek2(p) != '\n')
+ break;
+ fallthrough;
+ CASE_SPACE:
+ parse_whitespace(p);
+ continue;
+ case ']':
+ CASE_TERMINATOR:
+ return command;
}
- } else if (lil->code[lil->head] == sc) {
- lil->head++;
+
break;
- } else {
- lil_append_char(val, lil->code[lil->head]);
- }
- lil->head++;
+ } while (!eof(p));
+
+ continue;
+ case ']':
+ CASE_TERMINATOR:
+ return command;
}
- } else {
- start = lil->head;
- while (lil->head < lil->clen &&
- !isspace(lil->code[lil->head]) &&
- !islilspecial(lil->code[lil->head]))
- lil->head++;
- val = alloc_value_len(lil->code + start, lil->head - start);
- }
- return val ? val : alloc_value(NULL);
+
+ err_unexpected(p);
+ goto err;
+ } while (!eof(p));
+
+ return command;
+
+oom:
+ err_oom(p);
+err:
+ lil_free_list(command);
+ return NULL;
}
-static struct lil_list *substitute(struct lil *lil)
+/**
+ * parse_terminator() - Parse the end of a command
+ * @p: The parser
+ *
+ * FIRST(terminator) = ';' | '\n' | '#' ;
+ */
+static void parse_terminator(struct lil_parser *p)
{
- struct lil_list *words = lil_alloc_list();
+ switch (peek(p)) {
+ case '\n':
+ case ';':
+ pop(p);
+ return;
+ case '#':
+ parse_comment(p);
+ return;
+ }
+ assert(0);
+}
- lil_skip_spaces(lil);
- while (lil->head < lil->clen && !ateol(lil) && !lil->err) {
- struct lil_value *w = alloc_value(NULL);
+/**
+ * parse_script - Parse a script
+ *
+ * FIRST(script) = FIRST(whitespace) | FIRST(command) | FIRST(terminator) ;
+ * FOLLOW(script) = ']' | ;
+ *
+ * Return: A symbol containing a list of commands which compose the script, or
+ * %NULL on error
+ */
+static struct lil_list *parse_script(struct lil_parser *p)
+{
+ struct lil_list *command;
+ struct lil_list *script = lil_alloc_list();
- do {
- size_t head = lil->head;
- struct lil_value *wp = next_word(lil);
+ if (!script)
+ goto oom;
+ list_to_symbol(script)->type = LIL_SYMBOL_SCRIPT;
- if (head ==
- lil->head) { /* something wrong, the parser can't proceed */
- lil_free_value(w);
- lil_free_value(wp);
- lil_free_list(words);
- return NULL;
+ do {
+ while (!eof(p)) {
+ switch (peek(p)) {
+ case '\\':
+ if (peek2(p) != '\n')
+ break;
+ fallthrough;
+ CASE_SPACE:
+ parse_whitespace(p);
+ continue;
}
+ break;
+ }
- lil_append_val(w, wp);
- lil_free_value(wp);
- } while (lil->head < lil->clen &&
- !eolchar(lil->code[lil->head]) &&
- !isspace(lil->code[lil->head]) && !lil->err);
- lil_skip_spaces(lil);
+ switch (peek(p)) {
+ case '\\':
+ if (peek2(p) != '\n')
+ break;
+ fallthrough;
+ CASE_SPACE:
+ err_unexpected(p);
+ goto err;
+ CASE_TERMINATOR:
+ parse_terminator(p);
+ continue;
+ case ']':
+ return script;
+ }
- lil_list_append(words, w);
+ command = parse_command(p);
+ if (!command)
+ goto err;
+
+ if (lil_list_append(script, list_to_symbol(command)))
+ goto oom;
+ } while (!eof(p));
+ return script;
+
+oom:
+ err_oom(p);
+err:
+ lil_free_list(script);
+ return NULL;
+}
+
+#if IS_ENABLED(CONFIG_LIL_DEBUG)
+static void do_print_symbol(struct lil_symbol *sym, unsigned int level)
+{
+ unsigned int i;
+
+ for (i = 0; i < level; i++)
+ putc('\t');
+
+ switch (sym->type) {
+ case LIL_SYMBOL_VALUE:
+ puts(lil_to_string(&sym->value));
+ putc('\n');
+ break;
+ case LIL_SYMBOL_SCRIPT:
+ puts("<script>\n");
+ goto list;
+ case LIL_SYMBOL_COMMAND:
+ puts("<command>\n");
+ goto list;
+ case LIL_SYMBOL_LIST:
+ puts("<list>\n");
+list:
+ for (i = 0; i < sym->list.c; i++)
+ do_print_symbol(sym->list.v[i], level + 1);
+ break;
+ case LIL_SYMBOL_VARIABLE:
+ puts("<variable>\n");
+ do_print_symbol(sym->symbol, level + 1);
+ break;
+ default:
+ puts("<unknown>\n");
+ break;
+ }
+}
+
+/*
+ * Helper function for printing out symbols; insert where you would like to
+ * debug something
+ */
+static void __maybe_unused print_symbol(struct lil_symbol *sym)
+{
+ return do_print_symbol(sym, 0);
+}
+#endif
+
+/**
+ * lil_parser_init() - Initialize a parser
+ * @p: The parser to initialize
+ *
+ * This initializes the parser by setting @p->pos and @p->err.
+ */
+static void lil_parser_init(struct lil_parser *p)
+{
+ p->pos.head = 0;
+#if IS_ENABLED(CONFIG_LIL_FULL)
+ p->pos.line = 1;
+ p->pos.column = 1;
+#endif
+ p->err.type = PERR_NONE;
+}
+
+static void parser_set_error(struct lil *lil, struct lil_parser *p)
+{
+ switch (p->err.type) {
+ case PERR_OOM:
+ lil_set_error(lil, LIL_ERR_OOM,
+ IS_ENABLED(CONFIG_LIL_DEBUG) ? p->err.func : NULL);
+ return;
+ case PERR_EXPECTED:
+ case PERR_UNEXPECTED: {
+ enum lil_error err;
+
+ if (eof(p))
+ err = LIL_ERR_EOF;
+ else
+ err = LIL_ERR_SYNTAX;
+
+ if (IS_ENABLED(CONFIG_LIL_FULL)) {
+ char fmt[] = "character '%c'";
+ char ubuf[sizeof(fmt)], buf[128];
+ char *unexpected = ubuf;
+ size_t pos = 0;
+
+ if (eof(p))
+ unexpected = "end of file";
+ else if (peek(p))
+ snprintf(ubuf, sizeof(ubuf), fmt, peek(p));
+ else
+ unexpected = "character '\\0'";
+
+#define format(fmt, ...) \
+ pos += snprintf(buf + pos, sizeof(buf) - pos, fmt, __VA_ARGS__)
+#define format_pos(pos) format("%zu:%zu", pos.line, pos.column)
+
+ format_pos(p->pos);
+ format(": unexpected %s", unexpected);
+ if (p->err.type == PERR_EXPECTED || eof(p)) {
+ format(" while parsing '%c' at ",
+ p->code[p->err.matching.head]);
+ format_pos(p->err.matching);
+ }
+ if (p->err.type == PERR_EXPECTED)
+ format("; expected '%c'", p->err.expected);
+ if (IS_ENABLED(CONFIG_LIL_DEBUG))
+ format(" in %s", p->err.func);
+
+ lil_set_error(lil, err, buf);
+ } else {
+ lil_set_error(lil, err, "syntax error");
+ }
+ return;
+ }
+ case PERR_NONE:
+ return;
+ }
+ log_debug("unknown error %d\n", p->err.type);
+ assert(0);
+ lil_set_error(lil, LIL_ERR_CASE, NULL);
+}
+
+/**
+ * lil_parse() - Parse a script
+ * @name: The name of what we are parsing. This will be prepended to @error.
+ * @code: The script to parse
+ * @codelen: The length of @code, or %0 to use strlen
+ * @error: A pointer which will be set to a string describing the error (if
+ * there is one)
+ *
+ * Return: A list of commands in the script, which may be passed to lil_eval(),
+ * or %NULL on error.
+ */
+struct lil_list *lil_parse(struct lil *lil, const char *code, size_t codelen)
+{
+ struct lil_list *script;
+ struct lil_parser p;
+
+ p.code = code;
+ p.len = codelen ? codelen : strlen(code);
+ lil_parser_init(&p);
+ script = parse_script(&p);
+ if (script && !eof(&p)) {
+ lil_free_list(script);
+ script = NULL;
+ err_unexpected(&p);
}
- return words;
+ parser_set_error(lil, &p);
+ return script;
}
-struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code)
+static struct lil_list *substitute(struct lil *lil, struct lil_list *list);
+
+/**
+ * concat() - Convert a list into a value
+ * @lil: The interpreter
+ * @list: The list to convert
+ *
+ * This evaluates each symbol in the list, and then concatenates them together
+ * with no spaces. For example, the list ``{a $b [c]}`` might result in ``ade``
+ * if the value of variable ``b`` is ``d`` and the result of the command ``c``
+ * is ``e``.
+ *
+ * This is used for evaluating quoted words. For example, the word
+ * ``"a${b}[c]"`` would be parsed as a list, and therefore needs to be pasted
+ * together without the usual intervening spaces.
+ *
+ * Return: A value containing the evaluated, concatenated symbols, or %NULL on
+ * error
+ */
+static struct lil_value *concat(struct lil *lil, struct lil_list *list)
{
- const char *save_code = lil->code;
- size_t save_clen = lil->clen;
- size_t save_head = lil->head;
- int save_igeol = lil->ignoreeol;
- struct lil_list *words;
+ size_t i;
+ struct lil_list *parts;
+ struct lil_value *val = alloc_value(NULL);
- lil->code = lil_to_string(code);
- lil->clen = code->l;
- lil->head = 0;
- lil->ignoreeol = 1;
+ if (!val)
+ goto oom;
- words = substitute(lil);
- if (!words)
- words = lil_alloc_list();
+ parts = substitute(lil, list);
+ if (!parts)
+ goto err;
- lil->code = save_code;
- lil->clen = save_clen;
- lil->head = save_head;
- lil->ignoreeol = save_igeol;
- return words;
-}
-
-struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code)
-{
- struct lil_list *words = lil_subst_to_list(lil, code);
- struct lil_value *val;
-
- val = lil_list_to_value(words, 0);
- lil_free_list(words);
+ assert(list_to_symbol(list)->type == LIL_SYMBOL_LIST);
+ for (i = 0; i < parts->c; i++) {
+ assert(parts->v[i]->type == LIL_SYMBOL_VALUE);
+ if (lil_append_val(val, &parts->v[i]->value))
+ goto oom;
+ }
+ lil_free_list(parts);
return val;
+
+oom:
+ lil_set_error_oom(lil);
+err:
+ lil_free_list(parts);
+ lil_free_value(val);
+ return NULL;
+}
+
+/**
+ * substitute_symbol() - Substitute one symbol
+ * @lil: The interpreter
+ * @sym: The symbol to substitute
+ *
+ * This performs substitutions on one symbol. For values, this is a no-op. Lists
+ * (from parse_quote()) are concat()enated. Variables are dereferenced.
+ * Sub-scripts (from parse_bracket()) are lil_eval()ed. NB: it is an error to
+ * pass a command to this function! Use substitute() instead.
+ *
+ * Return: The value of the symbol in the current context, or %NULL on error
+ */
+static struct lil_value *substitute_symbol(struct lil *lil,
+ struct lil_symbol *sym)
+{
+ switch (sym->type) {
+ case LIL_SYMBOL_VALUE:
+ return lil_clone_value(&sym->value);
+ case LIL_SYMBOL_LIST:
+ return concat(lil, &sym->list);
+ case LIL_SYMBOL_VARIABLE: {
+ struct lil_value *val, *var =
+ substitute_symbol(lil, sym->symbol);
+
+ if (!var)
+ return NULL;
+
+ assert(value_to_symbol(var)->type == LIL_SYMBOL_VALUE);
+ val = lil_get_var(lil, lil_to_string(var));
+ lil_free_value(var);
+ return lil_clone_value(val);
+ }
+ case LIL_SYMBOL_SCRIPT:
+ return lil_eval(lil, &sym->list, false);
+ case LIL_SYMBOL_COMMAND:
+ break;
+ }
+ log_debug("invalid type %d\n", sym->type);
+ assert(0);
+ lil_set_error(lil, LIL_ERR_CASE, NULL);
+ return NULL;
+}
+
+/**
+ * substitute() - Substitute a list of symbols
+ * @lil: The interpreter
+ * @list: The list of symbols
+ *
+ * This performs substitutions using substitute_symbol() on each of the symbols
+ * in the list. NB: it is an error to pass a script to this function. use
+ * lil_eval() or substitute_symbol() instead.
+ *
+ * Return: A list of &struct lil_value, or %NULL on error.
+ */
+static struct lil_list *substitute(struct lil *lil, struct lil_list *list)
+{
+ size_t i;
+ enum lil_symbol_type type = list_to_symbol(list)->type;
+ struct lil_value *val = NULL;
+ struct lil_list *values;
+
+ values = lil_alloc_list();
+ if (!values)
+ goto oom;
+ assert(type == LIL_SYMBOL_COMMAND || type == LIL_SYMBOL_LIST);
+ list_to_symbol(values)->type = type;
+
+ for (i = 0; i < list->c; i++) {
+ struct lil_value *val = substitute_symbol(lil, list->v[i]);
+
+ if (!val)
+ goto err;
+
+ if (lil_list_append(values, val))
+ goto oom;
+ }
+ return values;
+
+oom:
+ lil_set_error_oom(lil);
+err:
+ lil_free_value(val);
+ lil_free_list(values);
+ return NULL;
}
static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
@@ -1197,7 +2096,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
val, LIL_SETVAR_LOCAL_NEW);
}
}
- r = lil_parse_value(lil, cmd->code, 1);
+ r = lil_parse_value(lil, cmd->code, true);
lil_pop_env(lil);
}
@@ -1205,102 +2104,329 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
return r;
}
-struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
- int funclevel)
+/**
+ * lil_eval() - Evaluate a script
+ * @lil: The interpreter
+ * @script: The script to be evaluated
+ * @new_frame: Whether this represents a new stack frame. If @new_frame is set,
+ * then subcommands which set &struct lil.breakrun (such as
+ * ``return``) will stop here. Otherwise, we will pass the buck and
+ * let our caller deal with it. Top-level calls to this lil_eval()
+ * should always set @new_frame.
+ *
+ * This evaluates a script. Each command in the script is substitute()ed, and
+ * then is passed to run_cmd().
+ *
+ * Return: The result of the last command in @script, &struct lil.retval if it
+ * is set, or %NULL on error.
+ */
+struct lil_value *lil_eval(struct lil *lil, struct lil_list *script,
+ bool new_frame)
{
- const char *save_code = lil->code;
- size_t save_clen = lil->clen;
- size_t save_head = lil->head;
- struct lil_value *val = NULL;
- struct lil_list *words = NULL;
+ size_t i;
+ struct lil_value *ret = NULL;
- if (!save_code)
- lil->rootcode = code;
- lil->code = code;
- lil->clen = codelen ? codelen : strlen(code);
- lil->head = 0;
+ if (!lil->depth)
+ lil->err = 0;
- lil_skip_spaces(lil);
- lil->parse_depth++;
- if (CONFIG_LIL_RECLIMIT && lil->parse_depth > CONFIG_LIL_RECLIMIT) {
+ if (CONFIG_LIL_RECLIMIT && lil->depth++ > CONFIG_LIL_RECLIMIT) {
lil_set_error(lil, LIL_ERR_DEPTH, "recursion limit reached");
- goto cleanup;
+ return NULL;
}
- if (lil->parse_depth == 1)
- lil->err = LIL_ERR_NONE;
+ assert(list_to_symbol(script)->type == LIL_SYMBOL_SCRIPT);
+ for (i = 0; i < script->c; i++) {
+ struct lil_list *command;
- if (funclevel)
- lil->env->breakrun = 0;
+ if (ret)
+ lil_free_value(ret);
+ ret = NULL;
- while (lil->head < lil->clen && !lil->err) {
- if (words)
- lil_free_list(words);
+ assert(script->v[i]->type == LIL_SYMBOL_COMMAND);
+ command = substitute(lil, &script->v[i]->list);
+ if (!command)
+ break;
- if (val)
- lil_free_value(val);
- val = NULL;
+ if (command->c) {
+ const char *funcname =
+ lil_to_string(lil_list_get(command, 0));
+ struct lil_func *func = lil_find_cmd(lil, funcname);
+
+ if (func)
+ ret = run_cmd(lil, func, command);
+ else if (funcname[0])
+ lil_set_error_nocmd(lil, funcname);
+ }
+ lil_free_list(command);
if (ctrlc()) {
- lil_set_error(lil, LIL_ERR_INTR, "interrupted");
- goto cleanup;
+ lil_set_error_intr(lil);
+ break;
+ } else if (lil->err || lil->env->breakrun) {
+ break;
}
-
- words = substitute(lil);
- if (!words || lil->err)
- goto cleanup;
-
- if (words->c) {
- const char *cmdname =
- lil_to_string(lil_list_get(words, 0));
- struct lil_func *cmd = lil_find_cmd(lil, cmdname);
-
- if (!cmd) {
- if (cmdname[0]) {
- lil_set_error_nocmd(lil, cmdname);
- goto cleanup;
- }
- } else {
- val = run_cmd(lil, cmd, words);
- }
-
- if (lil->env->breakrun)
- goto cleanup;
- }
-
- lil_skip_spaces(lil);
- while (ateol(lil))
- lil->head++;
- lil_skip_spaces(lil);
}
-cleanup:
- if (words)
- lil_free_list(words);
- lil->code = save_code;
- lil->clen = save_clen;
- lil->head = save_head;
-
- if (funclevel && lil->env->retval_set) {
- if (val)
- lil_free_value(val);
- val = lil->env->retval;
+ if (new_frame && lil->env->retval_set) {
+ lil_free_value(ret);
+ ret = lil->env->retval;
lil->env->retval = NULL;
- lil->env->retval_set = 0;
- lil->env->breakrun = 0;
+ lil->env->retval_set = false;
+ lil->env->breakrun = false;
}
- lil->parse_depth--;
- return val ? val : alloc_value(NULL);
+ if (lil->err) {
+ lil_free_value(ret);
+ ret = NULL;
+ }
+
+ lil->depth--;
+ return ret;
+}
+
+struct lil_value *lil_parse_eval(struct lil *lil, const char *code,
+ size_t codelen, bool new_frame)
+{
+ struct lil_value *result;
+ struct lil_list *script = lil_parse(lil, code, codelen);
+
+ if (!script)
+ return NULL;
+ result = lil_eval(lil, script, new_frame);
+ lil_free_list(script);
+ return result;
+}
+
+/**
+ * parse_list() - Parse a list
+ *
+ * FIRST(list) = FIRST(space) | FIRST(word) | ;
+ * FOLLOW(list) = ;
+ *
+ * Return: A list of words in the list, or %NULL on error.
+ */
+static struct lil_list *parse_list(struct lil_parser *p)
+{
+ struct lil_list *list = lil_alloc_list();
+
+ if (!list)
+ goto oom;
+
+ while (!eof(p)) {
+ switch (peek(p)) {
+ case '\n':
+ CASE_SPACE:
+ pop(p);
+ continue;
+ }
+ break;
+ }
+
+ if (eof(p))
+ return list;
+
+ do {
+ struct lil_symbol *sym = parse_word(p, true);
+
+ if (!sym)
+ goto err;
+
+ assert(sym->type == LIL_SYMBOL_VALUE);
+ if (lil_list_append(list, &sym->value))
+ goto oom;
+
+ if (eof(p))
+ return list;
+
+ do {
+ switch (peek(p)) {
+ case '\n':
+ CASE_SPACE:
+ pop(p);
+ continue;
+ }
+ break;
+ } while (!eof(p));
+ } while (!eof(p));
+
+ return list;
+
+oom:
+ err_oom(p);
+err:
+ lil_free_list(list);
+ return NULL;
+}
+
+/* FIXME: rename this to something better */
+struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code)
+{
+ struct lil_list *list;
+ struct lil_parser p;
+
+ p.code = code->d;
+ p.len = code->l;
+ lil_parser_init(&p);
+ list = parse_list(&p);
+ assert(!list || eof(&p));
+ parser_set_error(lil, &p);
+
+ /*
+ * FIXME: Callers of this function do not expect NULL, so we must always
+ * allocate something for them. Of course, lil_alloc_list can also fail,
+ * so we're screwed either way.
+ */
+ if (!list)
+ list = lil_alloc_list();
+ return list;
+}
+
+struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code)
+{
+ struct lil_symbol *sym;
+ struct lil_parser p;
+
+ p.code = code->d;
+ p.len = code->l;
+ lil_parser_init(&p);
+ sym = parse_quote(&p, -1, false);
+ parser_set_error(lil, &p);
+ if (!sym)
+ return NULL;
+
+ assert(eof(&p));
+ assert(sym->type == LIL_SYMBOL_LIST);
+ return substitute_symbol(lil, sym);
}
struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
- int funclevel)
+ bool new_frame)
{
if (!val || !val->d || !val->l)
return alloc_value(NULL);
- return lil_parse(lil, val->d, val->l, funclevel);
+ return lil_parse_eval(lil, val->d, val->l, new_frame);
+}
+
+/**
+ * enum lil_list_flags - Whether an item needs special treatment
+ * @NEEDS_NOTHING: This item needs no changes
+ * @NEEDS_BRACES: This item needs to be enclosed in braces
+ * @NEEDS_SINGLE: This item needs to be enclosed in double quotes
+ * @NEEDS_DOUBLE: This item needs to be enclosed in single quotes
+ * @NEEDS_QUOTES: This item needs to be enclosed in (some kind of) quotes
+ */
+enum needs {
+ NEEDS_NOTHING,
+ NEEDS_BRACES,
+ NEEDS_DOUBLE,
+ NEEDS_SINGLE,
+ NEEDS_QUOTES = NEEDS_DOUBLE,
+};
+
+static enum needs item_needs(const char *str, size_t n)
+{
+ bool was_backslash = false;
+ int nesting = 0;
+ enum needs needs = NEEDS_NOTHING;
+ size_t i, sq = 0, dq = 0;
+
+ if (!str || !str[0])
+ return NEEDS_BRACES;
+
+ for (i = 0; i < n; i++) {
+ switch (str[i]) {
+ case '{':
+ nesting++;
+ goto braces;
+ case '}':
+ nesting--;
+ if (nesting < 0)
+ needs = NEEDS_QUOTES;
+ goto braces;
+ case '\\':
+ was_backslash = !was_backslash;
+ if (needs == NEEDS_NOTHING)
+ needs = NEEDS_BRACES;
+ continue;
+ case '\'':
+ sq++;
+ goto braces;
+ case '"':
+ dq++;
+ fallthrough;
+ case '\n':
+ CASE_SPACE:
+braces:
+ if (needs == NEEDS_NOTHING)
+ needs = NEEDS_BRACES;
+ }
+ was_backslash = false;
+ }
+
+ if (nesting || was_backslash)
+ needs = NEEDS_QUOTES;
+
+ if (needs == NEEDS_QUOTES && dq > sq)
+ needs = NEEDS_SINGLE;
+ return needs;
+}
+
+struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape)
+{
+ struct lil_value *val = alloc_value(NULL);
+ size_t i, j;
+
+ for (i = 0; i < list->c; i++) {
+ char q;
+ struct lil_value *item = lil_list_get(list, i);
+ enum needs needs;
+
+ if (do_escape)
+ needs = item_needs(lil_to_string(item), item->l);
+ else
+ needs = NEEDS_NOTHING;
+
+ if (i)
+ lil_append_char(val, ' ');
+
+ switch (needs) {
+ case NEEDS_NOTHING:
+ if (lil_append_val(val, item))
+ goto err;
+ continue;
+ case NEEDS_BRACES:
+ if (lil_append_char(val, '{') ||
+ lil_append_val(val, item) ||
+ lil_append_char(val, '}'))
+ goto err;
+ continue;
+ case NEEDS_DOUBLE:
+ q = '"';
+ goto quote;
+ case NEEDS_SINGLE:
+ q = '\'';
+quote:
+ if (lil_append_char(val, q))
+ goto err;
+ for (j = 0; j < item->l; j++) {
+ char c = item->d[j];
+
+ if (c == '\\' || c == q)
+ if (lil_append_char(val, '\\'))
+ goto err;
+ if (lil_append_char(val, c))
+ goto err;
+ }
+ if (lil_append_char(val, q))
+ goto err;
+ }
+ }
+ return val;
+
+err:
+ lil_free_value(val);
+ return NULL;
}
static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg)
@@ -1801,7 +2927,7 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
struct expreval ee;
if (ctrlc()) {
- lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+ lil_set_error_intr(lil);
return NULL;
}
@@ -2076,18 +3202,6 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
if (!strcmp(type, "error"))
return lil->err_msg ? lil_alloc_string(lil->err_msg) : NULL;
- if (!strcmp(type, "this")) {
- struct lil_env *env = lil->env;
-
- while (env != lil->rootenv && !env->func)
- env = env->parent;
-
- if (env == lil->rootenv)
- return lil_alloc_string(lil->rootcode);
-
- return env->func ? env->func->code : NULL;
- }
-
if (!strcmp(type, "name")) {
struct lil_env *env = lil->env;
@@ -2564,10 +3678,12 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
lil_set_var(lil, varname, lil_list_get(list, i),
LIL_SETVAR_LOCAL_ONLY);
rv = lil_parse_value(lil, argv[codeidx], 0);
- if (rv->l)
- lil_list_append(rlist, rv);
- else
- lil_free_value(rv);
+ if (rv) {
+ if (rv->l)
+ lil_list_append(rlist, rv);
+ else
+ lil_free_value(rv);
+ }
if (lil->env->breakrun || lil->err)
break;
diff --git a/include/cli_lil.h b/include/cli_lil.h
index 40c822401e..290329372a 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -133,10 +133,13 @@ void lil_free(struct lil *lil);
int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc);
-struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
- int funclevel);
+struct lil_list *lil_parse(struct lil *lil, const char *code, size_t codelen);
+struct lil_value *lil_eval(struct lil *lil, struct lil_list *script,
+ bool new_frame);
+struct lil_value *lil_parse_eval(struct lil *lil, const char *code,
+ size_t codelen, bool new_frame);
struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
- int funclevel);
+ bool new_frame);
enum lil_error lil_error(struct lil *lil, const char **msg);
@@ -158,7 +161,7 @@ void lil_free_list(struct lil_list *list);
int lil_list_append(struct lil_list *list, void *item);
size_t lil_list_size(struct lil_list *list);
struct lil_value *lil_list_get(struct lil_list *list, size_t index);
-struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape);
+struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape);
struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code);
struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code);
diff --git a/test/cmd/lil.c b/test/cmd/lil.c
index 896b2fed15..fb33fa83a6 100644
--- a/test/cmd/lil.c
+++ b/test/cmd/lil.c
@@ -23,7 +23,7 @@ const char helpers[] =
"proc assert_err {cmd} {"
"set ok 1;"
"try {upeval $cmd; set ok 0} {};"
- "assert {$ok};"
+ "if not $ok { error $cmd }"
"};"
"proc asserteq {expr1 expr2} {"
"set val1 [upeval 'expr \"$expr1\"'];"
@@ -65,14 +65,13 @@ static const struct {
{"and",
"proc and args {"
"foreach [slice $args 1] {"
- "upeval 'downeval \\'set v \\'\\[${i}\\]';"
- "if not $v { return 0 }"
+ "upeval 'downeval \"set v \\[${i}\\]\"';"
+ "if not $v { return 0 };"
"};"
"return 1"
"};"
"set a 0;"
- "set final [and {set a 3} {return 0} {set a 32}];"
- "asserteq 0 {$final};"
+ "asserteq 0 {[and {set a 3} {return 0} {set a 32}]};"
"assert 3 {$a};"
},
{"assert",
@@ -108,11 +107,10 @@ static const struct {
"asserteq -6 {1 +~ (2*3 )};"
"asserteq -6 {~(2*3)+1};"
"asserteq 0 {1*!(2+2)};"
- "asserteq -1 {~!(!{})};"
+ "asserteq -1 {~!(!)};"
"asserteq 1 {1 +~*(2*3)};"
"asserteq 1 {'hello'};"
"asserteq 0 {0};"
- "asserteq 0 {{}};"
"asserteq 1 {()};"
"asserteq 1 {( )};"
"asserteq_str '' {[expr]};"
@@ -144,6 +142,7 @@ static const struct {
"return $ret"
"};"
"set list [list {bad's day} {good's day} eh??];"
+ "asserteq_list [lapply $list length] [list 9 10 4];"
"asserteq_list [lapply $list split] [list "
"[list {bad's} day] "
"[list {good's} day] "
@@ -159,37 +158,22 @@ static const struct {
"asserteq_str baz {[index $l 2]};"
"append l 'Hello, world!';"
"asserteq_list $l [list foo bar baz bad 'Hello, world!'];"
- "set l [subst $l];"
- "asserteq_list $l [list foo bar baz bad Hello, world!];"
"lmap $l foox barx bamia;"
"asserteq_str foo {$foox};"
"asserteq_str bar {$barx};"
"asserteq_str baz {$bamia};"
- "set l {one # linebreaks are ignored in list parsing mode\n"
+ "set l {one # linebreaks are whitespace in lists\n"
"\n"
- "two;three # a semicolon still counts as line break\n"
- " # (which in list mode is treated as a\n"
- " # separator for list entries)\n"
- "# of course a semicolon inside quotes is treated like normal\n"
- "three';'and';a;half'\n"
- "# like in code mode, a semicolon will stop the comment; four\n"
- "\n"
- "# below we have a quote, square brackets for inline\n"
- "# expansions are still taken into consideration\n"
- "[quote {this line will be ignored completely\n"
- " as will this line and instead be replaced\n"
- " with the 'five' below since while in code\n"
- " mode (that is, inside the brackets here)\n"
- " linebreaks are still processed}\n"
- " quote five]\n"
+ "two;three # a semicolon does not count as a line break\n"
+ "# a semicolon will not stop the comment; four\n"
"\n"
"# The curly brackets are also processed so the next three\n"
"# lines will show up as three separate lines\n"
"{six\n"
"seven\n"
"eight}}\n"
- "asserteq_list $l [list one two three 'three;and;a;half' four "
- "five 'six\\nseven\\neight'];"
+ "asserteq_list $l [list one 'two;three' "
+ "'six\\nseven\\neight'];"
},
{"local",
"proc bits-for {x} {"
@@ -210,33 +194,13 @@ static const struct {
"asserteq 45 {$x};"
"asserteq 6 {$bitsx}"
},
- {"multiline comment",
+ {"comment",
"# this line will not be executed, but the following will\n"
"set ok1 1\n"
- "## This is a multiline comment\n"
- " which, as the name implies,\n"
- " spans multiple lines.\n"
- "set ok2 1\n"
- " the code above wouldn't execute,\n"
- " but this will --> ##set ok3 1\n"
- "### more than two #s will not count as multiline comments\n"
- "set ok4 1\n"
- "# Note that semicolons can be used as linebreaks so\n"
- "# this code will be executed: ; set ok5 1\n"
- "##\n"
- " ...however inside multiline comments semicolons do not\n"
- " stop the comment section (pretty much like linebreaks)\n"
- " and this code will not be executed: ; set ok6 1\n"
- "##\n"
- "# Also note that unlike in regular code, semicolons cannot\n"
- "# be escaped in single-line comments, e.g.: ; set ok7 1\n"
+ "# Note that semicolons cannot be used as linebreaks so\n"
+ "# this code will not be executed: ; set ok5 1\n"
"asserteq_str 1 {$ok1};"
- "assert {![reflect has-var ok2]}"
- "asserteq_str 1 {$ok3};"
- "asserteq_str 1 {$ok4};"
- "asserteq_str 1 {$ok5};"
- "assert {![reflect has-var ok6]}"
- "asserteq_str 1 {$ok7};"
+ "assert {! [reflect has-var ok5]}"
},
{"multiline code",
"asserteq_list [list hello \\\n"
@@ -270,7 +234,7 @@ static const struct {
"asserteq 10 {[strpos $a string]};"
"asserteq 16 {[strpos $b string]};"
"asserteq -78 {[compare $a $b]};"
- "assert {![streq $a $b]};"
+ "assert {! [streq $a $b]};"
"asserteq_str 'This is a foo' {[repstr $a string foo]};"
"asserteq_str 'This is another foo' {[repstr $b string foo]};"
"asserteq_list [split $a] [list This is a string];"
@@ -321,9 +285,10 @@ static int lib_test_lil(struct unit_test_state *uts)
enum lil_error err;
struct lil *lil = lil_new(NULL);
- lil_free_value(lil_parse(lil, helpers, sizeof(helpers) - 1, 0));
+ lil_free_value(lil_parse_eval(lil, helpers, sizeof(helpers) - 1,
+ true));
ut_asserteq(LIL_ERR_NONE, lil_error(lil, &err_msg));
- lil_free_value(lil_parse(lil, lil_tests[i].cmd, 0, 0));
+ lil_free_value(lil_parse_eval(lil, lil_tests[i].cmd, 0, true));
err = lil_error(lil, &err_msg);
if (err) {
ut_failf(uts, __FILE__, __LINE__, __func__,
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 22/28] env: Add a priv pointer to hwalk_r
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (20 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 21/28] cli: lil: Add a distinct parsing step Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 20:10 ` Tom Rini
2021-07-01 6:16 ` [RFC PATCH 23/28] cli: lil: Handle OOM for hm_put Sean Anderson
` (7 subsequent siblings)
29 siblings, 1 reply; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This allows callers of hwalk_r to pass data to their callback. This mirrors
e.g. env_attr_walk.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
cmd/nvedit.c | 8 ++++----
env/callback.c | 4 ++--
env/flags.c | 4 ++--
include/search.h | 2 +-
lib/hashtable.c | 5 +++--
5 files changed, 12 insertions(+), 11 deletions(-)
diff --git a/cmd/nvedit.c b/cmd/nvedit.c
index d14ba10cef..b855e502c0 100644
--- a/cmd/nvedit.c
+++ b/cmd/nvedit.c
@@ -481,7 +481,7 @@ static int print_static_binding(const char *var_name, const char *callback_name,
return 0;
}
-static int print_active_callback(struct env_entry *entry)
+static int print_active_callback(struct env_entry *entry, void *priv)
{
struct env_clbk_tbl *clbkp;
int i;
@@ -544,7 +544,7 @@ int do_env_callback(struct cmd_tbl *cmdtp, int flag, int argc,
puts("Active callback bindings:\n");
printf("\t%-20s %-20s\n", "Variable Name", "Callback Name");
printf("\t%-20s %-20s\n", "-------------", "-------------");
- hwalk_r(&env_htab, print_active_callback);
+ hwalk_r(&env_htab, print_active_callback, NULL);
return 0;
}
#endif
@@ -563,7 +563,7 @@ static int print_static_flags(const char *var_name, const char *flags,
return 0;
}
-static int print_active_flags(struct env_entry *entry)
+static int print_active_flags(struct env_entry *entry, void *priv)
{
enum env_flags_vartype type;
enum env_flags_varaccess access;
@@ -617,7 +617,7 @@ int do_env_flags(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[])
"Variable Access");
printf("\t%-20s %-20s %-20s\n", "-------------", "-------------",
"---------------");
- hwalk_r(&env_htab, print_active_flags);
+ hwalk_r(&env_htab, print_active_flags, NULL);
return 0;
}
#endif
diff --git a/env/callback.c b/env/callback.c
index 638a02b28f..47075acb92 100644
--- a/env/callback.c
+++ b/env/callback.c
@@ -83,7 +83,7 @@ void env_callback_init(struct env_entry *var_entry)
* Called on each existing env var prior to the blanket update since removing
* a callback association should remove its callback.
*/
-static int clear_callback(struct env_entry *entry)
+static int clear_callback(struct env_entry *entry, void *priv)
{
entry->callback = NULL;
@@ -127,7 +127,7 @@ static int on_callbacks(const char *name, const char *value, enum env_op op,
int flags)
{
/* remove all callbacks */
- hwalk_r(&env_htab, clear_callback);
+ hwalk_r(&env_htab, clear_callback, NULL);
/* configure any static callback bindings */
env_attr_walk(ENV_CALLBACK_LIST_STATIC, set_callback, NULL);
diff --git a/env/flags.c b/env/flags.c
index e3e833c433..f2e36e3dd3 100644
--- a/env/flags.c
+++ b/env/flags.c
@@ -468,7 +468,7 @@ void env_flags_init(struct env_entry *var_entry)
* Called on each existing env var prior to the blanket update since removing
* a flag in the flag list should remove its flags.
*/
-static int clear_flags(struct env_entry *entry)
+static int clear_flags(struct env_entry *entry, void *priv)
{
entry->flags = 0;
@@ -503,7 +503,7 @@ static int on_flags(const char *name, const char *value, enum env_op op,
int flags)
{
/* remove all flags */
- hwalk_r(&env_htab, clear_flags);
+ hwalk_r(&env_htab, clear_flags, NULL);
/* configure any static flags */
env_attr_walk(ENV_FLAGS_LIST_STATIC, set_flags, NULL);
diff --git a/include/search.h b/include/search.h
index d0bb44388e..4a0828fb8d 100644
--- a/include/search.h
+++ b/include/search.h
@@ -105,7 +105,7 @@ int himport_r(struct hsearch_data *htab, const char *env, size_t size,
/* Walk the whole table calling the callback on each element */
int hwalk_r(struct hsearch_data *htab,
- int (*callback)(struct env_entry *entry));
+ int (*callback)(struct env_entry *entry, void *priv), void *priv);
/* Flags for himport_r(), hexport_r(), hdelete_r(), and hsearch_r() */
#define H_NOCLEAR (1 << 0) /* do not clear hash table before importing */
diff --git a/lib/hashtable.c b/lib/hashtable.c
index ff5ff72639..425a880222 100644
--- a/lib/hashtable.c
+++ b/lib/hashtable.c
@@ -998,14 +998,15 @@ end:
* Walk all of the entries in the hash, calling the callback for each one.
* this allows some generic operation to be performed on each element.
*/
-int hwalk_r(struct hsearch_data *htab, int (*callback)(struct env_entry *entry))
+int hwalk_r(struct hsearch_data *htab,
+ int (*callback)(struct env_entry *entry, void *priv), void *priv)
{
int i;
int retval;
for (i = 1; i <= htab->size; ++i) {
if (htab->table[i].used > 0) {
- retval = callback(&htab->table[i].entry);
+ retval = callback(&htab->table[i].entry, priv);
if (retval)
return retval;
}
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 22/28] env: Add a priv pointer to hwalk_r
2021-07-01 6:16 ` [RFC PATCH 22/28] env: Add a priv pointer to hwalk_r Sean Anderson
@ 2021-07-01 20:10 ` Tom Rini
0 siblings, 0 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-01 20:10 UTC (permalink / raw)
To: Sean Anderson
Cc: u-boot, Marek Behún, Wolfgang Denk, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 282 bytes --]
On Thu, Jul 01, 2021 at 02:16:05AM -0400, Sean Anderson wrote:
> This allows callers of hwalk_r to pass data to their callback. This mirrors
> e.g. env_attr_walk.
>
> Signed-off-by: Sean Anderson <seanga2@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread
* [RFC PATCH 23/28] cli: lil: Handle OOM for hm_put
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (21 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 22/28] env: Add a priv pointer to hwalk_r Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 24/28] cli: lil: Make proc always take 3 arguments Sean Anderson
` (6 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
hm_put allocates memory, and this can fail. Instead of failing silently,
return an error code. This also fixes up callers to handle this error.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 47 ++++++++++++++++++++++++++++++++---------------
1 file changed, 32 insertions(+), 15 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 2ed96ebc2d..7ec73675f3 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -326,22 +326,30 @@ static void hm_destroy(struct hashmap *hm)
}
}
-static void hm_put(struct hashmap *hm, const char *key, void *value)
+static enum lil_error hm_put(struct hashmap *hm, const char *key, void *value)
{
struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+ struct hashentry *newe;
size_t i;
for (i = 0; i < cell->c; i++) {
if (!strcmp(key, cell->e[i].k)) {
cell->e[i].v = value;
- return;
+ return LIL_ERR_NONE;
}
}
- cell->e = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
- cell->e[cell->c].k = strdup(key);
- cell->e[cell->c].v = value;
+ newe = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
+ if (!newe)
+ return LIL_ERR_OOM;
+ cell->e = newe;
+
+ newe[cell->c].k = strdup(key);
+ if (!newe[cell->c].k)
+ return LIL_ERR_OOM;
+ newe[cell->c].v = value;
cell->c++;
+ return LIL_ERR_NONE;
}
static void *hm_get(struct hashmap *hm, const char *key)
@@ -738,19 +746,24 @@ static struct lil_func *add_func(struct lil *lil, const char *name)
cmd = calloc(1, sizeof(struct lil_func));
if (!cmd)
- return NULL;
+ goto oom;
cmd->name = strdup(name);
ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
- if (!ncmd) {
- free(cmd);
- return NULL;
- }
-
+ if (!ncmd)
+ goto oom;
lil->cmd = ncmd;
+
ncmd[lil->cmds++] = cmd;
- hm_put(&lil->cmdmap, name, cmd);
+ if (hm_put(&lil->cmdmap, name, cmd))
+ goto oom;
+
return cmd;
+
+oom:
+ free(cmd);
+ lil_set_error_oom(lil);
+ return NULL;
}
static void del_func(struct lil *lil, struct lil_func *cmd)
@@ -766,7 +779,11 @@ static void del_func(struct lil *lil, struct lil_func *cmd)
if (index == lil->cmds)
return;
- hm_put(&lil->cmdmap, cmd->name, 0);
+ /*
+ * The only way this fails is if we don't find the command; this
+ * means our caller wants to delete a command which doesn't exist
+ */
+ assert(!hm_put(&lil->cmdmap, cmd->name, NULL));
if (cmd->argnames)
lil_free_list(cmd->argnames);
@@ -783,9 +800,9 @@ int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)
struct lil_func *cmd = add_func(lil, name);
if (!cmd)
- return 0;
+ return LIL_ERR_OOM;
cmd->proc = proc;
- return 1;
+ return LIL_ERR_NONE;
}
struct lil_var *lil_set_var(struct lil *lil, const char *name,
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 24/28] cli: lil: Make proc always take 3 arguments
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (22 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 23/28] cli: lil: Handle OOM for hm_put Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 25/28] cli: lil: Always quote items in lil_list_to_value Sean Anderson
` (5 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This rewrites proc to always take 3 arguments. It also adds proper error
handling. TCL does not allow for anonymous functions to be created with
proc. Allowing for a variable number of arguments makes the code much more
complex when adding error handling.
Since fnc_proc was the last user of lil_unused_name (other than
fnc_unusedname), remove it.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 103 ++++++++++++++++-------------------------------
test/cmd/lil.c | 6 ++-
2 files changed, 38 insertions(+), 71 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 7ec73675f3..1c7c340bda 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -2985,32 +2985,6 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
return lil_alloc_integer(ee.ival);
}
-struct lil_value *lil_unused_name(struct lil *lil, const char *part)
-{
- char *name = malloc(strlen(part) + 64);
- struct lil_value *val;
- size_t i;
-
- for (i = 0; i < (size_t)-1; i++) {
- sprintf(name, "!!un!%s!%09u!nu!!", part, (unsigned int)i);
- if (lil_find_cmd(lil, name))
- continue;
-
- if (lil_find_var(lil, lil->env, name))
- continue;
-
- val = lil_alloc_string(name);
- free(name);
- return val;
- }
- return NULL;
-}
-
-struct lil_value *lil_arg(struct lil_value **argv, size_t index)
-{
- return argv ? argv[index] : NULL;
-}
-
const char *lil_to_string(struct lil_value *val)
{
return (val && val->l) ? val->d : "";
@@ -3237,47 +3211,46 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
struct lil_value **argv)
{
- struct lil_value *name;
struct lil_func *cmd;
- struct lil_list *fargs;
+ struct lil_list *args;
+ struct lil_value *name, *code;
- if (argc < 1)
+ if (argc != 3) {
+ lil_set_error_argc(lil, 3);
return NULL;
-
- if (argc >= 3) {
- name = lil_clone_value(argv[0]);
- fargs = lil_subst_to_list(lil, argv[1]);
- cmd = add_func(lil, lil_to_string(argv[0]));
- if (!cmd)
- return NULL;
-
- cmd->argnames = fargs;
- cmd->code = lil_clone_value(argv[2]);
- } else {
- name = lil_unused_name(lil, "anonymous-function");
- if (argc < 2) {
- struct lil_value *tmp = lil_alloc_string("args");
-
- fargs = lil_subst_to_list(lil, tmp);
- lil_free_value(tmp);
- cmd = add_func(lil, lil_to_string(name));
- if (!cmd)
- return NULL;
-
- cmd->argnames = fargs;
- cmd->code = lil_clone_value(argv[0]);
- } else {
- fargs = lil_subst_to_list(lil, argv[0]);
- cmd = add_func(lil, lil_to_string(name));
- if (!cmd)
- return NULL;
-
- cmd->argnames = fargs;
- cmd->code = lil_clone_value(argv[1]);
- }
}
+ name = lil_clone_value(argv[0]);
+ if (!name) {
+ lil_set_error_oom(lil);
+ return NULL;
+ }
+
+ args = lil_subst_to_list(lil, argv[1]);
+ if (!args)
+ goto err_args;
+
+ code = lil_clone_value(argv[2]);
+ if (!code) {
+ lil_set_error_oom(lil);
+ goto err_code;
+ }
+
+ cmd = add_func(lil, lil_to_string(name));
+ if (!cmd)
+ goto err_func;
+ cmd->argnames = args;
+ cmd->code = code;
+
return name;
+
+err_func:
+ lil_free_value(code);
+err_code:
+ lil_free_list(args);
+err_args:
+ lil_free_value(name);
+ return NULL;
}
static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
@@ -3312,13 +3285,6 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
return r;
}
-static struct lil_value *fnc_unusedname(struct lil *lil, size_t argc,
- struct lil_value **argv)
-{
- return lil_unused_name(lil, argc > 0 ? lil_to_string(argv[0]) :
- "unusedname");
-}
-
static struct lil_value *fnc_quote(struct lil *lil, size_t argc,
struct lil_value **argv)
{
@@ -4310,7 +4276,6 @@ static void register_stdcmds(struct lil *lil)
lil_register(lil, "substr", fnc_substr);
lil_register(lil, "topeval", fnc_topeval);
lil_register(lil, "trim", fnc_trim);
- lil_register(lil, "unusedname", fnc_unusedname);
lil_register(lil, "upeval", fnc_upeval);
}
}
diff --git a/test/cmd/lil.c b/test/cmd/lil.c
index fb33fa83a6..58bc6ee842 100644
--- a/test/cmd/lil.c
+++ b/test/cmd/lil.c
@@ -149,9 +149,11 @@ static const struct {
"[list eh??]"
"];"
"asserteq_list [lapply $list length] [list 9 10 4];"
- "asserteq_list [lapply $list [proc {a} {"
+ "proc firstword {a} {"
"return [index [split $a] 0]"
- "}]] [list {bad's} {good's} eh??]"
+ "};"
+ "asserteq_list [lapply $list firstword] "
+ "[list {bad's} {good's} eh??]"
},
{"lists",
"set l [list foo bar baz bad];"
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 25/28] cli: lil: Always quote items in lil_list_to_value
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (23 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 24/28] cli: lil: Make proc always take 3 arguments Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 26/28] cli: lil: Allocate len even when str is NULL in alloc_value_len Sean Anderson
` (4 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This function took an argument do_quote which determined whether or not to
quote an item. All callers set it to true, so remove it and always quote.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 34 ++++++++++++++--------------------
include/cli_lil.h | 2 +-
2 files changed, 15 insertions(+), 21 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 1c7c340bda..153c34791b 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -2092,7 +2092,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
if (cmd->argnames->c == 1 &&
!strcmp(lil_to_string(lil_list_get(cmd->argnames, 0)),
"args")) {
- struct lil_value *args = lil_list_to_value(words, 1);
+ struct lil_value *args = lil_list_to_value(words);
lil_set_var(lil, "args", args, LIL_SETVAR_LOCAL_NEW);
lil_free_value(args);
@@ -2389,7 +2389,7 @@ braces:
return needs;
}
-struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape)
+struct lil_value *lil_list_to_value(struct lil_list *list)
{
struct lil_value *val = alloc_value(NULL);
size_t i, j;
@@ -2397,17 +2397,11 @@ struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape)
for (i = 0; i < list->c; i++) {
char q;
struct lil_value *item = lil_list_get(list, i);
- enum needs needs;
-
- if (do_escape)
- needs = item_needs(lil_to_string(item), item->l);
- else
- needs = NEEDS_NOTHING;
if (i)
lil_append_char(val, ' ');
- switch (needs) {
+ switch (item_needs(lil_to_string(item), item->l)) {
case NEEDS_NOTHING:
if (lil_append_val(val, item))
goto err;
@@ -3087,7 +3081,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
func = lil_find_cmd(lil, lil_to_string(argv[1]));
if (!func || !func->argnames)
return NULL;
- return lil_list_to_value(func->argnames, 1);
+ return lil_list_to_value(func->argnames);
}
if (!strcmp(type, "body")) {
@@ -3111,7 +3105,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
lil_list_append(funcs,
lil_alloc_string(lil->cmd[i]->name));
- r = lil_list_to_value(funcs, 1);
+ r = lil_list_to_value(funcs);
lil_free_list(funcs);
return r;
}
@@ -3130,7 +3124,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
env = env->parent;
}
- r = lil_list_to_value(vars, 1);
+ r = lil_list_to_value(vars);
lil_free_list(vars);
return r;
}
@@ -3145,7 +3139,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
lil_list_append(vars, var);
}
- r = lil_list_to_value(vars, 1);
+ r = lil_list_to_value(vars);
lil_free_list(vars);
return r;
}
@@ -3513,7 +3507,7 @@ static struct lil_value *fnc_append(struct lil *lil, size_t argc,
for (i = base; i < argc; i++)
lil_list_append(list, lil_clone_value(argv[i]));
- r = lil_list_to_value(list, 1);
+ r = lil_list_to_value(list);
lil_free_list(list);
lil_set_var(lil, varname, r, access);
return r;
@@ -3548,7 +3542,7 @@ static struct lil_value *fnc_slice(struct lil *lil, size_t argc,
lil_list_append(slice, lil_clone_value(lil_list_get(list, i)));
lil_free_list(list);
- r = lil_list_to_value(slice, 1);
+ r = lil_list_to_value(slice);
lil_free_list(slice);
return r;
}
@@ -3586,7 +3580,7 @@ static struct lil_value *fnc_filter(struct lil *lil, size_t argc,
}
lil_free_list(list);
- r = lil_list_to_value(filtered, 1);
+ r = lil_list_to_value(filtered);
lil_free_list(filtered);
return r;
}
@@ -3601,7 +3595,7 @@ static struct lil_value *fnc_list(struct lil *lil, size_t argc,
for (i = 0; i < argc; i++)
lil_list_append(list, lil_clone_value(argv[i]));
- r = lil_list_to_value(list, 1);
+ r = lil_list_to_value(list);
lil_free_list(list);
return r;
}
@@ -3628,7 +3622,7 @@ static struct lil_value *fnc_concat(struct lil *lil, size_t argc,
r = lil_alloc_string("");
for (i = 0; i < argc; i++) {
list = lil_subst_to_list(lil, argv[i]);
- tmp = lil_list_to_value(list, 1);
+ tmp = lil_list_to_value(list);
lil_free_list(list);
lil_append_val(r, tmp);
lil_free_value(tmp);
@@ -3672,7 +3666,7 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
break;
}
- r = lil_list_to_value(rlist, 1);
+ r = lil_list_to_value(rlist);
lil_free_list(list);
lil_free_list(rlist);
return r;
@@ -4147,7 +4141,7 @@ static struct lil_value *fnc_split(struct lil *lil, size_t argc,
}
lil_list_append(list, val);
- val = lil_list_to_value(list, 1);
+ val = lil_list_to_value(list);
lil_free_list(list);
return val;
}
diff --git a/include/cli_lil.h b/include/cli_lil.h
index 290329372a..6fbc270f1b 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -161,7 +161,7 @@ void lil_free_list(struct lil_list *list);
int lil_list_append(struct lil_list *list, void *item);
size_t lil_list_size(struct lil_list *list);
struct lil_value *lil_list_get(struct lil_list *list, size_t index);
-struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape);
+struct lil_value *lil_list_to_value(struct lil_list *list);
struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code);
struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code);
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 26/28] cli: lil: Allocate len even when str is NULL in alloc_value_len
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (24 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 25/28] cli: lil: Always quote items in lil_list_to_value Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 27/28] cli: lil: Add a function to quote values Sean Anderson
` (3 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This allows us to reserve some space ahead of time, avoiding another
alloc/copy/free.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 153c34791b..42659920b5 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -459,21 +459,19 @@ static struct lil_value *alloc_value_len(const char *str, size_t len)
if (!val)
return NULL;
value_to_symbol(val)->type = LIL_SYMBOL_VALUE;
+ ensure_capacity(val, len + 1);
+ if (!val->d) {
+ release_to_pool(val);
+ return NULL;
+ }
if (str) {
val->l = len;
- ensure_capacity(val, len + 1);
- if (!val->d) {
- release_to_pool(val);
- return NULL;
- }
memcpy(val->d, str, len);
val->d[len] = 0;
} else {
val->l = 0;
- ensure_capacity(val, 1);
- if (val->d)
- val->d[0] = '\0';
+ val->d[0] = '\0';
}
return val;
}
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 27/28] cli: lil: Add a function to quote values
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (25 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 26/28] cli: lil: Allocate len even when str is NULL in alloc_value_len Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 6:16 ` [RFC PATCH 28/28] cli: lil: Load procs from the environment Sean Anderson
` (2 subsequent siblings)
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
This allows us to convert lil_values into a form which can be re-parsed. We
were already doing this for lists, so we just have to expose the inner
loop.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli_lil.c | 89 ++++++++++++++++++++++++++++--------------------
1 file changed, 52 insertions(+), 37 deletions(-)
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 42659920b5..2a8600ffb6 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -2387,49 +2387,64 @@ braces:
return needs;
}
+static enum lil_error do_quote(struct lil_value *val, struct lil_value *item)
+{
+ char q;
+ size_t i;
+
+ switch (item_needs(lil_to_string(item), item->l)) {
+ case NEEDS_NOTHING:
+ return lil_append_val(val, item);
+ case NEEDS_BRACES:
+ return lil_append_char(val, '{') ?:
+ lil_append_val(val, item) ?:
+ lil_append_char(val, '}');
+ case NEEDS_DOUBLE:
+ q = '"';
+ goto quote;
+ case NEEDS_SINGLE:
+ q = '\'';
+quote:
+ if (lil_append_char(val, q))
+ return LIL_ERR_OOM;
+
+ for (i = 0; i < item->l; i++) {
+ char c = item->d[i];
+
+ if (c == '\\' || c == q)
+ if (lil_append_char(val, '\\'))
+ return LIL_ERR_OOM;
+ if (lil_append_char(val, c))
+ return LIL_ERR_OOM;
+ }
+ if (lil_append_char(val, q))
+ return LIL_ERR_OOM;
+ }
+ return LIL_ERR_NONE;
+}
+
+static struct lil_value *lil_quote_value(struct lil_value *val)
+{
+ struct lil_value *r = alloc_value(NULL);
+
+ if (do_quote(r, val)) {
+ lil_free_value(r);
+ return NULL;
+ }
+ return r;
+}
+
struct lil_value *lil_list_to_value(struct lil_list *list)
{
struct lil_value *val = alloc_value(NULL);
- size_t i, j;
+ size_t i;
for (i = 0; i < list->c; i++) {
- char q;
- struct lil_value *item = lil_list_get(list, i);
+ if (i && lil_append_char(val, ' '))
+ goto err;
- if (i)
- lil_append_char(val, ' ');
-
- switch (item_needs(lil_to_string(item), item->l)) {
- case NEEDS_NOTHING:
- if (lil_append_val(val, item))
- goto err;
- continue;
- case NEEDS_BRACES:
- if (lil_append_char(val, '{') ||
- lil_append_val(val, item) ||
- lil_append_char(val, '}'))
- goto err;
- continue;
- case NEEDS_DOUBLE:
- q = '"';
- goto quote;
- case NEEDS_SINGLE:
- q = '\'';
-quote:
- if (lil_append_char(val, q))
- goto err;
- for (j = 0; j < item->l; j++) {
- char c = item->d[j];
-
- if (c == '\\' || c == q)
- if (lil_append_char(val, '\\'))
- goto err;
- if (lil_append_char(val, c))
- goto err;
- }
- if (lil_append_char(val, q))
- goto err;
- }
+ if (do_quote(val, lil_list_get(list, i)))
+ goto err;
}
return val;
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* [RFC PATCH 28/28] cli: lil: Load procs from the environment
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (26 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 27/28] cli: lil: Add a function to quote values Sean Anderson
@ 2021-07-01 6:16 ` Sean Anderson
2021-07-01 20:21 ` [RFC PATCH 00/28] cli: Add a new shell Tom Rini
2021-07-08 3:49 ` Heiko Schocher
29 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-01 6:16 UTC (permalink / raw)
To: u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos, Sean Anderson
When we start up the LIL interpreter, go through every variable and see if
it looks like a new procedure. If it does, try and parse it. For the return
trip, every time that we create a new procedure, create a new global
variable containing that procedure.
The end result of this is that procedures should now be saved to the
environment in the same way that variables are. So you can do
=> proc foo {args} { ... }
=> env save
and foo will be there after you reboot.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
---
common/cli.c | 54 ++++++++++++++++++++++++++++-------
common/cli_lil.c | 71 ++++++++++++++++++++++++++++++++++-------------
include/cli_lil.h | 17 ++++++++++++
3 files changed, 113 insertions(+), 29 deletions(-)
diff --git a/common/cli.c b/common/cli.c
index 391fee0ec7..c71f75e684 100644
--- a/common/cli.c
+++ b/common/cli.c
@@ -16,9 +16,11 @@
#include <command.h>
#include <console.h>
#include <env.h>
+#include <env_internal.h>
#include <fdtdec.h>
#include <hang.h>
#include <malloc.h>
+#include <search.h>
#include <asm/global_data.h>
DECLARE_GLOBAL_DATA_PTR;
@@ -26,6 +28,21 @@ DECLARE_GLOBAL_DATA_PTR;
#ifdef CONFIG_LIL
static struct lil *lil;
+static enum lil_error print_lil_err(struct lil *lil)
+{
+ enum lil_error ret;
+ const char *err_msg;
+
+ ret = lil_error(lil, &err_msg);
+ if (ret) {
+ if (err_msg)
+ printf("error: %s\n", err_msg);
+ else
+ printf("error: %d\n", ret);
+ }
+ return ret;
+}
+
static int env_setvar(struct lil *lil, const char *name,
struct lil_value **value)
{
@@ -41,16 +58,40 @@ static int env_getvar(struct lil *lil, const char *name,
return 1;
}
+static int env_register_proc(struct env_entry *entry, void *priv)
+{
+ struct lil *lil = priv;
+ struct lil_value *name;
+ const char *name_str, prefix[] = "proc";
+
+ /* Skip variables which are obviously not procedures */
+ if (strncmp(entry->data, prefix, sizeof(prefix) - 1))
+ return 0;
+
+ name = lil_parse_eval(lil, entry->data, 0, true);
+ name_str = lil_to_string(name);
+ if (strcmp(entry->key, name_str))
+ log_debug("proc %s created by variable %s\n",
+ name_str, entry->key);
+ lil_free_value(name);
+ return print_lil_err(lil);
+}
+
+static int env_initprocs(struct lil *lil)
+{
+ return hwalk_r(&env_htab, env_register_proc, lil);
+}
+
static const struct lil_callbacks env_callbacks = {
.setvar = env_setvar,
.getvar = env_getvar,
+ .initprocs = env_initprocs,
};
static int lil_run(const char *cmd)
{
- int err;
struct lil_value *result = lil_parse_eval(lil, cmd, 0, true);
- const char *err_msg, *strres = lil_to_string(result);
+ const char *strres = lil_to_string(result);
/* The result may be very big, so use puts */
if (strres && strres[0]) {
@@ -59,14 +100,7 @@ static int lil_run(const char *cmd)
}
lil_free_value(result);
- err = lil_error(lil, &err_msg);
- if (err) {
- if (err_msg)
- printf("error: %s\n", err_msg);
- else
- printf("error: %d\n", err);
- }
- return !!err;
+ return !!print_lil_err(lil);
}
#endif
diff --git a/common/cli_lil.c b/common/cli_lil.c
index 2a8600ffb6..7b4a56dbd0 100644
--- a/common/cli_lil.c
+++ b/common/cli_lil.c
@@ -3218,9 +3218,11 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
struct lil_value **argv)
{
+ static const char fmt[] = "proc %s {%s} %s";
+ size_t n;
struct lil_func *cmd;
- struct lil_list *args;
- struct lil_value *name, *code;
+ struct lil_list *fargs;
+ struct lil_value *name, *args, *code, *val;
if (argc != 3) {
lil_set_error_argc(lil, 3);
@@ -3228,34 +3230,62 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
}
name = lil_clone_value(argv[0]);
- if (!name) {
- lil_set_error_oom(lil);
- return NULL;
- }
-
- args = lil_subst_to_list(lil, argv[1]);
- if (!args)
- goto err_args;
-
code = lil_clone_value(argv[2]);
- if (!code) {
+ if (!name || !code) {
lil_set_error_oom(lil);
- goto err_code;
+ goto err_name_fargs_code;
}
+ fargs = lil_subst_to_list(lil, argv[1]);
+ if (!fargs)
+ goto err_name_fargs_code;
+
cmd = add_func(lil, lil_to_string(name));
if (!cmd)
goto err_func;
- cmd->argnames = args;
+ cmd->argnames = fargs;
cmd->code = code;
+ args = lil_list_to_value(fargs);
+ code = lil_quote_value(code);
+ if (!args || !code) {
+ lil_set_error_oom(lil);
+ goto err_quote_val;
+ }
+
+ n = snprintf(NULL, 0, fmt, lil_to_string(name), lil_to_string(args),
+ lil_to_string(code));
+ val = alloc_value_len(NULL, n);
+ if (!val) {
+ lil_set_error_oom(lil);
+ goto err_quote_val;
+ }
+
+ snprintf(val->d, n + 1, fmt, lil_to_string(name), lil_to_string(args),
+ lil_to_string(code));
+ val->l = n;
+ if (!lil_set_var(lil, lil_to_string(name), val, LIL_SETVAR_GLOBAL))
+ goto err_set;
+
+ lil_free_value(val);
+ lil_free_value(code);
+ lil_free_value(args);
+
return name;
-err_func:
+err_set:
+ lil_free_value(val);
+err_quote_val:
+ del_func(lil, cmd);
+ lil_free_value(code);
+ lil_free_value(args);
+ lil_free_value(name);
+ return NULL;
+
+err_func:
+ lil_free_list(fargs);
+err_name_fargs_code:
lil_free_value(code);
-err_code:
- lil_free_list(args);
-err_args:
lil_free_value(name);
return NULL;
}
@@ -4235,12 +4265,15 @@ static void register_stdcmds(struct lil *lil)
struct cmd_tbl *cmdtp, *start = ll_entry_start(struct cmd_tbl, cmd);
const int len = ll_entry_count(struct cmd_tbl, cmd);
+ lil_register(lil, "proc", fnc_proc);
+ if (lil->callbacks.initprocs)
+ lil->callbacks.initprocs(lil);
+
lil_register(lil, "decr", fnc_decr);
lil_register(lil, "eval", fnc_eval);
lil_register(lil, "expr", fnc_expr);
lil_register(lil, "for", fnc_for);
lil_register(lil, "foreach", fnc_foreach);
- lil_register(lil, "proc", fnc_proc);
lil_register(lil, "if", fnc_if);
lil_register(lil, "incr", fnc_incr);
lil_register(lil, "local", fnc_local);
diff --git a/include/cli_lil.h b/include/cli_lil.h
index 6fbc270f1b..47a2eeefc9 100644
--- a/include/cli_lil.h
+++ b/include/cli_lil.h
@@ -94,6 +94,23 @@ struct lil_callbacks {
*/
int (*getvar)(struct lil *lil, const char *name,
struct lil_value **value);
+
+ /**
+ * @initprocs: Called when procedures should be created
+ *
+ * @lil: The LIL interpreter
+ *
+ * This is called once in lil_new() when user functions should be added.
+ * When this is called, the only function registered is "proc". You can
+ * register functions by calling lil_parse_eval() with code which
+ * initializes a procedure.
+ *
+ * In practice, this is expected to be used to initialize procedures
+ * from environmental variables.
+ *
+ * @Return: 0 if ok, or non-zero on error.
+ */
+ int (*initprocs)(struct lil *lil);
};
/**
--
2.32.0
^ permalink raw reply related [flat|nested] 107+ messages in thread* Re: [RFC PATCH 00/28] cli: Add a new shell
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (27 preceding siblings ...)
2021-07-01 6:16 ` [RFC PATCH 28/28] cli: lil: Load procs from the environment Sean Anderson
@ 2021-07-01 20:21 ` Tom Rini
2021-07-02 11:30 ` Wolfgang Denk
2021-07-02 14:07 ` Sean Anderson
2021-07-08 3:49 ` Heiko Schocher
29 siblings, 2 replies; 107+ messages in thread
From: Tom Rini @ 2021-07-01 20:21 UTC (permalink / raw)
To: Sean Anderson
Cc: u-boot, Marek Behún, Wolfgang Denk, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
[-- Attachment #1: Type: text/plain, Size: 13845 bytes --]
On Thu, Jul 01, 2021 at 02:15:43AM -0400, Sean Anderson wrote:
> Well, this has been sitting on my hard drive for too long without feedback
> ("Release early, release often"), so here's the first RFC. This is not ready to
> merge (see the "Future work" section below), but the shell is functional and at
> least partially tested.
>
> The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes
> over on sandbox.
A good goal, but perhaps slightly too strict?
>
> add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
>
> = Getting started
>
> Enable CONFIG_LIL. If you would like to run tests, enable CONFIG_LIL_FULL. Note
> that dm_test_acpi_cmd_dump and setexpr_test_str_oper will fail. CONFIG_LIL_POOLS
> is currently broken (with what appears to be a double free).
>
> For an overview of the language as a whole, refer to the original readme [1].
>
> [1] http://runtimeterror.com/tech/lil/readme.txt
>
> == Key patches
>
> The following patches are particularly significant for reviewing and
> understanding this series:
>
> cli: Add LIL shell
> This contains the LIL shell as originally written by Kostas with some
> major deletions and some minor additions.
> cli: lil: Wire up LIL to the rest of U-Boot
> This allows you to use LIL as a shell just like Hush.
> cli: lil: Document structures
> This adds documentation for the major structures of LIL. It is a good
> place to start looking at the internals.
> test: Add tests for LIL
> This adds some basic integration tests and provides some examples of
> LIL code.
> cli: lil: Add a distinct parsing step
> This adds a parser separate from the interpreter. This patch is the
> largest original work in this series.
> cli: lil: Load procs from the environment
> This allows procedures to be saved and loaded like variables.
>
> = A new shell
>
> This series adds a new shell for U-Boot. The aim is to eventually replace Hush
> as the primary shell for all boards which currently use it. Hush should be
> replaced because it has several major problems:
>
> - It has not had a major update in two decades, resulting in duplication of
> effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks
>
> So the specific problem has (long) been fixed in upstream, and
> instead of adding a patch to our old version, thus cementing the
> broken behaviour, we should upgrade hush to recent upstream code.
>
> -- Wolfgang Denk [2]
>
> These lack of updates are further compounded by a significant amount of
> ifdef-ing in the Hush code. This makes the shell hard to read and debug.
> Further, the original purpose of such ifdef-ing (upgrading to a newer Hush)
> has never happened.
>
> - It was designed for a preempting OS which supports pipes and processes. This
> fundamentally does not match the computing model of U-Boot where there is
> exactly one thread (and every other CPU is spinning or sleeping). Working
> around these design differences is a significant cause of the aformentioned
> ifdef-ing.
>
> - It lacks many major features expected of even the most basic shells, such
> as functions and command substitution ($() syntax). This makes it difficult
> to script with Hush. While it is desirable to write some code in C, much code
> *must* be written in C because there is no way to express the logic in Hush.
>
> I believe that U-Boot should have a shell which is more featureful, has cleaner
> code, and which is the same size as Hush (or less). The ergonomic advantages
> afforded by a new shell will make U-Boot easier to use and customize.
>
> [2] https://lore.kernel.org/u-boot/872080.1614764732@gemini.denx.de/
First, great! Thanks for doing this. A new shell really is the only
viable path forward here, and I appreciate you taking the time to
evaluate several and implement one.
> = Open questions
>
> While the primary purpose of this series is of course to get feedback on the
> code I have already written, there are several decisions where I am not sure
> what the best course of action is.
>
> - What should be done about 'expr'? The 'expr' command is a significant portion
> of the final code size. It cannot be removed outright, because it is used by
> several builtin functions like 'if', 'while', 'for', etc. The way I see it,
> there are two general approaches to take
>
> - Rewrite expr to parse expressions and then evaluate them. The parsing could
> re-use several of the existing parse functions like how parse_list does.
> This could reduce code, as instead of many functions each with their own
> while/switch statements, we could have two while/switch statements (one to
> parse, and one to evaluate). However, this may end up increasing code size
> (such as when the main language had evaluation split from parsing).
>
> - Don't parse infix expressions, and just make arithmetic operators normal
> functions. This would affect ergonomics a bit. For example, instead of
>
> if {$i < 10} { ... }
>
> one would need to write
>
> if {< $i 10} { ... }
>
> and instead of
>
> if {$some_bool} { ... }
>
> one would need to write
>
> if {quote $some_bool} { ... }
>
> Though, given how much setexpr is used (not much), this may not be such a
> big price to pay. This route is almost certain to reduce code size.
So, this is a question because we have cmd/setexpr.c that provides
"expr" today? Or because this is a likely place to reclaim some of that
800 byte growth?
> - How should LIL functions integrate with the rest of U-Boot? At the moment, lil
> functions and procedures exist in a completely separate world from normal
> commands. I would like to integrate them more closely, but I am not sure the
> best way to go about this. At the very minimum, each LIL builtin function
> needs to get its hands on the LIL interpreter somehow. I'd rather this didn't
> happen through gd_t or similar so that it is easier to unit test.
> Additionally, LIL functions expect an array of lil_values instead of strings.
> We could strip them out, but I worry that might start to impact performance
> (from all the copying).
I might be missing something here. But, given that whenever we have C
code run-around and generate a string to then pass to the interpreter to
run, someone asks why we don't just make API calls directly, perhaps the
answer is that we don't need to?
>
> The other half of this is adding LIL features into regular commands. The most
> important feature here is being able to return a string result. I took an
> initial crack at it [3], but I think with this series there is a stronger
> motivating factor (along with things like [4]).
>
> [3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
> [4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013
>
> = Future work
>
> The series as presented today is incomplete. The following are the major issues
> I see with it at the moment. I would like to address all of these issues, but
> some of them might be postponed until after first merging this series.
>
> - There is a serious error handling problem. Most original LIL code never
> checked errors. In almost every case, errors were silently ignored, even
> malloc failures! While I have designed new code to handle errors properly,
> there still remains a significant amount of original code which just ignores
> errors. In particular, I would like to ensure that the following categories of
> error conditions are handled:
>
> - Running out of memory.
> - Access to a nonexistant variable.
> - Passing the wrong number of arguments to a function.
> - Interpreting a value as the wrong type (e.g. "foo" should not have a numeric
> representation, instead of just being treated as 1).
>
> - There are many deviations from TCL with no purpose. For example, the list
> indexing function is named "index" and not "lindex". It is perfectly fine to
> drop features or change semantics to reduce code size, make parsing easier,
> or make execution easier. But changing things for the sake of it should be
> avoided.
>
> - The test suite is rather anemic compared with the amount of code this
> series introduces. I would like to expand it significantly. In particular,
> error conditions are not well tested (only the "happy path" is tested).
>
> - While I have documented all new functions I have written, there are many
> existing functions which remain to be documented. In addition, there is no
> user documentation, which is critical in driving adoption of any new
> programming language. Some of this cover letter might be integrated with any
> documentation written.
>
> - Some shell features such as command repetition and secondary shell prompts
> have not been implemented.
>
> - Arguments to native lil functions are incompatible with U-Boot functions. For
> example, the command
>
> foo bar baz
>
> would be passed to a U-Boot command as
>
> { "foo", "bar", "baz", NULL }
>
> but would be passed to a LIL function as
>
> { "bar", "baz" }
>
> This makes it more difficult to use the same function to parse several
> different commands. At the moment this is solved by passing the command name
> in lil->env->proc, but I would like to switch to the U-Boot argument list
> style.
>
> - Several existing tests break when using LIL because they expect no output on
> failure, but LIL produces some output notifying the user of the failure.
>
> - Implement DISTRO_BOOT in LIL. I think this is an important proof-of-concept to
> show what can be done with LIL, and to determine which features should be
> moved to LIL_FULL.
>
> = Why Lil?
>
> When looking for a suitable replacement shell, I evaluated implementations using
> the following criteria:
>
> - It must have a GPLv2-compatible license.
> - It must be written in C, and have no major external dependencies.
> - It must support bare function calls. That is, a script such as 'foo bar'
> should invoke the function 'foo' with the argument 'bar'. This preserves the
> shell-like syntax we expect.
> - It must be small. The eventual target is that it compiles to around 10KiB with
> -Os and -ffunction-sections.
> - There should be good tests. Any tests at all are good, but a functioning suite
> is better.
> - There should be good documentation
> - There should be comments in the source.
> - It should be "finished" or have only slow development. This will hopefully
> make it easier to port changes.
On this last point, I believe this is based on lil20190821 and current
is now lil20210502. With a quick diff between them, I can see that the
changes there are small enough that while you've introduced a number of
changes here, it would be a very easy update.
> Notably absent from the above list is performance. Most scripts in U-Boot will
> be run once on boot. As long as the time spent evaluating scripts is kept under
> a reasonable threshold (a fraction of the time spend initializing hardware or
> reading data from persistant storage), there is no need to optimize for speed.
>
> In addition, I did not consider updating Hush from Busybox. The mismatch in
> computing environment expectations (as noted in the "New shell" section above)
> still applies. IMO, this mismatch is the biggest reason that things like
> functions and command substitution have been excluded from the U-Boot's Hush.
>
> == lil
>
> - zLib
> - TCL
> - Compiles to around 10k with no builtins. To 25k with builtins.
> - Some tests, but not organized into a suite with expected output. Some evidence
> that the author ran APL, but no harness.
> - Some architectural documentation. Some for each functions, but not much.
> - No comments :l
> - 3.5k LoC
>
> == picol
>
> - 2-clause BSD
> - TCL
> - Compiles to around 25k with no builtins. To 80k with builtins.
> - Tests with suite (in-language). No evidence of fuzzing.
> - No documentation :l
> - No comments :l
> - 5k LoC
>
> == jimtcl
>
> - 2-clause BSD
> - TCL
> - Compiles to around 95k with no builtins. To 140k with builtins. Too big...
>
> == boron
>
> - LGPLv3+ (so this is right out)
> - REBOL
> - Compiles to around 125k with no builtins. To 190k with builtins. Too big...
>
> == libmawk
>
> - GPLv2
> - Awk
> - Compiles to around 225k. Too big...
>
> == libfawk
>
> - 3-clause BSD
> - Uses bison+yacc...
> - Awk; As it turns out, this has parentheses for function calls.
> - Compiles to around 24-30k. Not sure how to remove builtins.
> - Test suite (in-language). No fuzzing.
> - Tutorial book. No function reference.
> - No comments
> - Around 2-4k LoC
>
> == MicroPython
>
> - MIT
> - Python (but included for completeness)
> - Compiles to around 300k. Too big...
>
> == mruby/c
>
> - 3-clause BSD
> - Ruby
> - Compiles to around 85k without builtins and 120k with. Too big...
>
> == eLua
>
> - MIT
> - Lua
> - Build system is a royal pain (custom and written in Lua with external deps)
> - Base binary is around 250KiB and I don't want to deal with reducing it
>
> So the interesting/viable ones are
> - lil
> - picol
> - libfawk (maybe)
>
> I started with LIL because it was the smallest. I have found several
> issues with LIL along the way. Some of these are addressed in this series
> already, while others remain unaddressed (see the section "Future Work").
Thanks for the evaluations, of these, lil does make the most sense.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 00/28] cli: Add a new shell
2021-07-01 20:21 ` [RFC PATCH 00/28] cli: Add a new shell Tom Rini
@ 2021-07-02 11:30 ` Wolfgang Denk
2021-07-02 13:56 ` Sean Anderson
2021-07-02 14:07 ` Sean Anderson
1 sibling, 1 reply; 107+ messages in thread
From: Wolfgang Denk @ 2021-07-02 11:30 UTC (permalink / raw)
To: Tom Rini
Cc: Sean Anderson, u-boot, Marek Behún, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
Dear Tom,
In message <20210701202155.GQ9516@bill-the-cat> you wrote:
>
> First, great! Thanks for doing this. A new shell really is the only
> viable path forward here, and I appreciate you taking the time to
> evaluate several and implement one.
I disagree that a new shell is the _only_ way forward.
AFAICT, all the raised concerns have long been fixed in upstream
versions of hush; see for example [1]:
...
//config: hush is a small shell. It handles the normal flow control
//config: constructs such as if/then/elif/else/fi, for/in/do/done, while loops,
//config: case/esac. Redirections, here documents, $((arithmetic))
//config: and functions are supported.
//config:
[1] https://git.busybox.net/busybox/tree/shell/hush.c#n98
My gut feeling is that updating to a recent version of hush is the
most efficent _backward_compatible_ way.
And if we drop that requirement, we might even take a bigger step
and move to lua - which would allow for a complete new level of
script based extensions.
> > - There is a serious error handling problem. Most original LIL code never
> > checked errors. In almost every case, errors were silently ignored, even
> > malloc failures! While I have designed new code to handle errors properly,
> > there still remains a significant amount of original code which just ignores
> > errors. In particular, I would like to ensure that the following categories of
> > error conditions are handled:
This is something that scares me like hell. This in a shell? For
me this is close to a killing point.
> > - Running out of memory.
> > - Access to a nonexistant variable.
> > - Passing the wrong number of arguments to a function.
> > - Interpreting a value as the wrong type (e.g. "foo" should not have a numeric
> > representation, instead of just being treated as 1).
Who says so?
Bash says:
-> printf "%d\n" foo
-bash: printf: foo: invalid number
0
So it is _not_ 1 ...
> > - There are many deviations from TCL with no purpose. For example, the list
> > indexing function is named "index" and not "lindex". It is perfectly fine to
> > drop features or change semantics to reduce code size, make parsing easier,
> > or make execution easier. But changing things for the sake of it should be
> > avoided.
It's not a standard POSIX shell, it's not TCL (ick!), ... it's
something new, incompatible...
> Thanks for the evaluations, of these, lil does make the most sense.
You mean, adding a complete new, incompatible and non-standard shell
is a better approach than updating to a recent version of hush?
What makes you think so?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Ernest asks Frank how long he has been working for the company.
"Ever since they threatened to fire me."
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 00/28] cli: Add a new shell
2021-07-02 11:30 ` Wolfgang Denk
@ 2021-07-02 13:56 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-02 13:56 UTC (permalink / raw)
To: Wolfgang Denk, Tom Rini
Cc: u-boot, Marek Behún, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/2/21 7:30 AM, Wolfgang Denk wrote:
> Dear Tom,
>
> In message <20210701202155.GQ9516@bill-the-cat> you wrote:
>>
>> First, great! Thanks for doing this. A new shell really is the only
>> viable path forward here, and I appreciate you taking the time to
>> evaluate several and implement one.
>
> I disagree that a new shell is the _only_ way forward.
>
> AFAICT, all the raised concerns have long been fixed in upstream
> versions of hush; see for example [1]:
>
> ...
> //config: hush is a small shell. It handles the normal flow control
> //config: constructs such as if/then/elif/else/fi, for/in/do/done, while loops,
> //config: case/esac. Redirections, here documents, $((arithmetic))
> //config: and functions are supported.
> //config:
>
> [1] https://git.busybox.net/busybox/tree/shell/hush.c#n98
In fact, the code for most of this is present but ifdef'd out. The real
issue is that the implementation of much of the above relies on things
like fork() which we can't provide.
> My gut feeling is that updating to a recent version of hush is the
> most efficent _backward_compatible_ way.
>
> And if we drop that requirement, we might even take a bigger step
> and move to lua - which would allow for a complete new level of
> script based extensions.
I'm not aware of any Lua implementations which meet the size
requirements for U-Boot. Any port would need to be either written from
scratch or have some serious modification, not unlike what I've done
here.
>>> - There is a serious error handling problem. Most original LIL code never
>>> checked errors. In almost every case, errors were silently ignored, even
>>> malloc failures! While I have designed new code to handle errors properly,
>>> there still remains a significant amount of original code which just ignores
>>> errors. In particular, I would like to ensure that the following categories of
>>> error conditions are handled:
>
> This is something that scares me like hell. This in a shell? For
> me this is close to a killing point.
Yes, it was for me as well. But it is not something so obvious unless
you are looking for it. I believe I have addressed this issue in much of
the core code (parser and interpreter). But the builtin commands must be
gone through and converted.
>>> - Running out of memory.
>>> - Access to a nonexistant variable.
>>> - Passing the wrong number of arguments to a function.
>>> - Interpreting a value as the wrong type (e.g. "foo" should not have a numeric
>>> representation, instead of just being treated as 1).
>
> Who says so?
The current LIL code.
>
> Bash says:
>
> -> printf "%d\n" foo
> -bash: printf: foo: invalid number
> 0
>
> So it is _not_ 1 ...
And I would like to replicate this behavior.
>
>>> - There are many deviations from TCL with no purpose. For example, the list
>>> indexing function is named "index" and not "lindex". It is perfectly fine to
>>> drop features or change semantics to reduce code size, make parsing easier,
>>> or make execution easier. But changing things for the sake of it should be
>>> avoided.
>
> It's not a standard POSIX shell, it's not TCL (ick!), ... it's
> something new, incompatible...
And Hush isn't POSIX either :)
But as noted above, I would like to hew much closer to TCL than LIL has
traditionally done.
>> Thanks for the evaluations, of these, lil does make the most sense.
>
> You mean, adding a complete new, incompatible and non-standard shell
> is a better approach than updating to a recent version of hush?
Correct.
> What makes you think so?
Hush relies heavily on its posix environment. Porting it has, and will,
require substantial modification. IMO the work necessary will be around
the same or more for Hush as for any other language.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 00/28] cli: Add a new shell
2021-07-01 20:21 ` [RFC PATCH 00/28] cli: Add a new shell Tom Rini
2021-07-02 11:30 ` Wolfgang Denk
@ 2021-07-02 14:07 ` Sean Anderson
1 sibling, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-02 14:07 UTC (permalink / raw)
To: Tom Rini
Cc: u-boot, Marek Behún, Wolfgang Denk, Simon Glass,
Roland Gaudig, Heinrich Schuchardt, Kostas Michalopoulos
On 7/1/21 4:21 PM, Tom Rini wrote:
> On Thu, Jul 01, 2021 at 02:15:43AM -0400, Sean Anderson wrote:
>
>> Well, this has been sitting on my hard drive for too long without feedback
>> ("Release early, release often"), so here's the first RFC. This is not ready to
>> merge (see the "Future work" section below), but the shell is functional and at
>> least partially tested.
>>
>> The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes
>> over on sandbox.
>
> A good goal, but perhaps slightly too strict?
Perhaps. But I think getting in the ballpark will significantly help
drive adoption. I want to make it as easy as possible for maintainers to
enable LIL and start using it.
>
>>
>> add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
>>
>> = Getting started
>>
>> Enable CONFIG_LIL. If you would like to run tests, enable CONFIG_LIL_FULL. Note
>> that dm_test_acpi_cmd_dump and setexpr_test_str_oper will fail. CONFIG_LIL_POOLS
>> is currently broken (with what appears to be a double free).
>>
>> For an overview of the language as a whole, refer to the original readme [1].
>>
>> [1] http://runtimeterror.com/tech/lil/readme.txt
>>
>> == Key patches
>>
>> The following patches are particularly significant for reviewing and
>> understanding this series:
>>
>> cli: Add LIL shell
>> This contains the LIL shell as originally written by Kostas with some
>> major deletions and some minor additions.
>> cli: lil: Wire up LIL to the rest of U-Boot
>> This allows you to use LIL as a shell just like Hush.
>> cli: lil: Document structures
>> This adds documentation for the major structures of LIL. It is a good
>> place to start looking at the internals.
>> test: Add tests for LIL
>> This adds some basic integration tests and provides some examples of
>> LIL code.
>> cli: lil: Add a distinct parsing step
>> This adds a parser separate from the interpreter. This patch is the
>> largest original work in this series.
>> cli: lil: Load procs from the environment
>> This allows procedures to be saved and loaded like variables.
>>
>> = A new shell
>>
>> This series adds a new shell for U-Boot. The aim is to eventually replace Hush
>> as the primary shell for all boards which currently use it. Hush should be
>> replaced because it has several major problems:
>>
>> - It has not had a major update in two decades, resulting in duplication of
>> effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks
>>
>> So the specific problem has (long) been fixed in upstream, and
>> instead of adding a patch to our old version, thus cementing the
>> broken behaviour, we should upgrade hush to recent upstream code.
>>
>> -- Wolfgang Denk [2]
>>
>> These lack of updates are further compounded by a significant amount of
>> ifdef-ing in the Hush code. This makes the shell hard to read and debug.
>> Further, the original purpose of such ifdef-ing (upgrading to a newer Hush)
>> has never happened.
>>
>> - It was designed for a preempting OS which supports pipes and processes. This
>> fundamentally does not match the computing model of U-Boot where there is
>> exactly one thread (and every other CPU is spinning or sleeping). Working
>> around these design differences is a significant cause of the aformentioned
>> ifdef-ing.
>>
>> - It lacks many major features expected of even the most basic shells, such
>> as functions and command substitution ($() syntax). This makes it difficult
>> to script with Hush. While it is desirable to write some code in C, much code
>> *must* be written in C because there is no way to express the logic in Hush.
>>
>> I believe that U-Boot should have a shell which is more featureful, has cleaner
>> code, and which is the same size as Hush (or less). The ergonomic advantages
>> afforded by a new shell will make U-Boot easier to use and customize.
>>
>> [2] https://lore.kernel.org/u-boot/872080.1614764732@gemini.denx.de/
>
> First, great! Thanks for doing this. A new shell really is the only
> viable path forward here, and I appreciate you taking the time to
> evaluate several and implement one.
>
>> = Open questions
>>
>> While the primary purpose of this series is of course to get feedback on the
>> code I have already written, there are several decisions where I am not sure
>> what the best course of action is.
>>
>> - What should be done about 'expr'? The 'expr' command is a significant portion
>> of the final code size. It cannot be removed outright, because it is used by
>> several builtin functions like 'if', 'while', 'for', etc. The way I see it,
>> there are two general approaches to take
>>
>> - Rewrite expr to parse expressions and then evaluate them. The parsing could
>> re-use several of the existing parse functions like how parse_list does.
>> This could reduce code, as instead of many functions each with their own
>> while/switch statements, we could have two while/switch statements (one to
>> parse, and one to evaluate). However, this may end up increasing code size
>> (such as when the main language had evaluation split from parsing).
>>
>> - Don't parse infix expressions, and just make arithmetic operators normal
>> functions. This would affect ergonomics a bit. For example, instead of
>>
>> if {$i < 10} { ... }
>>
>> one would need to write
>>
>> if {< $i 10} { ... }
>>
>> and instead of
>>
>> if {$some_bool} { ... }
>>
>> one would need to write
>>
>> if {quote $some_bool} { ... }
>>
>> Though, given how much setexpr is used (not much), this may not be such a
>> big price to pay. This route is almost certain to reduce code size.
>
> So, this is a question because we have cmd/setexpr.c that provides
> "expr" today? Or because this is a likely place to reclaim some of that
> 800 byte growth?
The latter. setexpr cannot be used because it does not return a result,
and instead sets a (global) variable. The expression parsing
functionality is core to LIL and used in many builtin commands (such as
`if` above), and really needs to return a lil_value.
>
>> - How should LIL functions integrate with the rest of U-Boot? At the moment, lil
>> functions and procedures exist in a completely separate world from normal
>> commands. I would like to integrate them more closely, but I am not sure the
>> best way to go about this. At the very minimum, each LIL builtin function
>> needs to get its hands on the LIL interpreter somehow. I'd rather this didn't
>> happen through gd_t or similar so that it is easier to unit test.
>> Additionally, LIL functions expect an array of lil_values instead of strings.
>> We could strip them out, but I worry that might start to impact performance
>> (from all the copying).
>
> I might be missing something here. But, given that whenever we have C
> code run-around and generate a string to then pass to the interpreter to
> run, someone asks why we don't just make API calls directly, perhaps the
> answer is that we don't need to?
err, the issue here is that the signature for regular commands is rougly
int cmd(..., int argc, char **argv, ...)
and the signature for LIL commands is
struct lil_value *cmd(struct lil *lil, size_t argc, struct lil_value **argv)
where lil_value is
struct lil_value {
size_t l;
char *d;
};
so while regular commands can be reimplemented as LIL commands (just
create a new argv containing the strings directly), it is more difficult
to go the other way. I bring this up because I think having two separate
ways to write a command is not the best way to do things going forward.
>>
>> The other half of this is adding LIL features into regular commands. The most
>> important feature here is being able to return a string result. I took an
>> initial crack at it [3], but I think with this series there is a stronger
>> motivating factor (along with things like [4]).
>>
>> [3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
>> [4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013
>>
>> = Future work
>>
>> The series as presented today is incomplete. The following are the major issues
>> I see with it at the moment. I would like to address all of these issues, but
>> some of them might be postponed until after first merging this series.
>>
>> - There is a serious error handling problem. Most original LIL code never
>> checked errors. In almost every case, errors were silently ignored, even
>> malloc failures! While I have designed new code to handle errors properly,
>> there still remains a significant amount of original code which just ignores
>> errors. In particular, I would like to ensure that the following categories of
>> error conditions are handled:
>>
>> - Running out of memory.
>> - Access to a nonexistant variable.
>> - Passing the wrong number of arguments to a function.
>> - Interpreting a value as the wrong type (e.g. "foo" should not have a numeric
>> representation, instead of just being treated as 1).
>>
>> - There are many deviations from TCL with no purpose. For example, the list
>> indexing function is named "index" and not "lindex". It is perfectly fine to
>> drop features or change semantics to reduce code size, make parsing easier,
>> or make execution easier. But changing things for the sake of it should be
>> avoided.
>>
>> - The test suite is rather anemic compared with the amount of code this
>> series introduces. I would like to expand it significantly. In particular,
>> error conditions are not well tested (only the "happy path" is tested).
>>
>> - While I have documented all new functions I have written, there are many
>> existing functions which remain to be documented. In addition, there is no
>> user documentation, which is critical in driving adoption of any new
>> programming language. Some of this cover letter might be integrated with any
>> documentation written.
>>
>> - Some shell features such as command repetition and secondary shell prompts
>> have not been implemented.
>>
>> - Arguments to native lil functions are incompatible with U-Boot functions. For
>> example, the command
>>
>> foo bar baz
>>
>> would be passed to a U-Boot command as
>>
>> { "foo", "bar", "baz", NULL }
>>
>> but would be passed to a LIL function as
>>
>> { "bar", "baz" }
>>
>> This makes it more difficult to use the same function to parse several
>> different commands. At the moment this is solved by passing the command name
>> in lil->env->proc, but I would like to switch to the U-Boot argument list
>> style.
>>
>> - Several existing tests break when using LIL because they expect no output on
>> failure, but LIL produces some output notifying the user of the failure.
>>
>> - Implement DISTRO_BOOT in LIL. I think this is an important proof-of-concept to
>> show what can be done with LIL, and to determine which features should be
>> moved to LIL_FULL.
>>
>> = Why Lil?
>>
>> When looking for a suitable replacement shell, I evaluated implementations using
>> the following criteria:
>>
>> - It must have a GPLv2-compatible license.
>> - It must be written in C, and have no major external dependencies.
>> - It must support bare function calls. That is, a script such as 'foo bar'
>> should invoke the function 'foo' with the argument 'bar'. This preserves the
>> shell-like syntax we expect.
>> - It must be small. The eventual target is that it compiles to around 10KiB with
>> -Os and -ffunction-sections.
>> - There should be good tests. Any tests at all are good, but a functioning suite
>> is better.
>> - There should be good documentation
>> - There should be comments in the source.
>> - It should be "finished" or have only slow development. This will hopefully
>> make it easier to port changes.
>
> On this last point, I believe this is based on lil20190821 and current
> is now lil20210502. With a quick diff between them, I can see that the
> changes there are small enough that while you've introduced a number of
> changes here, it would be a very easy update.
From what I understand, the only changes are updated copyrights and the
addition of a license file to cover the tests.
--Sean
^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [RFC PATCH 00/28] cli: Add a new shell
2021-07-01 6:15 [RFC PATCH 00/28] cli: Add a new shell Sean Anderson
` (28 preceding siblings ...)
2021-07-01 20:21 ` [RFC PATCH 00/28] cli: Add a new shell Tom Rini
@ 2021-07-08 3:49 ` Heiko Schocher
2021-07-08 4:26 ` Sean Anderson
29 siblings, 1 reply; 107+ messages in thread
From: Heiko Schocher @ 2021-07-08 3:49 UTC (permalink / raw)
To: Sean Anderson, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
Hello Sean,
On 01.07.21 08:15, Sean Anderson wrote:
> Well, this has been sitting on my hard drive for too long without feedback
> ("Release early, release often"), so here's the first RFC. This is not ready to
> merge (see the "Future work" section below), but the shell is functional and at
> least partially tested.
>
> The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes
> over on sandbox.
>
> add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
Thanks for this comparison. Here just my thoughts, but I have currently to
low time to follow here completly, so sorry if I miss stuff already
discussed.
From my side of view we can add a second shell beside hush, and as you
already mentioned it may is even possible to start LIL from hush. This
would be cool.
But I disagree in the following points:
- I do not see why LIL should *replace* hush. This may break a lot of
current boards. Also I see no need to do this, as if you do complex
stuff in u-boot shell it may is the wrong way and you have to think
about... and I see no big discussions here about BUGS in our current
hush implementation ... may it makes sense to bring them up and fix?
- Is LIL really so good code, that it is done? Or is there only no community
which develop further?
- Do we really want code, which does no error checking? Commentless code...
you wrote:
> - There is a serious error handling problem. Most original LIL code never
> checked errors. In almost every case, errors were silently ignored, even
> malloc failures! While I have designed new code to handle errors properly,
Which really let me think, nobody is interested in this code ... this
sounds like crap... and this brings me to the next point
- Do we not have the same problem as with hush, if we change LIL code?
Do you plan to upstream your changes?
So please do not understand me wrong, I see you invested much time here,
but I see LIL not in the short term as a replacement for hush.
I would accept to add LIL as an option and we will see, how good it works.
And if we have in lets say 2-3 years more boards which use LIL instead of
hush *and* also active maintained LIL code, we may can think of getting rid
of hush... and may have a "hush compatibility" layer...
I have no idea of current hush mainline code and how many work it will be
to update U-Boot to a current hush version. But may this would be make more
sense as hush is in active development. Also may to discuss with hush maintainers
to upstream U-Boot changes... this would make in my eyes more sense.
> = Why Lil?
>
> When looking for a suitable replacement shell, I evaluated implementations using
> the following criteria:
>
> - It must have a GPLv2-compatible license.
> - It must be written in C, and have no major external dependencies.
> - It must support bare function calls. That is, a script such as 'foo bar'
> should invoke the function 'foo' with the argument 'bar'. This preserves the
> shell-like syntax we expect.
> - It must be small. The eventual target is that it compiles to around 10KiB with
> -Os and -ffunction-sections.
> - There should be good tests. Any tests at all are good, but a functioning suite
> is better.
> - There should be good documentation
> - There should be comments in the source.
> - It should be "finished" or have only slow development. This will hopefully
> make it easier to port changes.
>
> Notably absent from the above list is performance. Most scripts in U-Boot will
> be run once on boot. As long as the time spent evaluating scripts is kept under
> a reasonable threshold (a fraction of the time spend initializing hardware or
> reading data from persistant storage), there is no need to optimize for speed.
>
> In addition, I did not consider updating Hush from Busybox. The mismatch in
> computing environment expectations (as noted in the "New shell" section above)
> still applies. IMO, this mismatch is the biggest reason that things like
> functions and command substitution have been excluded from the U-Boot's Hush.
>
> == lil
>
> - zLib
> - TCL
> - Compiles to around 10k with no builtins. To 25k with builtins.
> - Some tests, but not organized into a suite with expected output. Some evidence
> that the author ran APL, but no harness.
> - Some architectural documentation. Some for each functions, but not much.
> - No comments :l
> - 3.5k LoC
>
> == picol
>
> - 2-clause BSD
> - TCL
> - Compiles to around 25k with no builtins. To 80k with builtins.
> - Tests with suite (in-language). No evidence of fuzzing.
> - No documentation :l
> - No comments :l
> - 5k LoC
>
> == jimtcl
>
> - 2-clause BSD
> - TCL
> - Compiles to around 95k with no builtins. To 140k with builtins. Too big...
>
> == boron
>
> - LGPLv3+ (so this is right out)
> - REBOL
> - Compiles to around 125k with no builtins. To 190k with builtins. Too big...
>
> == libmawk
>
> - GPLv2
> - Awk
> - Compiles to around 225k. Too big...
>
> == libfawk
>
> - 3-clause BSD
> - Uses bison+yacc...
> - Awk; As it turns out, this has parentheses for function calls.
> - Compiles to around 24-30k. Not sure how to remove builtins.
> - Test suite (in-language). No fuzzing.
> - Tutorial book. No function reference.
> - No comments
> - Around 2-4k LoC
>
> == MicroPython
>
> - MIT
> - Python (but included for completeness)
> - Compiles to around 300k. Too big...
>
> == mruby/c
>
> - 3-clause BSD
> - Ruby
> - Compiles to around 85k without builtins and 120k with. Too big...
>
> == eLua
>
> - MIT
> - Lua
> - Build system is a royal pain (custom and written in Lua with external deps)
> - Base binary is around 250KiB and I don't want to deal with reducing it
>
> So the interesting/viable ones are
> - lil
> - picol
> - libfawk (maybe)
Very nice overview, thanks!
bye,
Heiko
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52 Fax: +49-8142-66989-80 Email: hs@denx.de
^ permalink raw reply [flat|nested] 107+ messages in thread* Re: [RFC PATCH 00/28] cli: Add a new shell
2021-07-08 3:49 ` Heiko Schocher
@ 2021-07-08 4:26 ` Sean Anderson
0 siblings, 0 replies; 107+ messages in thread
From: Sean Anderson @ 2021-07-08 4:26 UTC (permalink / raw)
To: hs, u-boot, Tom Rini
Cc: Marek Behún, Wolfgang Denk, Simon Glass, Roland Gaudig,
Heinrich Schuchardt, Kostas Michalopoulos
On 7/7/21 11:49 PM, Heiko Schocher wrote:
> Hello Sean,
>
> On 01.07.21 08:15, Sean Anderson wrote:
>> Well, this has been sitting on my hard drive for too long without feedback
>> ("Release early, release often"), so here's the first RFC. This is not ready to
>> merge (see the "Future work" section below), but the shell is functional and at
>> least partially tested.
>>
>> The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes
>> over on sandbox.
>>
>> add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
>
> Thanks for this comparison. Here just my thoughts, but I have currently to
> low time to follow here completly, so sorry if I miss stuff already
> discussed.
>
> From my side of view we can add a second shell beside hush, and as you
> already mentioned it may is even possible to start LIL from hush. This
> would be cool.
>
> But I disagree in the following points:
>
> - I do not see why LIL should *replace* hush.
This is the eventual goal. This series just adds an option at build time
(CONFIG_LIL) which uses LIL as the shell instead of Hush. But anyone who
writes a series like this must be convinced that it is better than what
already exists. So I write that my goal is to replace hush, even if I
think that it will take a long while.
> This may break a lot of current boards. Also I see no need to do
> this, as if you do complex stuff in u-boot shell it may is the
> wrong way and you have to think about... and I see no big
> discussions here about BUGS in our current hush implementation ...
> may it makes sense to bring them up and fix?
I have been told not to work on the current shell; that it is too old
and there is no point in improving it. Since we will be starting afresh
anyway, and I do not think that the Hush model is particularly good, I
chose to work from a different base.
> - Is LIL really so good code, that it is done? Or is there only no community
> which develop further?
AFAIK there is not too much of a LIL community. The code quality is
something I found out after working on it for a while. From a cursory
inspection, it was around the same as all the other shells I looked at
(no comments, etc). FWIW the current Hush shell does not really handle
errors either. For example, on (x)malloc failure, we print and hang.
This could easily be done for LIL, but it is not so nice when your
language has recursion. So I have tried to add error checking where
possible. And of course, Hush ignores missing variables as well.
> - Do we really want code, which does no error checking? Commentless code...
>
> you wrote:
>> - There is a serious error handling problem. Most original LIL code never
>> checked errors. In almost every case, errors were silently ignored, even
>> malloc failures! While I have designed new code to handle errors properly,
>
> Which really let me think, nobody is interested in this code ... this
> sounds like crap... and this brings me to the next point
>
> - Do we not have the same problem as with hush, if we change LIL code?
I think the primary problem with Hush was that we expected to backport
updates at all. I think such a core UI feature should not have so many
ifdefs in it when no backport was ever made. It is difficult to modify
Hush because not only do you have to understand how U-Boot uses some
feature, you also have to understand how Busybox uses some feature. And
Hush of course has less comments as LIL does (after all patches in this
series are applied).
> Do you plan to upstream your changes?
I don't view it as especially important. I view LIL as a starting point
which can be modified to suit U-Boot. It may end up that each individual
component is changed in some way, but starting with a working shell has
been very helpful for development.
> So please do not understand me wrong, I see you invested much time here,
> but I see LIL not in the short term as a replacement for hush.
>
> I would accept to add LIL as an option and we will see, how good it works.
That is what I intend to do.
> And if we have in lets say 2-3 years more boards which use LIL instead of
> hush *and* also active maintained LIL code, we may can think of getting rid
> of hush... and may have a "hush compatibility" layer...
>
> I have no idea of current hush mainline code and how many work it will be
> to update U-Boot to a current hush version. But may this would be make more
> sense as hush is in active development. Also may to discuss with hush maintainers
> to upstream U-Boot changes... this would make in my eyes more sense.
AFAICT most of the U-Boot changes are removal of features, workarounds
for unimplemented functionality (e.g. anywhere Hush does fork/exec), and
some small additions like command repetition which cannot be upstreamed
because it would break any semblance of POSIX compatibility.
--Sean
>> = Why Lil?
>>
>> When looking for a suitable replacement shell, I evaluated implementations using
>> the following criteria:
>>
>> - It must have a GPLv2-compatible license.
>> - It must be written in C, and have no major external dependencies.
>> - It must support bare function calls. That is, a script such as 'foo bar'
>> should invoke the function 'foo' with the argument 'bar'. This preserves the
>> shell-like syntax we expect.
>> - It must be small. The eventual target is that it compiles to around 10KiB with
>> -Os and -ffunction-sections.
>> - There should be good tests. Any tests at all are good, but a functioning suite
>> is better.
>> - There should be good documentation
>> - There should be comments in the source.
>> - It should be "finished" or have only slow development. This will hopefully
>> make it easier to port changes.
>>
>> Notably absent from the above list is performance. Most scripts in U-Boot will
>> be run once on boot. As long as the time spent evaluating scripts is kept under
>> a reasonable threshold (a fraction of the time spend initializing hardware or
>> reading data from persistant storage), there is no need to optimize for speed.
>>
>> In addition, I did not consider updating Hush from Busybox. The mismatch in
>> computing environment expectations (as noted in the "New shell" section above)
>> still applies. IMO, this mismatch is the biggest reason that things like
>> functions and command substitution have been excluded from the U-Boot's Hush.
>>
>> == lil
>>
>> - zLib
>> - TCL
>> - Compiles to around 10k with no builtins. To 25k with builtins.
>> - Some tests, but not organized into a suite with expected output. Some evidence
>> that the author ran APL, but no harness.
>> - Some architectural documentation. Some for each functions, but not much.
>> - No comments :l
>> - 3.5k LoC
>>
>> == picol
>>
>> - 2-clause BSD
>> - TCL
>> - Compiles to around 25k with no builtins. To 80k with builtins.
>> - Tests with suite (in-language). No evidence of fuzzing.
>> - No documentation :l
>> - No comments :l
>> - 5k LoC
>>
>> == jimtcl
>>
>> - 2-clause BSD
>> - TCL
>> - Compiles to around 95k with no builtins. To 140k with builtins. Too big...
>>
>> == boron
>>
>> - LGPLv3+ (so this is right out)
>> - REBOL
>> - Compiles to around 125k with no builtins. To 190k with builtins. Too big...
>>
>> == libmawk
>>
>> - GPLv2
>> - Awk
>> - Compiles to around 225k. Too big...
>>
>> == libfawk
>>
>> - 3-clause BSD
>> - Uses bison+yacc...
>> - Awk; As it turns out, this has parentheses for function calls.
>> - Compiles to around 24-30k. Not sure how to remove builtins.
>> - Test suite (in-language). No fuzzing.
>> - Tutorial book. No function reference.
>> - No comments
>> - Around 2-4k LoC
>>
>> == MicroPython
>>
>> - MIT
>> - Python (but included for completeness)
>> - Compiles to around 300k. Too big...
>>
>> == mruby/c
>>
>> - 3-clause BSD
>> - Ruby
>> - Compiles to around 85k without builtins and 120k with. Too big...
>>
>> == eLua
>>
>> - MIT
>> - Lua
>> - Build system is a royal pain (custom and written in Lua with external deps)
>> - Base binary is around 250KiB and I don't want to deal with reducing it
>>
>> So the interesting/viable ones are
>> - lil
>> - picol
>> - libfawk (maybe)
>
> Very nice overview, thanks!
>
> bye,
> Heiko
>
^ permalink raw reply [flat|nested] 107+ messages in thread