Adding map read write API to bpftool gen skeleton sub command.

Adding map read write API to bpftool gen skeleton sub command.

[Dushyant Behl]

Hi folks,

I have been testing the use of BTF to generate read/write API on maps
with specific key value types which can be extracted from the BTF
info.
I have already developed a small tool to test this which uses the BTF
information in the ebpf binary to automatically generate map
type-specific CRUD APIs. This can be built on top of the libbpf api in
the sense that it can provide key and value type info and type
checking on top of the existing api.

Our goal is to ease the development of ebpf user space applications
and was wondering if this feature could be integrated to "bpftool gen
skeleton" sub command.
I was wondering if you think that such a
feature will be inline with the intent of bpftool and will be of value
to its users.
I am happy to have more discussion or a meeting on how this could be
approached and implemented and if it would be a good addition to
bpftool.

Please let me know.

Thanks,
Dushyant

Re: Adding map read write API to bpftool gen skeleton sub command.

[Alexei Starovoitov]

On Tue, Jan 31, 2023 at 10:41 PM Dushyant Behl
<myselfdushyantbehl@gmail.com> wrote:
>
> Hi folks,
>
> I have been testing the use of BTF to generate read/write API on maps
> with specific key value types which can be extracted from the BTF
> info.
> I have already developed a small tool to test this which uses the BTF
> information in the ebpf binary to automatically generate map
> type-specific CRUD APIs. This can be built on top of the libbpf api in
> the sense that it can provide key and value type info and type
> checking on top of the existing api.

What is 'CRUD APIs' ?
Could you give an example of what kind of code will be generated?

> Our goal is to ease the development of ebpf user space applications
> and was wondering if this feature could be integrated to "bpftool gen
> skeleton" sub command.
I was wondering if you think that such a
> feature will be inline with the intent of bpftool and will be of value
> to its users.
> I am happy to have more discussion or a meeting on how this could be
> approached and implemented and if it would be a good addition to
> bpftool.
>
> Please let me know.
>
> Thanks,
> Dushyant

Re: Adding map read write API to bpftool gen skeleton sub command.

[Dushyant Behl]

On Fri, Feb 3, 2023 at 10:12 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Jan 31, 2023 at 10:41 PM Dushyant Behl
> <myselfdushyantbehl@gmail.com> wrote:
> >
> > Hi folks,
> >
> > I have been testing the use of BTF to generate read/write API on maps
> > with specific key value types which can be extracted from the BTF
> > info.
> > I have already developed a small tool to test this which uses the BTF
> > information in the ebpf binary to automatically generate map
> > type-specific CRUD APIs. This can be built on top of the libbpf api in
> > the sense that it can provide key and value type info and type
> > checking on top of the existing api.
>
> What is 'CRUD APIs' ?
> Could you give an example of what kind of code will be generated?

Hi Alexei,

By CRUD, I wanted to mean the API for Create, Read, Update and Delete
functionality on the maps.
In equivalent terms to libbpf it would be
bpf_map_<update/lookup/delete>_elem API.

Currently with the generated skeleton users are able to create and load BPF
objects and my idea was to extract actual map key and value types from
the BTF info
and provide an API along with the current skeleton to read, update,
and delete map fields for users.

As an example, I have taken a slightly modified version of a sample
from the bpftool-gen manpage

$ cat example2_modified.c
    #include <linux/ptrace.h>
    #include <linux/bpf.h>
    #include <bpf/bpf_helpers.h>

    struct my_key_t { int k; };
    struct my_value_t { long v; };

    struct {
         __uint(type, BPF_MAP_TYPE_HASH);
         __uint(max_entries, 128);
         __type(key, struct my_key_t);
         __type(value, struct my_value_t);
     } my_map SEC(".maps");

     SEC("raw_tp/sys_exit")
     int handle_sys_exit(struct pt_regs *ctx)
     {
         struct my_key_t zero;
         zero.k = 0;
         bpf_map_lookup_elem(&my_map, &zero);
         return 0;
      }

Currently the gen-skeleton structure for this program looks like below,

struct example2 {
    struct bpf_object_skeleton *skeleton;
    struct bpf_object *obj;
    struct {
        struct bpf_map *my_map;
    } maps;
    struct {
        struct bpf_program *handle_sys_exit;
    } progs;
    struct {
        struct bpf_link *handle_sys_exit;
    } links;

    #ifdef __cplusplus
    static inline struct example2 *open(const struct
bpf_object_open_opts *opts = nullptr);
    static inline struct example2 *open_and_load();
    static inline int load(struct example2 *skel);
    static inline int attach(struct example2 *skel);
    static inline void detach(struct example2 *skel);
    static inline void destroy(struct example2 *skel);
    static inline const void *elf_bytes(size_t *sz);
    #endif /* __cplusplus */
};

The extra code and API I wanted to expose would look something like below,

/* types reconstructed from BTF */
struct my_key_t { int k; };
struct my_value_t { long v; };

/* Generic read write API */
static inline void *read_map(bpf_map *m, void *k) {
 /* read the map and return value */
}

static inline void write_map(bpf_map *m, void* k, void *v) {
  /* write the map and return value */
}

static inline void delete_map(bpf_map *m, void *k) {
 /* delete the key */
}

/* Macros for direct access to map type */
#define READ_MY_MAP(k) read_map(example2->maps.my_map, k)
#define WRITE_MY_MAP(k,v) write_map(example2->maps.my_map, k, v)
#define DELETE_MY_MAP(k) read_map(example2->maps.my_map, k)

Currently I have a python utility which I have tested to detect the type of a
BTF object and this is what I generated when I run it on the above example,
it currently outputs json so I am just showing that here

"maps": [{
  "name": "my_map",
  "key": {
    "variable_name": "my_key_t",
    "kind": "STRUCT",
     "member": [ {
        "variable_name": "k",
        "type_name": "int",
        "size": 4,
        "kind": "INT",
        "input": null
    }]
  },
  "value": {
    "variable_name": "my_value_t",
    "kind": "STRUCT",
     "member": [{
       "variable_name": "v",
       "type_name": "long int",
       "size": 8,
       "kind": "INT",
       "input": null
     }]
  }
}]

This is a work in progress and I am unclear if certain type information
might not be correctly extractable.

Please let me know if this a) clarifies the intent, b) whether this
feature will be inline
with bpftool's philosophy and c) whether such additional features will
be useful.

Thanks,
Dushyant

> > Our goal is to ease the development of ebpf user space applications
> > and was wondering if this feature could be integrated to "bpftool gen
> > skeleton" sub command.
I was wondering if you think that such a
> > feature will be inline with the intent of bpftool and will be of value
> > to its users.
> > I am happy to have more discussion or a meeting on how this could be
> > approached and implemented and if it would be a good addition to
> > bpftool.
> >
> > Please let me know.
> >
> > Thanks,
> > Dushyant

Re: Adding map read write API to bpftool gen skeleton sub command.

[Andrii Nakryiko]

On Tue, Feb 7, 2023 at 10:57 PM Dushyant Behl
<myselfdushyantbehl@gmail.com> wrote:
>
> On Fri, Feb 3, 2023 at 10:12 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Tue, Jan 31, 2023 at 10:41 PM Dushyant Behl
> > <myselfdushyantbehl@gmail.com> wrote:
> > >
> > > Hi folks,
> > >
> > > I have been testing the use of BTF to generate read/write API on maps
> > > with specific key value types which can be extracted from the BTF
> > > info.
> > > I have already developed a small tool to test this which uses the BTF
> > > information in the ebpf binary to automatically generate map
> > > type-specific CRUD APIs. This can be built on top of the libbpf api in
> > > the sense that it can provide key and value type info and type
> > > checking on top of the existing api.
> >
> > What is 'CRUD APIs' ?
> > Could you give an example of what kind of code will be generated?
>
> Hi Alexei,
>
> By CRUD, I wanted to mean the API for Create, Read, Update and Delete
> functionality on the maps.
> In equivalent terms to libbpf it would be
> bpf_map_<update/lookup/delete>_elem API.
>
> Currently with the generated skeleton users are able to create and load BPF
> objects and my idea was to extract actual map key and value types from
> the BTF info
> and provide an API along with the current skeleton to read, update,
> and delete map fields for users.
>
> As an example, I have taken a slightly modified version of a sample
> from the bpftool-gen manpage
>
> $ cat example2_modified.c
>     #include <linux/ptrace.h>
>     #include <linux/bpf.h>
>     #include <bpf/bpf_helpers.h>
>
>     struct my_key_t { int k; };
>     struct my_value_t { long v; };
>
>     struct {
>          __uint(type, BPF_MAP_TYPE_HASH);
>          __uint(max_entries, 128);
>          __type(key, struct my_key_t);
>          __type(value, struct my_value_t);
>      } my_map SEC(".maps");
>
>      SEC("raw_tp/sys_exit")
>      int handle_sys_exit(struct pt_regs *ctx)
>      {
>          struct my_key_t zero;
>          zero.k = 0;
>          bpf_map_lookup_elem(&my_map, &zero);
>          return 0;
>       }
>
> Currently the gen-skeleton structure for this program looks like below,
>
> struct example2 {
>     struct bpf_object_skeleton *skeleton;
>     struct bpf_object *obj;
>     struct {
>         struct bpf_map *my_map;
>     } maps;
>     struct {
>         struct bpf_program *handle_sys_exit;
>     } progs;
>     struct {
>         struct bpf_link *handle_sys_exit;
>     } links;
>
>     #ifdef __cplusplus
>     static inline struct example2 *open(const struct
> bpf_object_open_opts *opts = nullptr);
>     static inline struct example2 *open_and_load();
>     static inline int load(struct example2 *skel);
>     static inline int attach(struct example2 *skel);
>     static inline void detach(struct example2 *skel);
>     static inline void destroy(struct example2 *skel);
>     static inline const void *elf_bytes(size_t *sz);
>     #endif /* __cplusplus */
> };
>
> The extra code and API I wanted to expose would look something like below,
>
> /* types reconstructed from BTF */
> struct my_key_t { int k; };
> struct my_value_t { long v; };
>
> /* Generic read write API */
> static inline void *read_map(bpf_map *m, void *k) {
>  /* read the map and return value */
> }
>
> static inline void write_map(bpf_map *m, void* k, void *v) {
>   /* write the map and return value */
> }
>
> static inline void delete_map(bpf_map *m, void *k) {
>  /* delete the key */
> }
>
> /* Macros for direct access to map type */
> #define READ_MY_MAP(k) read_map(example2->maps.my_map, k)
> #define WRITE_MY_MAP(k,v) write_map(example2->maps.my_map, k, v)
> #define DELETE_MY_MAP(k) read_map(example2->maps.my_map, k)
>
> Currently I have a python utility which I have tested to detect the type of a
> BTF object and this is what I generated when I run it on the above example,
> it currently outputs json so I am just showing that here
>
> "maps": [{
>   "name": "my_map",
>   "key": {
>     "variable_name": "my_key_t",
>     "kind": "STRUCT",
>      "member": [ {
>         "variable_name": "k",
>         "type_name": "int",
>         "size": 4,
>         "kind": "INT",
>         "input": null
>     }]
>   },
>   "value": {
>     "variable_name": "my_value_t",
>     "kind": "STRUCT",
>      "member": [{
>        "variable_name": "v",
>        "type_name": "long int",
>        "size": 8,
>        "kind": "INT",
>        "input": null
>      }]
>   }
> }]
>
> This is a work in progress and I am unclear if certain type information
> might not be correctly extractable.
>
> Please let me know if this a) clarifies the intent, b) whether this
> feature will be inline
> with bpftool's philosophy and c) whether such additional features will
> be useful.
>

I'm not really convinced, to be honest. Those generic
{read,write,delete}_map helpers are very similar to libbpf's
bpf_map_{lookup,update,delete}_elem(), except they accept fd, which
you can get with bpf_map__fd().

{READ, WRITE, DELETE}_MY_MAP() macros definitely go quite against the
spirit of BPF skeleton.

Also keep in mind that libbpf provides
bpf_map__{lookup,update,delete}_elem() high-level APIs, that take
struct bpf_map * directly. If you want to avoid hard-coding sizes of
key/value, then you can fetch them at runtime with
bpf_map__{key,value}_size().

So in short, all you are trying to do seems pretty doable completely
outside of BPF skeleton.

> Thanks,
> Dushyant
>
> > > Our goal is to ease the development of ebpf user space applications
> > > and was wondering if this feature could be integrated to "bpftool gen
> > > skeleton" sub command.
I was wondering if you think that such a
> > > feature will be inline with the intent of bpftool and will be of value
> > > to its users.
> > > I am happy to have more discussion or a meeting on how this could be
> > > approached and implemented and if it would be a good addition to
> > > bpftool.
> > >
> > > Please let me know.
> > >
> > > Thanks,
> > > Dushyant