libuuid related tasks for next v2.39

libuuid related tasks for next v2.39

[Karel Zak]

 Hi,

 I'd like to invest time to libuuid and uuidd in the next v2.39.

 Stanislav had same ideas and a lot of research in libuuid code to
 make things more stable, and I'd like to improve testability of the
 all UUID stack we have in util-linux. 

 TODO:

  * uuid_generate_* functions refactoring to make things readable and
    to separate ways how UUIDs are generated

  * clean up uuidd to generate UUID only by the most robust UUIDs way 
    without any fallback (use uuid_generate_time_sync_by_file())

  * report all issues on uuidd stderr

  * optionally keep clockfile (/var/lib/libuuid/clock.txt) in uuidd hands
    to make sure the file is not shared with other users if the system
    is only-uuidd based; for example add --clock-file uuidd option to
    overwrite libuuid default.

  * invite our SAP friends to the upstream development to share ideas
    and requirement :-)

  * rewrite misc-utils/test_uuidd.c to something commonly usable
    (uuidinfo tool?) and use it in util-linux regression tests,
    features:

     - detect if uuidd is running
     - add statistics to the uuidd (number of generated UUIDs,
       clients, errors, ...)
     - add new UUIDD_OP_* to the uuidd to get additional information
       about the daemon (paths, statistics, etc.)
     - stress test (like the current test_uuidd.c)


  * anything else?


   Karel
  

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: libuuid related tasks for next v2.39

[Stanislav Brabec]

Dne 01. 04. 22 v 11:21 Karel Zak napsal(a):
>
>   Hi,
>
>   I'd like to invest time to libuuid and uuidd in the next v2.39.
>
>   Stanislav had same ideas and a lot of research in libuuid code to
>   make things more stable, and I'd like to improve testability of the
>   all UUID stack we have in util-linux.
>
>   TODO:
>
>    * uuid_generate_* functions refactoring to make things readable and
>      to separate ways how UUIDs are generated
Yes, it makes sense.
>    * clean up uuidd to generate UUID only by the most robust UUIDs way
>      without any fallback (use uuid_generate_time_sync_by_file())
Yes.
>    * report all issues on uuidd stderr
Yes. Or even exit with error on any (potentially permanent) issue.

But more important:

All issues should generate error to the libuuid. Otherwise the 
application gets downgraded unsafe UUIDs without properly detecting it.

>    * optionally keep clockfile (/var/lib/libuuid/clock.txt) in uuidd hands
>      to make sure the file is not shared with other users if the system
>      is only-uuidd based; for example add --clock-file uuidd option to
>      overwrite libuuid default.
I guess it is mandatory, otherwise we will have a problem of the root 
stealing of the lock file.

The proposed default was ~/.libuuid_clock.txt

There should be an internal function that sets the path to the clock 
file. It probably makes sense to make the path configurable in the 
config file, but the function should not be public:
- Only uuidd and the configuration read will use the function.
- Applications on systems systems without writable home could configure 
it e. g. to /run.
- Using different paths for different applications would cause bad 
things; private function makes it impossible.
- It will ensure that nothing else than uuidd will use 
/var/lib/libuuid/clock.txt.

I have applied a temporary work-around for the root stealing problem:

uuidd.service:

ExecStartPre=+-/usr/bin/chown uuidd:uuidd /var/lib/libuuid/clock.txt

But it is ugly and it cures just a symptoms and does not work with older version of systemd.

>    * invite our SAP friends to the upstream development to share ideas
>      and requirement :-)
>
>    * rewrite misc-utils/test_uuidd.c to something commonly usable
>      (uuidinfo tool?) and use it in util-linux regression tests,
>      features:
>
>       - detect if uuidd is running
>       - add statistics to the uuidd (number of generated UUIDs,
>         clients, errors, ...)
>       - add new UUIDD_OP_* to the uuidd to get additional information
>         about the daemon (paths, statistics, etc.)
>       - stress test (like the current test_uuidd.c)
>
>
>    * anything else?
>
Add support for three levels of safe state for uuid_generate_time_safe():
- Absolutely safe (uuidd was used, and there was no error in uuidd). The 
UUID is guaranteed to be unique.
- Safe for the current UID (clock file was used, but no uuidd). The UUID 
is guaranteed to be unique for the current UID, but there is no 
guarantee that other UIDs.
- Unsafe (clock file use failed, uuidd use failed). The UUID is not 
guaranteed to be unique at all. Just a random.

It opens a discussion whether the new return value should be integrated 
to the current functions, or add a new function with three return 
values. Or even change the current function to the macro, so the newly 
compiled code will stop to use the old function name.

If we will change the API, then no code will fail. But the result will 
depend on the check implementation:
If the return value will be compared by == -1, it will catch only the 
unsafe return.
If the return value will be compared by != 0, it will catch both unsafe 
and current-UID-safe.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Křižíkova 148/34 (Corso IIa)                    tel: +420 284 084 060
186 00 Praha 8-Karlín                          fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: libuuid related tasks for next v2.39

[Karel Zak]

On Mon, Apr 11, 2022 at 03:36:01PM +0200, Stanislav Brabec wrote:
> Dne 01. 04. 22 v 11:21 Karel Zak napsal(a):
> >    * uuid_generate_* functions refactoring to make things readable and
> >      to separate ways how UUIDs are generated
> Yes, it makes sense.
> >    * clean up uuidd to generate UUID only by the most robust UUIDs way
> >      without any fallback (use uuid_generate_time_sync_by_file())
> Yes.
> >    * report all issues on uuidd stderr
> Yes. Or even exit with error on any (potentially permanent) issue.
> 
> But more important:
> 
> All issues should generate error to the libuuid. Otherwise the application
> gets downgraded unsafe UUIDs without properly detecting it.
 
Yes.

> >    * optionally keep clockfile (/var/lib/libuuid/clock.txt) in uuidd hands
> >      to make sure the file is not shared with other users if the system
> >      is only-uuidd based; for example add --clock-file uuidd option to
> >      overwrite libuuid default.
> I guess it is mandatory, otherwise we will have a problem of the root
> stealing of the lock file.
> 
> The proposed default was ~/.libuuid_clock.txt
> 
> There should be an internal function that sets the path to the clock file.
> It probably makes sense to make the path configurable in the config file,
> but the function should not be public:
>
> - Only uuidd and the configuration read will use the function.
> - Applications on systems systems without writable home could configure it
> e. g. to /run.
> - Using different paths for different applications would cause bad things;
> private function makes it impossible.

I can imagine this as public :-) The majority of users will use basic
simple API (just give me UUID). The advanced minority has to read docs
:-)

I can imagine use-case where you have more independent users who need
time based UUIDs, but they don't care about collisions with the other
users (for example two web applications with independent DBs, etc.)/

The default should be safe as much as possible, but we should not try
to be more smart than admins/developers as we have no clue about their
use-cases ;-)

But this is not so important for now, we can keep it as uuidd only
in UUIDD_PRIVATE namespace in libuuid/src/libuuid.sym.

> - It will ensure that nothing else than uuidd will use
> /var/lib/libuuid/clock.txt.
> 
> I have applied a temporary work-around for the root stealing problem:
> 
> uuidd.service:
> 
> ExecStartPre=+-/usr/bin/chown uuidd:uuidd /var/lib/libuuid/clock.txt
> 
> But it is ugly and it cures just a symptoms and does not work with older version of systemd.
> 
> >    * invite our SAP friends to the upstream development to share ideas
> >      and requirement :-)
> > 
> >    * rewrite misc-utils/test_uuidd.c to something commonly usable
> >      (uuidinfo tool?) and use it in util-linux regression tests,
> >      features:
> > 
> >       - detect if uuidd is running
> >       - add statistics to the uuidd (number of generated UUIDs,
> >         clients, errors, ...)
> >       - add new UUIDD_OP_* to the uuidd to get additional information
> >         about the daemon (paths, statistics, etc.)
> >       - stress test (like the current test_uuidd.c)
> > 
> > 
> >    * anything else?
> > 
> Add support for three levels of safe state for uuid_generate_time_safe():
> - Absolutely safe (uuidd was used, and there was no error in uuidd). The
> UUID is guaranteed to be unique.
> - Safe for the current UID (clock file was used, but no uuidd). The UUID is
> guaranteed to be unique for the current UID, but there is no guarantee that
> other UIDs.
> - Unsafe (clock file use failed, uuidd use failed). The UUID is not
> guaranteed to be unique at all. Just a random.
> 
> It opens a discussion whether the new return value should be integrated to
> the current functions, or add a new function with three return values. Or
> even change the current function to the macro, so the newly compiled code
> will stop to use the old function name.
> 
> If we will change the API, then no code will fail. But the result will
> depend on the check implementation:
> If the return value will be compared by == -1, it will catch only the unsafe
> return.
> If the return value will be compared by != 0, it will catch both unsafe and
> current-UID-safe.

If we decide to change the API (introduce a new functions) then I'd like to 
see something based on handler/context. IMHO only context/handler based libraries
are extendible and easy to maintain and ready for relatively complex tasks.

    libuuid_context *cxt = uuid_new_context();

    uuid_context_set_clockfile(cxt, "/somewhere/clock.txt");

    uuid_context_generate_time(cxt, &uuid, &state);

where @state is UUID_STATE_{SAFE,UNSAFE,UUIDD,...}

we can also add

    uuid_context_force_state(cxt, UUID_STATE_ABSOLUTE);

to make sure uuid_context_generate_time() returns only requested
quality or error, etc.

We can also for uuidd make locking optional:

    uuid_context_disable_lock(cxt, TRUE);

etc.

We can also use the context for communication with uuidd

    uuid_uuidd *uuidd = uuid_new_uuidd();

    uuid_uuidd_get_state()
    uuid_uuidd_get_pid()
    uuid_uuidd_get_statistic()
    uuid_uuidd_get_clockfile()
    uuid_uuidd_get_error()

    uuid_uuidd_get_uuid()
    ....


Anyway, the first step we need uuid_generate_* functions
refactoring to clean up the spaghetti stuff.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: libuuid related tasks for next v2.39

[Stanislav Brabec]

Dne 07. 04. 22 v 10:23 Trapp, Michael napsal(a):
> For uuidd this would result in:
> -no need for a lock on LIBUUID_CLOCK_FILE, what also might help to 
> solve some authorization issues.
This should be a benefit.

The only possible downside would be losing of context on uuidd restart. 
That happens e. g. during a security update.Could this be a problem? 
Really fast machines restart in a small fraction of second.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                         e-mail: sbrabec@suse.com
Křižíkova 148/34 (Corso IIa)                    tel: +420 284 084 060
186 00 Praha 8-Karlín                          fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: libuuid related tasks for next v2.39

[Karel Zak]

On Fri, Apr 01, 2022 at 11:49:17AM +0200, Aurélien Lajoie wrote:
> What about nice helpers to get fast access to the different part of the
> uuid?

I'm not sure if I understand. What do you mean with the helpers?

> We need to keep an eye on performance, php uuid code was more perf than the
> previous version of our lib.

Yes, printf sucks. It's fixed now. Thanks to you :-)

> Perhaps add some test to catch big impact on serialization and
> deserialization

Good point, maybe it's something we can add to (suggested)
uuidinfo command.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: libuuid related tasks for next v2.39

[Karel Zak]

On Tue, Apr 12, 2022 at 01:15:56PM +0200, Aurélien Lajoie wrote:
> On Tue, Apr 12, 2022 at 12:35 PM Karel Zak <kzak@redhat.com> wrote:
> >
> > On Fri, Apr 01, 2022 at 11:49:17AM +0200, Aurélien Lajoie wrote:
> > > What about nice helpers to get fast access to the different part of the
> > > uuid?
> >
> > I'm not sure if I understand. What do you mean with the helpers?
> 
> Accessing the different part of the uuid
> version
> time_low
> time_mid
> time_hi_and_version
> clock_seq_hi_and_res clock_seq_low
> node

This is obvious, I'm trying to understand "Accessing". Do you mean
unparse the specified parts or update/set it? For what use-case?

We have something like this for command line (misc-utils/uuidparse.c).

$ uuidparse $(uuidgen --time)
UUID                                  VARIANT TYPE       TIME
970f197e-ba55-11ec-aa93-43a3097bca4a  DCE     time-based 2022-04-12 13:42:11,097843+02:00


Do you mean anything like;

     uuid_time_get_version(uuidstr);
     uuid_time_get_clock_seq()
     ...

or the same but for already parsed uuid (uuid_t)?


    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com