Unable to find pw entry for uid

Unable to find pw entry for uid

[Calvin Chiang]

Hi



I’m attempting to get autofs (using cifs) to automatically mount user
directories for me using existing Kerberos credentials.

But it doesn’t even make it to the Kerberos section of cifs.upcall as



My /etc/auto.master config looks like this:



/cifs /etc/auto.cifs



My /etc/auto.cifs config looks like this:



folder1   -fstype=cifs,multiuser,uid=alice,user=alice,cruid=alice,sec=krb5,vers=3.0
   ://member-server.cyberloop.local/sharedfolder/folder1



Note:

    I’ve hardcoded the uid/cruid/user, as the expansion didn’t seem
tobe working properly.
    The user “alice” has uid 1023001106
    The user “alice”, is the owner of the krb5 ticket /tmp/krb5_1023001106





when I attempt to access the folder /cifs/folder1 I get the following error:



May  3 14:34:41 centos8 kernel: fs/cifs/cifs_spnego.c: key description
= ver=0x2;host=member-server.cyberloop.local;ip4=192.168.0.102;sec=krb5;uid=0x3cf9c212;creduid=0x3cf9c212;user=alice;pid=0x10ad28

May  3 14:34:41 centos8 cifs.upcall[1092907]: key description:
cifs.spnego;0;0;39010000;ver=0x2;host=member-server.cyberloop.local;ip4=192.168.0.102;sec=krb5;uid=0x3cf9c212;creduid=0x3cf9c212;user=alice;pid=0x10ad28

May  3 14:34:41 centos8 cifs.upcall[1092907]: ver=2

May  3 14:34:41 centos8 cifs.upcall[1092907]: host=member-server.cyberloop.local

May  3 14:34:41 centos8 cifs.upcall[1092907]: ip=192.168.0.102

May  3 14:34:41 centos8 cifs.upcall[1092907]: sec=1

May  3 14:34:41 centos8 cifs.upcall[1092907]: uid=1023001106

May  3 14:34:41 centos8 cifs.upcall[1092907]: creduid=1023001106

May  3 14:34:41 centos8 cifs.upcall[1092907]: user=alice

May  3 14:34:41 centos8 cifs.upcall[1092907]: pid=1092904

May  3 14:34:41 centos8 cifs.upcall[1092907]: Unable to find pw entry
for uid 1023001106: Success

May  3 14:34:41 centos8 cifs.upcall[1092907]: Exit status 1



The weird thing here is that it hits this section of cifs.upcall.c and
errors here:



    pw = getpwuid(uid);

    if (!pw) {

        syslog(LOG_ERR, "Unable to find pw entry for uid %d: %s\n",

            uid, strerror(errno));

        rc = 1;

        goto out;

    }



Now oddly the strerror(errno) is actually returning SUCCESS



But the pw = getpwuid(uid); is failing.



Getpwuid(uid) is calling nss.



My nss config looks like this:



passwd:         files systemd sss

group:          files systemd sss

shadow:         files sss

gshadow:        files



hosts:          files dns

networks:       files



protocols:      db files

services:       db files sss

ethers:         db files

rpc:            db files



netgroup:       nis sss

sudoers:        files sss

automount:      sss



and the output from the sssd_nss.log is:



    (Mon May  3 13:05:12 2021) [sssd[nss]] [cache_req_search_send]
(0x0400): CR #114: Object found, but needs to be refreshed.

    (Mon May  3 13:05:12 2021) [sssd[nss]] [cache_req_search_dp]
(0x0400): CR #114: Performing midpoint cache update of
[UID:1023001106@cyberloop.local]

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_dp_issue_request]
(0x0400): Issuing request for
[0x559bd1d69e70:1:1023001106@cyberloop.local]

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_dp_get_account_msg]
(0x0400): Creating request for
[cyberloop.local][0x1][BE_REQ_USER][idnumber=1023001106:-]

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sbus_add_timeout]
(0x2000): 0x559bd20df680

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_dp_internal_get_send]
(0x0400): Entering request
[0x559bd1d69e70:1:1023001106@cyberloop.local]

    (Mon May  3 13:05:12 2021) [sssd[nss]]
[cache_req_search_ncache_filter] (0x0400): CR #114: Filtering out
results by negative cache

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_ncache_check_str]
(0x2000): Checking negative cache for
[NCE/USER/cyberloop.local/alice@cyberloop.local]

    (Mon May  3 13:05:12 2021) [sssd[nss]]
[cache_req_create_and_add_result] (0x0400): CR #114: Found 1 entries
in domain cyberloop.local

    (Mon May  3 13:05:12 2021) [sssd[nss]] [cache_req_done] (0x0400):
CR #114: Finished: Success

    (Mon May  3 13:05:12 2021) [sssd[nss]] [nss_protocol_done]
(0x4000): Sending reply: success

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sbus_remove_timeout]
(0x2000): 0x559bd20df680

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sbus_dispatch] (0x4000):
dbus conn: 0x559bd20d9230

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sbus_dispatch] (0x4000):
Dispatching.

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_dp_get_reply]
(0x1000): Got reply from Data Provider - DP error code: 0 errno: 0
error message: Success

    (Mon May  3 13:05:12 2021) [sssd[nss]] [cache_req_search_oob_done]
(0x2000): Out of band request finished

    (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_dp_req_destructor]
(0x0400): Deleting request:
[0x559bd1d69e70:1:1023001106@cyberloop.local]

    (Mon May  3 13:05:42 2021) [sssd[nss]] [setup_client_idle_timer]
(0x4000): Idle timer re-set for client [0x559bd20f19a0][21]

    (Mon May  3 13:06:12 2021) [sssd[nss]] [setup_client_idle_timer]
(0x4000): Idle timer re-set for client [0x559bd20f19a0][21]

    (Mon May  3 13:06:42 2021) [sssd[nss]] [client_idle_handler]
(0x2000): Terminating idle client [0x559bd20f19a0][21]

    (Mon May  3 13:06:42 2021) [sssd[nss]]



So I don’t quite see how !pw is actually matched here…



Calvin

Re: Unable to find pw entry for uid

[Aurélien Aptel]

Hi Calvin,

Calvin Chiang <calvin.chiang@gmail.com> writes:
> Now oddly the strerror(errno) is actually returning SUCCESS
> But the pw = getpwuid(uid); is failing.
> Getpwuid(uid) is calling nss.

...

> and the output from the sssd_nss.log is:

...

>     (Mon May  3 13:05:12 2021) [sssd[nss]]
> [cache_req_search_ncache_filter] (0x0400): CR #114: Filtering out
> results by negative cache
>
>     (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_ncache_check_str]
> (0x2000): Checking negative cache for
> [NCE/USER/cyberloop.local/alice@cyberloop.local]
>
>     (Mon May  3 13:05:12 2021) [sssd[nss]]
> [cache_req_create_and_add_result] (0x0400): CR #114: Found 1 entries
> in domain cyberloop.local

I don't know much about sssd but if it's finding 1 entry in the
*negative* cache that means it knows it doesn't exist. (I'm assuming
negative cache means cache of queries that have no results)

That being said I've had issue with getpwuid() before that were solved
by updating glibc. Could be totally unrelated to your problem though.

You should try to attach gdb to the cifs.upcall process. You can do this
by adding

    syslog(LOG_ERR, "my pid is %d", getpid());
    sleep(5);

in cifs.upcall.c before the getpwuid() call. Then try triggering the
mount. It should block because of the sleep(). In a different terminal,
look at your system journal for the PID, and run gdb -p $pid. From there
you will be able to step into the getpwuid (glibc), nss, sss, calls.

If your linux distribution has a debug symbol server setup for gdb to
use, gdb will dynamically fetch symbols and source code on the
fly. Otherwise you will probably need to install debug and source
packages for glibc, nss and sssd packages to be able to see function
names and source code in gdb.

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: Unable to find pw entry for uid

[Calvin Chiang]

Hey Aurelien

aweseom thanks for this!
>You should try to attach gdb to the cifs.upcall process

stepping through with a debugger was the "next level" i was thinking
of attempting but couldnt work out how to get GDB to hook into the
process.

one more newbie question:

I guess after i make the change in the cifs.upcall.c file i need to
autoreconf /config /make /make install ?
is it correct that this will overwrite all the files from the
cifs-utils package on my machine?

Cheers

On Thu, 6 May 2021 at 14:42, Aurélien Aptel <aaptel@suse.com> wrote:
>
> Hi Calvin,
>
> Calvin Chiang <calvin.chiang@gmail.com> writes:
> > Now oddly the strerror(errno) is actually returning SUCCESS
> > But the pw = getpwuid(uid); is failing.
> > Getpwuid(uid) is calling nss.
>
> ...
>
> > and the output from the sssd_nss.log is:
>
> ...
>
> >     (Mon May  3 13:05:12 2021) [sssd[nss]]
> > [cache_req_search_ncache_filter] (0x0400): CR #114: Filtering out
> > results by negative cache
> >
> >     (Mon May  3 13:05:12 2021) [sssd[nss]] [sss_ncache_check_str]
> > (0x2000): Checking negative cache for
> > [NCE/USER/cyberloop.local/alice@cyberloop.local]
> >
> >     (Mon May  3 13:05:12 2021) [sssd[nss]]
> > [cache_req_create_and_add_result] (0x0400): CR #114: Found 1 entries
> > in domain cyberloop.local
>
> I don't know much about sssd but if it's finding 1 entry in the
> *negative* cache that means it knows it doesn't exist. (I'm assuming
> negative cache means cache of queries that have no results)
>
> That being said I've had issue with getpwuid() before that were solved
> by updating glibc. Could be totally unrelated to your problem though.
>
> You should try to attach gdb to the cifs.upcall process. You can do this
> by adding
>
>     syslog(LOG_ERR, "my pid is %d", getpid());
>     sleep(5);
>
> in cifs.upcall.c before the getpwuid() call. Then try triggering the
> mount. It should block because of the sleep(). In a different terminal,
> look at your system journal for the PID, and run gdb -p $pid. From there
> you will be able to step into the getpwuid (glibc), nss, sss, calls.
>
> If your linux distribution has a debug symbol server setup for gdb to
> use, gdb will dynamically fetch symbols and source code on the
> fly. Otherwise you will probably need to install debug and source
> packages for glibc, nss and sssd packages to be able to see function
> names and source code in gdb.
>
> Cheers,
> --
> Aurélien Aptel / SUSE Labs Samba Team
> GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
>

Re: Unable to find pw entry for uid

[Aurélien Aptel]

Calvin Chiang <calvin.chiang@gmail.com> writes:
> I guess after i make the change in the cifs.upcall.c file i need to
> autoreconf /config /make /make install ?
> is it correct that this will overwrite all the files from the
> cifs-utils package on my machine?

Yes you need to make sure you have all the dependencies required to
build cifs.upcall (your package manager of your distro might provide a
way to get 'build dependencies' of a package). If it's missing some the
configure script might disable the build of cifs.upcall so make sure it
is built. You can run

as regular user:

   autoreconf -i
   ./configure
   make

Check where is installed your current cifs.upcall ($ whereis
cifs.upcall) usually it's in /usr/sbin/. I would recommend keeping a
copy of the old one.

as root (or sudo):

make backup once:

    cp /usr/sbin/cifs.upcall{,.backup}

then to build and use new one (rm is to make sure it is rebuilt):

     rm -f cifs.upcall && make && sudo cp cifs.upcall /usr/sbin/

Good luck

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Re: Unable to find pw entry for uid

[Calvin Chiang]

many thanks Aurelien for helping a newbie to get started!
just getting this setup now. i'll post back if i manage to find something.

On Thu, 6 May 2021 at 23:19, Aurélien Aptel <aaptel@suse.com> wrote:
>
> Calvin Chiang <calvin.chiang@gmail.com> writes:
> > I guess after i make the change in the cifs.upcall.c file i need to
> > autoreconf /config /make /make install ?
> > is it correct that this will overwrite all the files from the
> > cifs-utils package on my machine?
>
> Yes you need to make sure you have all the dependencies required to
> build cifs.upcall (your package manager of your distro might provide a
> way to get 'build dependencies' of a package). If it's missing some the
> configure script might disable the build of cifs.upcall so make sure it
> is built. You can run
>
> as regular user:
>
>    autoreconf -i
>    ./configure
>    make
>
> Check where is installed your current cifs.upcall ($ whereis
> cifs.upcall) usually it's in /usr/sbin/. I would recommend keeping a
> copy of the old one.
>
> as root (or sudo):
>
> make backup once:
>
>     cp /usr/sbin/cifs.upcall{,.backup}
>
> then to build and use new one (rm is to make sure it is rebuilt):
>
>      rm -f cifs.upcall && make && sudo cp cifs.upcall /usr/sbin/
>
> Good luck
>
> Cheers,
> --
> Aurélien Aptel / SUSE Labs Samba Team
> GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
>