* bmcweb 30 second lockout @ 2020-09-09 22:13 Neil Bradley 2020-09-09 23:20 ` Joseph Reynolds 0 siblings, 1 reply; 9+ messages in thread From: Neil Bradley @ 2020-09-09 22:13 UTC (permalink / raw) To: OpenBMC Maillist [-- Attachment #1: Type: text/plain, Size: 535 bytes --] I had recently read somewhere on the OpenBMC mailing list (forgive me, as I can't find it anywhere now) recently indicating that there'd be a 30 second lockout for a given user if there were 3 consecutive failed login attempts. My question is firstly, is this the case, and secondly, is it tied to the user globally regardless of connection or is it per user and connection? The reason I ask is that the former would still allow for a denial of service attack and want to make sure that's not actually the case. Thanks! -->Neil [-- Attachment #2: Type: text/html, Size: 2386 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: bmcweb 30 second lockout 2020-09-09 22:13 bmcweb 30 second lockout Neil Bradley @ 2020-09-09 23:20 ` Joseph Reynolds 2020-09-09 23:26 ` Neil Bradley 0 siblings, 1 reply; 9+ messages in thread From: Joseph Reynolds @ 2020-09-09 23:20 UTC (permalink / raw) To: Neil Bradley, OpenBMC Maillist On 9/9/20 5:13 PM, Neil Bradley wrote: > > I had recently read somewhere on the OpenBMC mailing list (forgive me, > as I can’t find it anywhere now) recently indicating that there’d be a > 30 second lockout for a given user if there were 3 consecutive failed > login attempts. My question is firstly, is this the case, and > secondly, is it tied to the user globally regardless of connection or > is it per user and connection? The reason I ask is that the former > would still allow for a denial of service attack and want to make sure > that’s not actually the case. > I can think of two items: 1. I had pushed an experimental gerrit code review to do what you described. Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 My idea has not gained much traction and is recorded here: https://github.com/ibm-openbmc/dev/issues/2434 and here: https://github.com/linux-pam/linux-pam/issues/216 and here: https://github.com/deksai/pam_abl/issues/4 and other places. If this ever gets merged, it would NOT the be default behavior. 2. The user lockouts for failed authentication attempts is handled by pam_tally2 and controlled by Redfish APIs. See https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-extended/pam/libpam/pam.d/common-auth In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts are never locked because of failed authentication attempts". The Redfish APIs are implemented here: https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account_service.hpp Specifically, PATCHing /redfish/v1/AccountService/ property AccountLockoutDurationor AccountLockoutThreshold invokes a D-Bus method which ultimately modifies the pam.d/common-auth config file above. Note that downstream projects may typically want to modify these default settings. 3. I don't think you mean this: There is a current code review for a BMCWeb enhancement to allow the BMC admin to control the idle session SessionTimeout property. The minimum is 30 seconds. See https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 - Joseph > Thanks! > > àNeil > ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: bmcweb 30 second lockout 2020-09-09 23:20 ` Joseph Reynolds @ 2020-09-09 23:26 ` Neil Bradley 2020-09-11 21:10 ` Joseph Reynolds 0 siblings, 1 reply; 9+ messages in thread From: Neil Bradley @ 2020-09-09 23:26 UTC (permalink / raw) To: Joseph Reynolds, OpenBMC Maillist Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. -->Neil -----Original Message----- From: Joseph Reynolds <jrey@linux.ibm.com> Sent: Wednesday, September 9, 2020 4:21 PM To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> Subject: Re: bmcweb 30 second lockout On 9/9/20 5:13 PM, Neil Bradley wrote: > > I had recently read somewhere on the OpenBMC mailing list (forgive me, > as I can't find it anywhere now) recently indicating that there'd be a > 30 second lockout for a given user if there were 3 consecutive failed > login attempts. My question is firstly, is this the case, and > secondly, is it tied to the user globally regardless of connection or > is it per user and connection? The reason I ask is that the former > would still allow for a denial of service attack and want to make sure > that's not actually the case. > I can think of two items: 1. I had pushed an experimental gerrit code review to do what you described. Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 My idea has not gained much traction and is recorded here: https://github.com/ibm-openbmc/dev/issues/2434 and here: https://github.com/linux-pam/linux-pam/issues/216 and here: https://github.com/deksai/pam_abl/issues/4 and other places. If this ever gets merged, it would NOT the be default behavior. 2. The user lockouts for failed authentication attempts is handled by pam_tally2 and controlled by Redfish APIs. See https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-extended/pam/libpam/pam.d/common-auth In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts are never locked because of failed authentication attempts". The Redfish APIs are implemented here: https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account_service.hpp Specifically, PATCHing /redfish/v1/AccountService/ property AccountLockoutDurationor AccountLockoutThreshold invokes a D-Bus method which ultimately modifies the pam.d/common-auth config file above. Note that downstream projects may typically want to modify these default settings. 3. I don't think you mean this: There is a current code review for a BMCWeb enhancement to allow the BMC admin to control the idle session SessionTimeout property. The minimum is 30 seconds. See https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 - Joseph > Thanks! > > àNeil > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: bmcweb 30 second lockout 2020-09-09 23:26 ` Neil Bradley @ 2020-09-11 21:10 ` Joseph Reynolds 2020-09-11 21:42 ` Neil Bradley 0 siblings, 1 reply; 9+ messages in thread From: Joseph Reynolds @ 2020-09-11 21:10 UTC (permalink / raw) To: Neil Bradley, OpenBMC Maillist On 9/9/20 6:26 PM, Neil Bradley wrote: > Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: > > " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." > > This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. Thanks Neil. Its you and me against the world. Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem? That is a difficult problem, and I think solving it will require defenses in multiple layers. I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below). I think a PAM authentication module like pam_abl could be a solution here. Would that work for you? But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them. My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt. Is this what you want? The approach you mentioned ("delay a few seconds when credentials are wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways: - When the attacker establishes multiple connections to the BMC. A defense against this might be limiting the number of connections. - When the attacker has a long time to attack, for example, a year or more. A defense might be password expiration. Which of these were you thinking? I would be happy to make progress on any of them. - Joseph > -->Neil > > > -----Original Message----- > From: Joseph Reynolds <jrey@linux.ibm.com> > Sent: Wednesday, September 9, 2020 4:21 PM > To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> > Subject: Re: bmcweb 30 second lockout > > On 9/9/20 5:13 PM, Neil Bradley wrote: >> I had recently read somewhere on the OpenBMC mailing list (forgive me, >> as I can't find it anywhere now) recently indicating that there'd be a >> 30 second lockout for a given user if there were 3 consecutive failed >> login attempts. My question is firstly, is this the case, and >> secondly, is it tied to the user globally regardless of connection or >> is it per user and connection? The reason I ask is that the former >> would still allow for a denial of service attack and want to make sure >> that's not actually the case. >> > I can think of two items: > > 1. I had pushed an experimental gerrit code review to do what you described. > Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 > My idea has not gained much traction and is recorded here: > https://github.com/ibm-openbmc/dev/issues/2434 > and here: https://github.com/linux-pam/linux-pam/issues/216 > and here: https://github.com/deksai/pam_abl/issues/4 > and other places. > If this ever gets merged, it would NOT the be default behavior. > > 2. The user lockouts for failed authentication attempts is handled by > pam_tally2 and controlled by Redfish APIs. > See > https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-extended/pam/libpam/pam.d/common-auth > In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts are > never locked because of failed authentication attempts". > > The Redfish APIs are implemented here: > https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account_service.hpp > Specifically, PATCHing /redfish/v1/AccountService/ property > AccountLockoutDurationor AccountLockoutThreshold invokes a D-Bus method > which ultimately modifies the pam.d/common-auth config file above. > > Note that downstream projects may typically want to modify these default > settings. > > 3. I don't think you mean this: There is a current code review for a > BMCWeb enhancement to allow the BMC admin to control the idle session > SessionTimeout property. The minimum is 30 seconds. See > https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 > > - Joseph > >> Thanks! >> >> àNeil >> ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: bmcweb 30 second lockout 2020-09-11 21:10 ` Joseph Reynolds @ 2020-09-11 21:42 ` Neil Bradley 2020-09-14 14:31 ` Joseph Reynolds 2020-09-14 14:55 ` Ed Tanous 0 siblings, 2 replies; 9+ messages in thread From: Neil Bradley @ 2020-09-11 21:42 UTC (permalink / raw) To: OpenBMC Maillist Hi Joseph! Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in. I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address. I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. 😉 Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away. Let me know what you think. -->Neil -----Original Message----- From: Joseph Reynolds <jrey@linux.ibm.com> Sent: Friday, September 11, 2020 2:11 PM To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> Subject: Re: bmcweb 30 second lockout On 9/9/20 6:26 PM, Neil Bradley wrote: > Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: > > " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." > > This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. Thanks Neil. Its you and me against the world. Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem? That is a difficult problem, and I think solving it will require defenses in multiple layers. I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below). I think a PAM authentication module like pam_abl could be a solution here. Would that work for you? But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them. My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt. Is this what you want? The approach you mentioned ("delay a few seconds when credentials are wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways: - When the attacker establishes multiple connections to the BMC. A defense against this might be limiting the number of connections. - When the attacker has a long time to attack, for example, a year or more. A defense might be password expiration. Which of these were you thinking? I would be happy to make progress on any of them. - Joseph > -->Neil > > > -----Original Message----- > From: Joseph Reynolds <jrey@linux.ibm.com> > Sent: Wednesday, September 9, 2020 4:21 PM > To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist > <openbmc@lists.ozlabs.org> > Subject: Re: bmcweb 30 second lockout > > On 9/9/20 5:13 PM, Neil Bradley wrote: >> I had recently read somewhere on the OpenBMC mailing list (forgive >> me, as I can't find it anywhere now) recently indicating that there'd >> be a >> 30 second lockout for a given user if there were 3 consecutive failed >> login attempts. My question is firstly, is this the case, and >> secondly, is it tied to the user globally regardless of connection or >> is it per user and connection? The reason I ask is that the former >> would still allow for a denial of service attack and want to make >> sure that's not actually the case. >> > I can think of two items: > > 1. I had pushed an experimental gerrit code review to do what you described. > Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 > My idea has not gained much traction and is recorded here: > https://github.com/ibm-openbmc/dev/issues/2434 > and here: https://github.com/linux-pam/linux-pam/issues/216 > and here: https://github.com/deksai/pam_abl/issues/4 > and other places. > If this ever gets merged, it would NOT the be default behavior. > > 2. The user lockouts for failed authentication attempts is handled by > pam_tally2 and controlled by Redfish APIs. > See > https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e > xtended/pam/libpam/pam.d/common-auth > In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts > are never locked because of failed authentication attempts". > > The Redfish APIs are implemented here: > https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account > _service.hpp Specifically, PATCHing /redfish/v1/AccountService/ > property AccountLockoutDurationor AccountLockoutThreshold invokes a > D-Bus method which ultimately modifies the pam.d/common-auth config > file above. > > Note that downstream projects may typically want to modify these > default settings. > > 3. I don't think you mean this: There is a current code review for a > BMCWeb enhancement to allow the BMC admin to control the idle session > SessionTimeout property. The minimum is 30 seconds. See > https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 > > - Joseph > >> Thanks! >> >> àNeil >> ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: bmcweb 30 second lockout 2020-09-11 21:42 ` Neil Bradley @ 2020-09-14 14:31 ` Joseph Reynolds 2020-09-14 14:55 ` Ed Tanous 1 sibling, 0 replies; 9+ messages in thread From: Joseph Reynolds @ 2020-09-14 14:31 UTC (permalink / raw) To: Neil Bradley, OpenBMC Maillist, Ed Tanous On 9/11/20 4:42 PM, Neil Bradley wrote: > Hi Joseph! > > Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in. > > I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address. > > I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. 😉 Neil, thanks for Cc: +Ed Tanous for BMCWeb. I created an issue on BMCWeb to track this specific approach: https://github.com/openbmc/bmcweb/issues/151 So I would support a design for BMCWeb to have a 2-4 second delay for every authentication failure. But I think it comes with problems of its own and I don't know if it is the right answer for everyone. Specifically, as I wrote in the issue/151 this solution gives attackers another way to DoS the machine. Also, what defense does it provide against an attacker who uses many multiple connections? Without additional defenses, such an attacker can theoretically try 1000 passwords in that 2-4 second time window. To be sure this is a more sophisticated attacker and the attack requires more system resources. What is the right way to defend against password-guessing attacks (short of going to 100% certificate-based auth)? - Joseph > Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away. > > Let me know what you think. > > -->Neil > > -----Original Message----- > From: Joseph Reynolds <jrey@linux.ibm.com> > Sent: Friday, September 11, 2020 2:11 PM > To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> > Subject: Re: bmcweb 30 second lockout > > On 9/9/20 6:26 PM, Neil Bradley wrote: >> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: >> >> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." >> >> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. > Thanks Neil. Its you and me against the world. > > Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem? That is a difficult problem, and I think solving it will require defenses in multiple layers. > > I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below). I think a PAM authentication module like pam_abl could be a solution here. Would that work for you? But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them. My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt. Is this what you want? > > The approach you mentioned ("delay a few seconds when credentials are > wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways: > - When the attacker establishes multiple connections to the BMC. A defense against this might be limiting the number of connections. > - When the attacker has a long time to attack, for example, a year or more. A defense might be password expiration. > > Which of these were you thinking? I would be happy to make progress on any of them. > > - Joseph > >> -->Neil >> >> >> -----Original Message----- >> From: Joseph Reynolds <jrey@linux.ibm.com> >> Sent: Wednesday, September 9, 2020 4:21 PM >> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist >> <openbmc@lists.ozlabs.org> >> Subject: Re: bmcweb 30 second lockout >> >> On 9/9/20 5:13 PM, Neil Bradley wrote: >>> I had recently read somewhere on the OpenBMC mailing list (forgive >>> me, as I can't find it anywhere now) recently indicating that there'd >>> be a >>> 30 second lockout for a given user if there were 3 consecutive failed >>> login attempts. My question is firstly, is this the case, and >>> secondly, is it tied to the user globally regardless of connection or >>> is it per user and connection? The reason I ask is that the former >>> would still allow for a denial of service attack and want to make >>> sure that's not actually the case. >>> >> I can think of two items: >> >> 1. I had pushed an experimental gerrit code review to do what you described. >> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 >> My idea has not gained much traction and is recorded here: >> https://github.com/ibm-openbmc/dev/issues/2434 >> and here: https://github.com/linux-pam/linux-pam/issues/216 >> and here: https://github.com/deksai/pam_abl/issues/4 >> and other places. >> If this ever gets merged, it would NOT the be default behavior. >> >> 2. The user lockouts for failed authentication attempts is handled by >> pam_tally2 and controlled by Redfish APIs. >> See >> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e >> xtended/pam/libpam/pam.d/common-auth >> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts >> are never locked because of failed authentication attempts". >> >> The Redfish APIs are implemented here: >> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account >> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/ >> property AccountLockoutDurationor AccountLockoutThreshold invokes a >> D-Bus method which ultimately modifies the pam.d/common-auth config >> file above. >> >> Note that downstream projects may typically want to modify these >> default settings. >> >> 3. I don't think you mean this: There is a current code review for a >> BMCWeb enhancement to allow the BMC admin to control the idle session >> SessionTimeout property. The minimum is 30 seconds. See >> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 >> >> - Joseph >> >>> Thanks! >>> >>> àNeil >>> ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: bmcweb 30 second lockout 2020-09-11 21:42 ` Neil Bradley 2020-09-14 14:31 ` Joseph Reynolds @ 2020-09-14 14:55 ` Ed Tanous 2020-09-15 16:14 ` Joseph Reynolds 1 sibling, 1 reply; 9+ messages in thread From: Ed Tanous @ 2020-09-14 14:55 UTC (permalink / raw) To: Neil Bradley; +Cc: OpenBMC Maillist On Fri, Sep 11, 2020 at 2:43 PM Neil Bradley <Neil_Bradley@phoenix.com> wrote: > > Hi Joseph! > > Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in. > > I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address. > > I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. This type of approach works great in an apache-type server, where we have thread per connection, and we can rely on running out of threads to slow us down. For better or worse, bmcweb is an async reactor model (a lot like nginx), so while a 2-4 second delay would slow that that specific connection, all an attacker would have to do is to spin up N connections in parallel to bypass this per-connection limit. For what it's worth, I think something similar to the fail2ban approach would work well here, where after a number of failures, we lock out specific IPs for a specific amount of time at the networking layer using something like nftables. Unfortunately, fail2ban is python, so that's a non-starter but there's a similar "copy" of it here that I've been meaning to play with at some point: https://github.com/swillner/regban (credit goes to James F for finding it). > > > Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away. > > > Let me know what you think. > > -->Neil > > -----Original Message----- > From: Joseph Reynolds <jrey@linux.ibm.com> > Sent: Friday, September 11, 2020 2:11 PM > To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> > Subject: Re: bmcweb 30 second lockout > > On 9/9/20 6:26 PM, Neil Bradley wrote: > > Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: > > > > " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." > > > > This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. > > Thanks Neil. Its you and me against the world. > > Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem? That is a difficult problem, and I think solving it will require defenses in multiple layers. > > I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below). I think a PAM authentication module like pam_abl could be a solution here. Would that work for you? But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them. My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt. Is this what you want? > > The approach you mentioned ("delay a few seconds when credentials are > wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways: > - When the attacker establishes multiple connections to the BMC. A defense against this might be limiting the number of connections. > - When the attacker has a long time to attack, for example, a year or more. A defense might be password expiration. > > Which of these were you thinking? I would be happy to make progress on any of them. > > - Joseph > > > -->Neil > > > > > > -----Original Message----- > > From: Joseph Reynolds <jrey@linux.ibm.com> > > Sent: Wednesday, September 9, 2020 4:21 PM > > To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist > > <openbmc@lists.ozlabs.org> > > Subject: Re: bmcweb 30 second lockout > > > > On 9/9/20 5:13 PM, Neil Bradley wrote: > >> I had recently read somewhere on the OpenBMC mailing list (forgive > >> me, as I can't find it anywhere now) recently indicating that there'd > >> be a > >> 30 second lockout for a given user if there were 3 consecutive failed > >> login attempts. My question is firstly, is this the case, and > >> secondly, is it tied to the user globally regardless of connection or > >> is it per user and connection? The reason I ask is that the former > >> would still allow for a denial of service attack and want to make > >> sure that's not actually the case. > >> > > I can think of two items: > > > > 1. I had pushed an experimental gerrit code review to do what you described. > > Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 > > My idea has not gained much traction and is recorded here: > > https://github.com/ibm-openbmc/dev/issues/2434 > > and here: https://github.com/linux-pam/linux-pam/issues/216 > > and here: https://github.com/deksai/pam_abl/issues/4 > > and other places. > > If this ever gets merged, it would NOT the be default behavior. > > > > 2. The user lockouts for failed authentication attempts is handled by > > pam_tally2 and controlled by Redfish APIs. > > See > > https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e > > xtended/pam/libpam/pam.d/common-auth > > In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts > > are never locked because of failed authentication attempts". > > > > The Redfish APIs are implemented here: > > https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account > > _service.hpp Specifically, PATCHing /redfish/v1/AccountService/ > > property AccountLockoutDurationor AccountLockoutThreshold invokes a > > D-Bus method which ultimately modifies the pam.d/common-auth config > > file above. > > > > Note that downstream projects may typically want to modify these > > default settings. > > > > 3. I don't think you mean this: There is a current code review for a > > BMCWeb enhancement to allow the BMC admin to control the idle session > > SessionTimeout property. The minimum is 30 seconds. See > > https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 > > > > - Joseph > > > >> Thanks! > >> > >> àNeil > >> > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: bmcweb 30 second lockout 2020-09-14 14:55 ` Ed Tanous @ 2020-09-15 16:14 ` Joseph Reynolds 2020-09-15 16:20 ` Ed Tanous 0 siblings, 1 reply; 9+ messages in thread From: Joseph Reynolds @ 2020-09-15 16:14 UTC (permalink / raw) To: Ed Tanous, Neil Bradley; +Cc: OpenBMC Maillist On 9/14/20 9:55 AM, Ed Tanous wrote: > On Fri, Sep 11, 2020 at 2:43 PM Neil Bradley <Neil_Bradley@phoenix.com> wrote: >> Hi Joseph! >> >> Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in. >> >> I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address. >> >> I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. > > This type of approach works great in an apache-type server, where we > have thread per connection, and we can rely on running out of threads > to slow us down. For better or worse, bmcweb is an async reactor > model (a lot like nginx), so while a 2-4 second delay would slow that > that specific connection, all an attacker would have to do is to spin > up N connections in parallel to bypass this per-connection limit. Ed, thanks. Would the solution above help if BMCWeb also limited the number of simultaneous connections, for example N=32. Then the attacker can guess only 32 password every few seconds. But what are the consequences? During an attack, can a legitimate user compete for one of these connections? I am summarizing the results of this email conversation here: https://github.com/openbmc/bmcweb/issues/151 - Joseph > > For what it's worth, I think something similar to the fail2ban > approach would work well here, where after a number of failures, we > lock out specific IPs for a specific amount of time at the networking > layer using something like nftables. Unfortunately, fail2ban is > python, so that's a non-starter but there's a similar "copy" of it > here that I've been meaning to play with at some point: > https://github.com/swillner/regban > (credit goes to James F for finding it). For what its worth, I think fail2ban or regban could be part of a solution. >> Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away. >> >> >> Let me know what you think. >> >> -->Neil >> >> -----Original Message----- >> From: Joseph Reynolds <jrey@linux.ibm.com> >> Sent: Friday, September 11, 2020 2:11 PM >> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> >> Subject: Re: bmcweb 30 second lockout >> >> On 9/9/20 6:26 PM, Neil Bradley wrote: >>> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: >>> >>> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." >>> >>> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. >> Thanks Neil. Its you and me against the world. >> >> Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem? That is a difficult problem, and I think solving it will require defenses in multiple layers. >> >> I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below). I think a PAM authentication module like pam_abl could be a solution here. Would that work for you? But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them. My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt. Is this what you want? >> >> The approach you mentioned ("delay a few seconds when credentials are >> wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways: >> - When the attacker establishes multiple connections to the BMC. A defense against this might be limiting the number of connections. >> - When the attacker has a long time to attack, for example, a year or more. A defense might be password expiration. >> >> Which of these were you thinking? I would be happy to make progress on any of them. >> >> - Joseph >> >>> -->Neil >>> >>> >>> -----Original Message----- >>> From: Joseph Reynolds <jrey@linux.ibm.com> >>> Sent: Wednesday, September 9, 2020 4:21 PM >>> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist >>> <openbmc@lists.ozlabs.org> >>> Subject: Re: bmcweb 30 second lockout >>> >>> On 9/9/20 5:13 PM, Neil Bradley wrote: >>>> I had recently read somewhere on the OpenBMC mailing list (forgive >>>> me, as I can't find it anywhere now) recently indicating that there'd >>>> be a >>>> 30 second lockout for a given user if there were 3 consecutive failed >>>> login attempts. My question is firstly, is this the case, and >>>> secondly, is it tied to the user globally regardless of connection or >>>> is it per user and connection? The reason I ask is that the former >>>> would still allow for a denial of service attack and want to make >>>> sure that's not actually the case. >>>> >>> I can think of two items: >>> >>> 1. I had pushed an experimental gerrit code review to do what you described. >>> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 >>> My idea has not gained much traction and is recorded here: >>> https://github.com/ibm-openbmc/dev/issues/2434 >>> and here: https://github.com/linux-pam/linux-pam/issues/216 >>> and here: https://github.com/deksai/pam_abl/issues/4 >>> and other places. >>> If this ever gets merged, it would NOT the be default behavior. >>> >>> 2. The user lockouts for failed authentication attempts is handled by >>> pam_tally2 and controlled by Redfish APIs. >>> See >>> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e >>> xtended/pam/libpam/pam.d/common-auth >>> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts >>> are never locked because of failed authentication attempts". >>> >>> The Redfish APIs are implemented here: >>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account >>> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/ >>> property AccountLockoutDurationor AccountLockoutThreshold invokes a >>> D-Bus method which ultimately modifies the pam.d/common-auth config >>> file above. >>> >>> Note that downstream projects may typically want to modify these >>> default settings. >>> >>> 3. I don't think you mean this: There is a current code review for a >>> BMCWeb enhancement to allow the BMC admin to control the idle session >>> SessionTimeout property. The minimum is 30 seconds. See >>> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 >>> >>> - Joseph >>> >>>> Thanks! >>>> >>>> àNeil >>>> ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: bmcweb 30 second lockout 2020-09-15 16:14 ` Joseph Reynolds @ 2020-09-15 16:20 ` Ed Tanous 0 siblings, 0 replies; 9+ messages in thread From: Ed Tanous @ 2020-09-15 16:20 UTC (permalink / raw) To: Joseph Reynolds; +Cc: Neil Bradley, OpenBMC Maillist On Tue, Sep 15, 2020 at 9:15 AM Joseph Reynolds <jrey@linux.ibm.com> wrote: > > On 9/14/20 9:55 AM, Ed Tanous wrote: > > On Fri, Sep 11, 2020 at 2:43 PM Neil Bradley <Neil_Bradley@phoenix.com> wrote: > >> Hi Joseph! > >> > >> Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in. > >> > >> I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address. > >> > >> I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. > > > > This type of approach works great in an apache-type server, where we > > have thread per connection, and we can rely on running out of threads > > to slow us down. For better or worse, bmcweb is an async reactor > > model (a lot like nginx), so while a 2-4 second delay would slow that > > that specific connection, all an attacker would have to do is to spin > > up N connections in parallel to bypass this per-connection limit. > > Ed, thanks. Would the solution above help if BMCWeb also limited the > number of simultaneous connections, for example N=32. Then the attacker > can guess only 32 password every few seconds. But what are the > consequences? During an attack, can a legitimate user compete for one > of these connections? They can compete, but if the connections are slow and long lived, a user is unlikely to succeed. With that said, it's easy enough to run the bmc out of bandwidth and cause a DOS attack that way, so we're not really any worse off. With that said, I'd still much rather a per-ip lockout. > > I am summarizing the results of this email conversation here: > https://github.com/openbmc/bmcweb/issues/151 > > - Joseph > > > > > For what it's worth, I think something similar to the fail2ban > > approach would work well here, where after a number of failures, we > > lock out specific IPs for a specific amount of time at the networking > > layer using something like nftables. Unfortunately, fail2ban is > > python, so that's a non-starter but there's a similar "copy" of it > > here that I've been meaning to play with at some point: > > https://github.com/swillner/regban > > (credit goes to James F for finding it). > > For what its worth, I think fail2ban or regban could be part of a solution. > > >> Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away. > >> > >> > >> Let me know what you think. > >> > >> -->Neil > >> > >> -----Original Message----- > >> From: Joseph Reynolds <jrey@linux.ibm.com> > >> Sent: Friday, September 11, 2020 2:11 PM > >> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org> > >> Subject: Re: bmcweb 30 second lockout > >> > >> On 9/9/20 6:26 PM, Neil Bradley wrote: > >>> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here: > >>> > >>> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts." > >>> > >>> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective. > >> Thanks Neil. Its you and me against the world. > >> > >> Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem? That is a difficult problem, and I think solving it will require defenses in multiple layers. > >> > >> I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below). I think a PAM authentication module like pam_abl could be a solution here. Would that work for you? But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them. My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt. Is this what you want? > >> > >> The approach you mentioned ("delay a few seconds when credentials are > >> wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways: > >> - When the attacker establishes multiple connections to the BMC. A defense against this might be limiting the number of connections. > >> - When the attacker has a long time to attack, for example, a year or more. A defense might be password expiration. > >> > >> Which of these were you thinking? I would be happy to make progress on any of them. > >> > >> - Joseph > >> > >>> -->Neil > >>> > >>> > >>> -----Original Message----- > >>> From: Joseph Reynolds <jrey@linux.ibm.com> > >>> Sent: Wednesday, September 9, 2020 4:21 PM > >>> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist > >>> <openbmc@lists.ozlabs.org> > >>> Subject: Re: bmcweb 30 second lockout > >>> > >>> On 9/9/20 5:13 PM, Neil Bradley wrote: > >>>> I had recently read somewhere on the OpenBMC mailing list (forgive > >>>> me, as I can't find it anywhere now) recently indicating that there'd > >>>> be a > >>>> 30 second lockout for a given user if there were 3 consecutive failed > >>>> login attempts. My question is firstly, is this the case, and > >>>> secondly, is it tied to the user globally regardless of connection or > >>>> is it per user and connection? The reason I ask is that the former > >>>> would still allow for a denial of service attack and want to make > >>>> sure that's not actually the case. > >>>> > >>> I can think of two items: > >>> > >>> 1. I had pushed an experimental gerrit code review to do what you described. > >>> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841 > >>> My idea has not gained much traction and is recorded here: > >>> https://github.com/ibm-openbmc/dev/issues/2434 > >>> and here: https://github.com/linux-pam/linux-pam/issues/216 > >>> and here: https://github.com/deksai/pam_abl/issues/4 > >>> and other places. > >>> If this ever gets merged, it would NOT the be default behavior. > >>> > >>> 2. The user lockouts for failed authentication attempts is handled by > >>> pam_tally2 and controlled by Redfish APIs. > >>> See > >>> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e > >>> xtended/pam/libpam/pam.d/common-auth > >>> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts > >>> are never locked because of failed authentication attempts". > >>> > >>> The Redfish APIs are implemented here: > >>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account > >>> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/ > >>> property AccountLockoutDurationor AccountLockoutThreshold invokes a > >>> D-Bus method which ultimately modifies the pam.d/common-auth config > >>> file above. > >>> > >>> Note that downstream projects may typically want to modify these > >>> default settings. > >>> > >>> 3. I don't think you mean this: There is a current code review for a > >>> BMCWeb enhancement to allow the BMC admin to control the idle session > >>> SessionTimeout property. The minimum is 30 seconds. See > >>> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016 > >>> > >>> - Joseph > >>> > >>>> Thanks! > >>>> > >>>> àNeil > >>>> > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-09-15 16:20 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-09-09 22:13 bmcweb 30 second lockout Neil Bradley 2020-09-09 23:20 ` Joseph Reynolds 2020-09-09 23:26 ` Neil Bradley 2020-09-11 21:10 ` Joseph Reynolds 2020-09-11 21:42 ` Neil Bradley 2020-09-14 14:31 ` Joseph Reynolds 2020-09-14 14:55 ` Ed Tanous 2020-09-15 16:14 ` Joseph Reynolds 2020-09-15 16:20 ` Ed Tanous
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.