bmcweb 30 second lockout

bmcweb 30 second lockout

[Neil Bradley]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

I had recently read somewhere on the OpenBMC mailing list (forgive me, as I can't find it anywhere now) recently indicating that there'd be a 30 second lockout for a given user if there were 3 consecutive failed login attempts. My question is firstly, is this the case, and secondly, is it tied to the user globally regardless of connection or is it per user and connection? The reason I ask is that the former would still allow for a denial of service attack and want to make sure that's not actually the case.

Thanks!

-->Neil

[-- Attachment #2: Type: text/html, Size: 2386 bytes --]

Re: bmcweb 30 second lockout

[Joseph Reynolds]

On 9/9/20 5:13 PM, Neil Bradley wrote:
>
> I had recently read somewhere on the OpenBMC mailing list (forgive me, 
> as I can’t find it anywhere now) recently indicating that there’d be a 
> 30 second lockout for a given user if there were 3 consecutive failed 
> login attempts. My question is firstly, is this the case, and 
> secondly, is it tied to the user globally regardless of connection or 
> is it per user and connection? The reason I ask is that the former 
> would still allow for a denial of service attack and want to make sure 
> that’s not actually the case.
>

I can think of two items:

1. I had pushed an experimental gerrit code review to do what you described.
Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
My idea has not gained much traction and is recorded here: 
https://github.com/ibm-openbmc/dev/issues/2434
and here: https://github.com/linux-pam/linux-pam/issues/216
and here: https://github.com/deksai/pam_abl/issues/4
and other places.
If this ever gets merged, it would NOT the be default behavior.

2. The user lockouts for failed authentication attempts is handled by 
pam_tally2 and controlled by Redfish APIs.
See 
https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-extended/pam/libpam/pam.d/common-auth
In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts are 
never locked because of failed authentication attempts".

The Redfish APIs are implemented here: 
https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account_service.hpp
Specifically, PATCHing /redfish/v1/AccountService/ property 
AccountLockoutDurationor AccountLockoutThreshold invokes a D-Bus method 
which ultimately modifies the pam.d/common-auth config file above.

Note that downstream projects may typically want to modify these default 
settings.

3. I don't think you mean this: There is a current code review for a 
BMCWeb enhancement to allow the BMC admin to control the idle session 
SessionTimeout property.  The minimum is 30 seconds. See 
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016

- Joseph

> Thanks!
>
> àNeil
>

RE: bmcweb 30 second lockout

[Neil Bradley]

Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:

" This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."

This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.

-->Neil


-----Original Message-----
From: Joseph Reynolds <jrey@linux.ibm.com> 
Sent: Wednesday, September 9, 2020 4:21 PM
To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: bmcweb 30 second lockout

On 9/9/20 5:13 PM, Neil Bradley wrote:
>
> I had recently read somewhere on the OpenBMC mailing list (forgive me, 
> as I can't find it anywhere now) recently indicating that there'd be a
> 30 second lockout for a given user if there were 3 consecutive failed 
> login attempts. My question is firstly, is this the case, and 
> secondly, is it tied to the user globally regardless of connection or 
> is it per user and connection? The reason I ask is that the former 
> would still allow for a denial of service attack and want to make sure 
> that's not actually the case.
>

I can think of two items:

1. I had pushed an experimental gerrit code review to do what you described.
Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
My idea has not gained much traction and is recorded here: 
https://github.com/ibm-openbmc/dev/issues/2434
and here: https://github.com/linux-pam/linux-pam/issues/216
and here: https://github.com/deksai/pam_abl/issues/4
and other places.
If this ever gets merged, it would NOT the be default behavior.

2. The user lockouts for failed authentication attempts is handled by 
pam_tally2 and controlled by Redfish APIs.
See 
https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-extended/pam/libpam/pam.d/common-auth
In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts are 
never locked because of failed authentication attempts".

The Redfish APIs are implemented here: 
https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account_service.hpp
Specifically, PATCHing /redfish/v1/AccountService/ property 
AccountLockoutDurationor AccountLockoutThreshold invokes a D-Bus method 
which ultimately modifies the pam.d/common-auth config file above.

Note that downstream projects may typically want to modify these default 
settings.

3. I don't think you mean this: There is a current code review for a 
BMCWeb enhancement to allow the BMC admin to control the idle session 
SessionTimeout property.  The minimum is 30 seconds. See 
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016

- Joseph

> Thanks!
>
> àNeil
>

Re: bmcweb 30 second lockout

[Joseph Reynolds]

On 9/9/20 6:26 PM, Neil Bradley wrote:
> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:
>
> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."
>
> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.

Thanks Neil.  Its you and me against the world.

Let's be clear about the problem we want to solve: stop brute force 
password-guessing attacks but not deny service to legitimate users. Is 
that also your problem?  That is a difficult problem, and I think 
solving it will require defenses in multiple layers.

I believe the most promising solution is limiting the number of 
authentication attempts after too many failures (item 1 in the email 
below).  I think a PAM authentication module like pam_abl could be a 
solution here.  Would that work for you?  But I want to go a step 
farther and actually notify the person logging in when their attempt was 
rejected because authentication rate-limiting is in effect. Right now 
they get an "authentication failure" message which leads them to believe 
their password was wrong, or their account is missing, and that confuses 
them.  My issues with linux-pam and pam_abl are to help provide clarity 
to the user what happeded to their login attempt.  Is this what you want?

The approach you mentioned ("delay a few seconds when credentials are 
wrong") defends against a single-threaded attacker (or a misbehaving 
tool), but it breaks down in (at least) two ways:
- When the attacker establishes multiple connections to the BMC.  A 
defense against this might be limiting the number of connections.
- When the attacker has a long time to attack, for example, a year or 
more.  A defense might be password expiration.

Which of these were you thinking?  I would be happy to make progress on 
any of them.

- Joseph

> -->Neil
>
>
> -----Original Message-----
> From: Joseph Reynolds <jrey@linux.ibm.com>
> Sent: Wednesday, September 9, 2020 4:21 PM
> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
> Subject: Re: bmcweb 30 second lockout
>
> On 9/9/20 5:13 PM, Neil Bradley wrote:
>> I had recently read somewhere on the OpenBMC mailing list (forgive me,
>> as I can't find it anywhere now) recently indicating that there'd be a
>> 30 second lockout for a given user if there were 3 consecutive failed
>> login attempts. My question is firstly, is this the case, and
>> secondly, is it tied to the user globally regardless of connection or
>> is it per user and connection? The reason I ask is that the former
>> would still allow for a denial of service attack and want to make sure
>> that's not actually the case.
>>
> I can think of two items:
>
> 1. I had pushed an experimental gerrit code review to do what you described.
> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
> My idea has not gained much traction and is recorded here:
> https://github.com/ibm-openbmc/dev/issues/2434
> and here: https://github.com/linux-pam/linux-pam/issues/216
> and here: https://github.com/deksai/pam_abl/issues/4
> and other places.
> If this ever gets merged, it would NOT the be default behavior.
>
> 2. The user lockouts for failed authentication attempts is handled by
> pam_tally2 and controlled by Redfish APIs.
> See
> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-extended/pam/libpam/pam.d/common-auth
> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts are
> never locked because of failed authentication attempts".
>
> The Redfish APIs are implemented here:
> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account_service.hpp
> Specifically, PATCHing /redfish/v1/AccountService/ property
> AccountLockoutDurationor AccountLockoutThreshold invokes a D-Bus method
> which ultimately modifies the pam.d/common-auth config file above.
>
> Note that downstream projects may typically want to modify these default
> settings.
>
> 3. I don't think you mean this: There is a current code review for a
> BMCWeb enhancement to allow the BMC admin to control the idle session
> SessionTimeout property.  The minimum is 30 seconds. See
> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016
>
> - Joseph
>
>> Thanks!
>>
>> àNeil
>>

RE: bmcweb 30 second lockout

[Neil Bradley]

Hi Joseph!

Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in. 

I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address. 

I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. 😉

Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away.

Let me know what you think.

-->Neil

-----Original Message-----
From: Joseph Reynolds <jrey@linux.ibm.com> 
Sent: Friday, September 11, 2020 2:11 PM
To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
Subject: Re: bmcweb 30 second lockout

On 9/9/20 6:26 PM, Neil Bradley wrote:
> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:
>
> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."
>
> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.

Thanks Neil.  Its you and me against the world.

Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem?  That is a difficult problem, and I think solving it will require defenses in multiple layers.

I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below).  I think a PAM authentication module like pam_abl could be a solution here.  Would that work for you?  But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them.  My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt.  Is this what you want?

The approach you mentioned ("delay a few seconds when credentials are
wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways:
- When the attacker establishes multiple connections to the BMC.  A defense against this might be limiting the number of connections.
- When the attacker has a long time to attack, for example, a year or more.  A defense might be password expiration.

Which of these were you thinking?  I would be happy to make progress on any of them.

- Joseph

> -->Neil
>
>
> -----Original Message-----
> From: Joseph Reynolds <jrey@linux.ibm.com>
> Sent: Wednesday, September 9, 2020 4:21 PM
> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist 
> <openbmc@lists.ozlabs.org>
> Subject: Re: bmcweb 30 second lockout
>
> On 9/9/20 5:13 PM, Neil Bradley wrote:
>> I had recently read somewhere on the OpenBMC mailing list (forgive 
>> me, as I can't find it anywhere now) recently indicating that there'd 
>> be a
>> 30 second lockout for a given user if there were 3 consecutive failed 
>> login attempts. My question is firstly, is this the case, and 
>> secondly, is it tied to the user globally regardless of connection or 
>> is it per user and connection? The reason I ask is that the former 
>> would still allow for a denial of service attack and want to make 
>> sure that's not actually the case.
>>
> I can think of two items:
>
> 1. I had pushed an experimental gerrit code review to do what you described.
> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
> My idea has not gained much traction and is recorded here:
> https://github.com/ibm-openbmc/dev/issues/2434
> and here: https://github.com/linux-pam/linux-pam/issues/216
> and here: https://github.com/deksai/pam_abl/issues/4
> and other places.
> If this ever gets merged, it would NOT the be default behavior.
>
> 2. The user lockouts for failed authentication attempts is handled by
> pam_tally2 and controlled by Redfish APIs.
> See
> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e
> xtended/pam/libpam/pam.d/common-auth
> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts 
> are never locked because of failed authentication attempts".
>
> The Redfish APIs are implemented here:
> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account
> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/ 
> property AccountLockoutDurationor AccountLockoutThreshold invokes a 
> D-Bus method which ultimately modifies the pam.d/common-auth config 
> file above.
>
> Note that downstream projects may typically want to modify these 
> default settings.
>
> 3. I don't think you mean this: There is a current code review for a 
> BMCWeb enhancement to allow the BMC admin to control the idle session 
> SessionTimeout property.  The minimum is 30 seconds. See
> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016
>
> - Joseph
>
>> Thanks!
>>
>> àNeil
>>

Re: bmcweb 30 second lockout

[Joseph Reynolds]

On 9/11/20 4:42 PM, Neil Bradley wrote:
> Hi Joseph!
>
> Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in.
>
> I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address.
>
> I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it. 😉
Neil, thanks for

Cc: +Ed Tanous for BMCWeb.  I created an issue on BMCWeb to track this 
specific approach: https://github.com/openbmc/bmcweb/issues/151

So I would support a design for BMCWeb to have a 2-4 second delay for 
every authentication failure.  But I think it comes with problems of its 
own and I don't know if it is the right answer for everyone.

Specifically, as I wrote in the issue/151 this solution gives attackers 
another way to DoS the machine.
Also, what defense does it provide against an attacker who uses many 
multiple connections?  Without additional defenses, such an attacker can 
theoretically try 1000 passwords in that 2-4 second time window.  To be 
sure this is a more sophisticated attacker and the attack requires more 
system resources.

What is the right way to defend against password-guessing attacks (short 
of going to 100% certificate-based auth)?

- Joseph

> Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away.
>
> Let me know what you think.
>
> -->Neil
>
> -----Original Message-----
> From: Joseph Reynolds <jrey@linux.ibm.com>
> Sent: Friday, September 11, 2020 2:11 PM
> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
> Subject: Re: bmcweb 30 second lockout
>
> On 9/9/20 6:26 PM, Neil Bradley wrote:
>> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:
>>
>> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."
>>
>> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.
> Thanks Neil.  Its you and me against the world.
>
> Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem?  That is a difficult problem, and I think solving it will require defenses in multiple layers.
>
> I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below).  I think a PAM authentication module like pam_abl could be a solution here.  Would that work for you?  But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them.  My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt.  Is this what you want?
>
> The approach you mentioned ("delay a few seconds when credentials are
> wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways:
> - When the attacker establishes multiple connections to the BMC.  A defense against this might be limiting the number of connections.
> - When the attacker has a long time to attack, for example, a year or more.  A defense might be password expiration.
>
> Which of these were you thinking?  I would be happy to make progress on any of them.
>
> - Joseph
>
>> -->Neil
>>
>>
>> -----Original Message-----
>> From: Joseph Reynolds <jrey@linux.ibm.com>
>> Sent: Wednesday, September 9, 2020 4:21 PM
>> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist
>> <openbmc@lists.ozlabs.org>
>> Subject: Re: bmcweb 30 second lockout
>>
>> On 9/9/20 5:13 PM, Neil Bradley wrote:
>>> I had recently read somewhere on the OpenBMC mailing list (forgive
>>> me, as I can't find it anywhere now) recently indicating that there'd
>>> be a
>>> 30 second lockout for a given user if there were 3 consecutive failed
>>> login attempts. My question is firstly, is this the case, and
>>> secondly, is it tied to the user globally regardless of connection or
>>> is it per user and connection? The reason I ask is that the former
>>> would still allow for a denial of service attack and want to make
>>> sure that's not actually the case.
>>>
>> I can think of two items:
>>
>> 1. I had pushed an experimental gerrit code review to do what you described.
>> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
>> My idea has not gained much traction and is recorded here:
>> https://github.com/ibm-openbmc/dev/issues/2434
>> and here: https://github.com/linux-pam/linux-pam/issues/216
>> and here: https://github.com/deksai/pam_abl/issues/4
>> and other places.
>> If this ever gets merged, it would NOT the be default behavior.
>>
>> 2. The user lockouts for failed authentication attempts is handled by
>> pam_tally2 and controlled by Redfish APIs.
>> See
>> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e
>> xtended/pam/libpam/pam.d/common-auth
>> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts
>> are never locked because of failed authentication attempts".
>>
>> The Redfish APIs are implemented here:
>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account
>> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/
>> property AccountLockoutDurationor AccountLockoutThreshold invokes a
>> D-Bus method which ultimately modifies the pam.d/common-auth config
>> file above.
>>
>> Note that downstream projects may typically want to modify these
>> default settings.
>>
>> 3. I don't think you mean this: There is a current code review for a
>> BMCWeb enhancement to allow the BMC admin to control the idle session
>> SessionTimeout property.  The minimum is 30 seconds. See
>> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016
>>
>> - Joseph
>>
>>> Thanks!
>>>
>>> àNeil
>>>

Re: bmcweb 30 second lockout

[Ed Tanous]

On Fri, Sep 11, 2020 at 2:43 PM Neil Bradley <Neil_Bradley@phoenix.com> wrote:
>
> Hi Joseph!
>
> Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in.
>
> I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address.
>
> I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it.


This type of approach works great in an apache-type server, where we
have thread per connection, and we can rely on running out of threads
to slow us down.  For better or worse, bmcweb is an async reactor
model (a lot like nginx), so while a 2-4 second delay would slow that
that specific connection, all an attacker would have to do is to spin
up N connections in parallel to bypass this per-connection limit.

For what it's worth, I think something similar to the fail2ban
approach would work well here, where after a number of failures, we
lock out specific IPs for a specific amount of time at the networking
layer using something like nftables.  Unfortunately, fail2ban is
python, so that's a non-starter but there's a similar "copy" of it
here that I've been meaning to play with at some point:
https://github.com/swillner/regban
(credit goes to James F for finding it).

>
>
> Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away.
>
>
> Let me know what you think.
>
> -->Neil
>
> -----Original Message-----
> From: Joseph Reynolds <jrey@linux.ibm.com>
> Sent: Friday, September 11, 2020 2:11 PM
> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
> Subject: Re: bmcweb 30 second lockout
>
> On 9/9/20 6:26 PM, Neil Bradley wrote:
> > Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:
> >
> > " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."
> >
> > This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.
>
> Thanks Neil.  Its you and me against the world.
>
> Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem?  That is a difficult problem, and I think solving it will require defenses in multiple layers.
>
> I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below).  I think a PAM authentication module like pam_abl could be a solution here.  Would that work for you?  But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them.  My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt.  Is this what you want?
>
> The approach you mentioned ("delay a few seconds when credentials are
> wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways:
> - When the attacker establishes multiple connections to the BMC.  A defense against this might be limiting the number of connections.
> - When the attacker has a long time to attack, for example, a year or more.  A defense might be password expiration.
>
> Which of these were you thinking?  I would be happy to make progress on any of them.
>
> - Joseph
>
> > -->Neil
> >
> >
> > -----Original Message-----
> > From: Joseph Reynolds <jrey@linux.ibm.com>
> > Sent: Wednesday, September 9, 2020 4:21 PM
> > To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist
> > <openbmc@lists.ozlabs.org>
> > Subject: Re: bmcweb 30 second lockout
> >
> > On 9/9/20 5:13 PM, Neil Bradley wrote:
> >> I had recently read somewhere on the OpenBMC mailing list (forgive
> >> me, as I can't find it anywhere now) recently indicating that there'd
> >> be a
> >> 30 second lockout for a given user if there were 3 consecutive failed
> >> login attempts. My question is firstly, is this the case, and
> >> secondly, is it tied to the user globally regardless of connection or
> >> is it per user and connection? The reason I ask is that the former
> >> would still allow for a denial of service attack and want to make
> >> sure that's not actually the case.
> >>
> > I can think of two items:
> >
> > 1. I had pushed an experimental gerrit code review to do what you described.
> > Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
> > My idea has not gained much traction and is recorded here:
> > https://github.com/ibm-openbmc/dev/issues/2434
> > and here: https://github.com/linux-pam/linux-pam/issues/216
> > and here: https://github.com/deksai/pam_abl/issues/4
> > and other places.
> > If this ever gets merged, it would NOT the be default behavior.
> >
> > 2. The user lockouts for failed authentication attempts is handled by
> > pam_tally2 and controlled by Redfish APIs.
> > See
> > https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e
> > xtended/pam/libpam/pam.d/common-auth
> > In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts
> > are never locked because of failed authentication attempts".
> >
> > The Redfish APIs are implemented here:
> > https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account
> > _service.hpp Specifically, PATCHing /redfish/v1/AccountService/
> > property AccountLockoutDurationor AccountLockoutThreshold invokes a
> > D-Bus method which ultimately modifies the pam.d/common-auth config
> > file above.
> >
> > Note that downstream projects may typically want to modify these
> > default settings.
> >
> > 3. I don't think you mean this: There is a current code review for a
> > BMCWeb enhancement to allow the BMC admin to control the idle session
> > SessionTimeout property.  The minimum is 30 seconds. See
> > https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016
> >
> > - Joseph
> >
> >> Thanks!
> >>
> >> àNeil
> >>
>

Re: bmcweb 30 second lockout

[Joseph Reynolds]

On 9/14/20 9:55 AM, Ed Tanous wrote:
> On Fri, Sep 11, 2020 at 2:43 PM Neil Bradley <Neil_Bradley@phoenix.com> wrote:
>> Hi Joseph!
>>
>> Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in.
>>
>> I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address.
>>
>> I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it.
>
> This type of approach works great in an apache-type server, where we
> have thread per connection, and we can rely on running out of threads
> to slow us down.  For better or worse, bmcweb is an async reactor
> model (a lot like nginx), so while a 2-4 second delay would slow that
> that specific connection, all an attacker would have to do is to spin
> up N connections in parallel to bypass this per-connection limit.

Ed, thanks.  Would the solution above help if BMCWeb also limited the 
number of simultaneous connections, for example N=32.  Then the attacker 
can guess only 32 password every few seconds.  But what are the 
consequences?  During an attack, can a legitimate user compete for one 
of these connections?

I am summarizing the results of this email conversation here: 
https://github.com/openbmc/bmcweb/issues/151

- Joseph

>
> For what it's worth, I think something similar to the fail2ban
> approach would work well here, where after a number of failures, we
> lock out specific IPs for a specific amount of time at the networking
> layer using something like nftables.  Unfortunately, fail2ban is
> python, so that's a non-starter but there's a similar "copy" of it
> here that I've been meaning to play with at some point:
> https://github.com/swillner/regban
> (credit goes to James F for finding it).

For what its worth, I think fail2ban or regban could be part of a solution.

>> Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away.
>>
>>
>> Let me know what you think.
>>
>> -->Neil
>>
>> -----Original Message-----
>> From: Joseph Reynolds <jrey@linux.ibm.com>
>> Sent: Friday, September 11, 2020 2:11 PM
>> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
>> Subject: Re: bmcweb 30 second lockout
>>
>> On 9/9/20 6:26 PM, Neil Bradley wrote:
>>> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:
>>>
>>> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."
>>>
>>> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.
>> Thanks Neil.  Its you and me against the world.
>>
>> Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem?  That is a difficult problem, and I think solving it will require defenses in multiple layers.
>>
>> I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below).  I think a PAM authentication module like pam_abl could be a solution here.  Would that work for you?  But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them.  My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt.  Is this what you want?
>>
>> The approach you mentioned ("delay a few seconds when credentials are
>> wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways:
>> - When the attacker establishes multiple connections to the BMC.  A defense against this might be limiting the number of connections.
>> - When the attacker has a long time to attack, for example, a year or more.  A defense might be password expiration.
>>
>> Which of these were you thinking?  I would be happy to make progress on any of them.
>>
>> - Joseph
>>
>>> -->Neil
>>>
>>>
>>> -----Original Message-----
>>> From: Joseph Reynolds <jrey@linux.ibm.com>
>>> Sent: Wednesday, September 9, 2020 4:21 PM
>>> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist
>>> <openbmc@lists.ozlabs.org>
>>> Subject: Re: bmcweb 30 second lockout
>>>
>>> On 9/9/20 5:13 PM, Neil Bradley wrote:
>>>> I had recently read somewhere on the OpenBMC mailing list (forgive
>>>> me, as I can't find it anywhere now) recently indicating that there'd
>>>> be a
>>>> 30 second lockout for a given user if there were 3 consecutive failed
>>>> login attempts. My question is firstly, is this the case, and
>>>> secondly, is it tied to the user globally regardless of connection or
>>>> is it per user and connection? The reason I ask is that the former
>>>> would still allow for a denial of service attack and want to make
>>>> sure that's not actually the case.
>>>>
>>> I can think of two items:
>>>
>>> 1. I had pushed an experimental gerrit code review to do what you described.
>>> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
>>> My idea has not gained much traction and is recorded here:
>>> https://github.com/ibm-openbmc/dev/issues/2434
>>> and here: https://github.com/linux-pam/linux-pam/issues/216
>>> and here: https://github.com/deksai/pam_abl/issues/4
>>> and other places.
>>> If this ever gets merged, it would NOT the be default behavior.
>>>
>>> 2. The user lockouts for failed authentication attempts is handled by
>>> pam_tally2 and controlled by Redfish APIs.
>>> See
>>> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e
>>> xtended/pam/libpam/pam.d/common-auth
>>> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts
>>> are never locked because of failed authentication attempts".
>>>
>>> The Redfish APIs are implemented here:
>>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account
>>> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/
>>> property AccountLockoutDurationor AccountLockoutThreshold invokes a
>>> D-Bus method which ultimately modifies the pam.d/common-auth config
>>> file above.
>>>
>>> Note that downstream projects may typically want to modify these
>>> default settings.
>>>
>>> 3. I don't think you mean this: There is a current code review for a
>>> BMCWeb enhancement to allow the BMC admin to control the idle session
>>> SessionTimeout property.  The minimum is 30 seconds. See
>>> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016
>>>
>>> - Joseph
>>>
>>>> Thanks!
>>>>
>>>> àNeil
>>>>

Re: bmcweb 30 second lockout

[Ed Tanous]

On Tue, Sep 15, 2020 at 9:15 AM Joseph Reynolds <jrey@linux.ibm.com> wrote:
>
> On 9/14/20 9:55 AM, Ed Tanous wrote:
> > On Fri, Sep 11, 2020 at 2:43 PM Neil Bradley <Neil_Bradley@phoenix.com> wrote:
> >> Hi Joseph!
> >>
> >> Correct: The problem we wish to solve is to stop or make brute force password attacks logistically impossible/useless, but still allow credentialed individuals in.
> >>
> >> I don't have any particular objection to a short lockout period, provided that lockout period is bound to both the user and the actual TCP connection (assuming a TCP link is maintained during the login process) or perhaps the source IP address.
> >>
> >> I still think a simple 2-4 second delay (without the 30 second delay) is more than sufficient. For example, let's say there's 96 possible characters and an 8 character password. That's 7,213,895,789,838,336 possibilities. Let's also say there are 8000 incoming TCP connections to the web server - I know, it's kind of absurd, but let's assume a massively coordinated attack. /8000=901,736,973,729. Multiply that by 2 seconds, and you get roughly 57,187 years to exhaust all possibilities. Granted, it's likely the password would be found in a quarter of that time, but that's still 14,296 years. The problem gets lots worse quickly when the password is longer. Even a 4 character password is 84,934,656 possibilities and 8000 incoming connections it'd still take 5.8 hours to exhaust everything, and by then the system would have lots of other problems with that many attempts. Though it also could be successfully argued that anyone using a 4 character password is asking for it.
> >
> > This type of approach works great in an apache-type server, where we
> > have thread per connection, and we can rely on running out of threads
> > to slow us down.  For better or worse, bmcweb is an async reactor
> > model (a lot like nginx), so while a 2-4 second delay would slow that
> > that specific connection, all an attacker would have to do is to spin
> > up N connections in parallel to bypass this per-connection limit.
>
> Ed, thanks.  Would the solution above help if BMCWeb also limited the
> number of simultaneous connections, for example N=32.  Then the attacker
> can guess only 32 password every few seconds.  But what are the
> consequences?  During an attack, can a legitimate user compete for one
> of these connections?

They can compete, but if the connections are slow and long lived, a
user is unlikely to succeed.  With that said, it's easy enough to run
the bmc out of bandwidth and cause a DOS attack that way, so we're not
really any worse off.
With that said, I'd still much rather a per-ip lockout.

>
> I am summarizing the results of this email conversation here:
> https://github.com/openbmc/bmcweb/issues/151
>
> - Joseph
>
> >
> > For what it's worth, I think something similar to the fail2ban
> > approach would work well here, where after a number of failures, we
> > lock out specific IPs for a specific amount of time at the networking
> > layer using something like nftables.  Unfortunately, fail2ban is
> > python, so that's a non-starter but there's a similar "copy" of it
> > here that I've been meaning to play with at some point:
> > https://github.com/swillner/regban
> > (credit goes to James F for finding it).
>
> For what its worth, I think fail2ban or regban could be part of a solution.
>
> >> Unless I'm completely missing something, I don't see that a longer lockout period is necessary or at all. I do understand the confusion of a user who doesn't know if they're being locked out because their password , however, but without a lockout period, that problem goes away.
> >>
> >>
> >> Let me know what you think.
> >>
> >> -->Neil
> >>
> >> -----Original Message-----
> >> From: Joseph Reynolds <jrey@linux.ibm.com>
> >> Sent: Friday, September 11, 2020 2:11 PM
> >> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist <openbmc@lists.ozlabs.org>
> >> Subject: Re: bmcweb 30 second lockout
> >>
> >> On 9/9/20 6:26 PM, Neil Bradley wrote:
> >>> Thanks Joseph! I'll look at this in more detail in a bit but I noticed your comments here:
> >>>
> >>> " This issue is to add capability for "authentication rate limiting" for BMCWeb (login and Basic Auth), SSH login, and ideally for IPMI authentication. This capability is to be independent from account lockouts, specifically so someone could use the auth rate limiting as an effective defense and not need to lock accounts."
> >>>
> >>> This is precisely what I was getting at. I'm not a fan of lockouts generally because they can be abused (depending upon implementation of course), and adding in auth rate limiting (just delaying a few seconds when credentials are wrong) winds up making brute force attacks ineffective.
> >> Thanks Neil.  Its you and me against the world.
> >>
> >> Let's be clear about the problem we want to solve: stop brute force password-guessing attacks but not deny service to legitimate users. Is that also your problem?  That is a difficult problem, and I think solving it will require defenses in multiple layers.
> >>
> >> I believe the most promising solution is limiting the number of authentication attempts after too many failures (item 1 in the email below).  I think a PAM authentication module like pam_abl could be a solution here.  Would that work for you?  But I want to go a step farther and actually notify the person logging in when their attempt was rejected because authentication rate-limiting is in effect. Right now they get an "authentication failure" message which leads them to believe their password was wrong, or their account is missing, and that confuses them.  My issues with linux-pam and pam_abl are to help provide clarity to the user what happeded to their login attempt.  Is this what you want?
> >>
> >> The approach you mentioned ("delay a few seconds when credentials are
> >> wrong") defends against a single-threaded attacker (or a misbehaving tool), but it breaks down in (at least) two ways:
> >> - When the attacker establishes multiple connections to the BMC.  A defense against this might be limiting the number of connections.
> >> - When the attacker has a long time to attack, for example, a year or more.  A defense might be password expiration.
> >>
> >> Which of these were you thinking?  I would be happy to make progress on any of them.
> >>
> >> - Joseph
> >>
> >>> -->Neil
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Joseph Reynolds <jrey@linux.ibm.com>
> >>> Sent: Wednesday, September 9, 2020 4:21 PM
> >>> To: Neil Bradley <Neil_Bradley@phoenix.com>; OpenBMC Maillist
> >>> <openbmc@lists.ozlabs.org>
> >>> Subject: Re: bmcweb 30 second lockout
> >>>
> >>> On 9/9/20 5:13 PM, Neil Bradley wrote:
> >>>> I had recently read somewhere on the OpenBMC mailing list (forgive
> >>>> me, as I can't find it anywhere now) recently indicating that there'd
> >>>> be a
> >>>> 30 second lockout for a given user if there were 3 consecutive failed
> >>>> login attempts. My question is firstly, is this the case, and
> >>>> secondly, is it tied to the user globally regardless of connection or
> >>>> is it per user and connection? The reason I ask is that the former
> >>>> would still allow for a denial of service attack and want to make
> >>>> sure that's not actually the case.
> >>>>
> >>> I can think of two items:
> >>>
> >>> 1. I had pushed an experimental gerrit code review to do what you described.
> >>> Here: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/31841
> >>> My idea has not gained much traction and is recorded here:
> >>> https://github.com/ibm-openbmc/dev/issues/2434
> >>> and here: https://github.com/linux-pam/linux-pam/issues/216
> >>> and here: https://github.com/deksai/pam_abl/issues/4
> >>> and other places.
> >>> If this ever gets merged, it would NOT the be default behavior.
> >>>
> >>> 2. The user lockouts for failed authentication attempts is handled by
> >>> pam_tally2 and controlled by Redfish APIs.
> >>> See
> >>> https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-e
> >>> xtended/pam/libpam/pam.d/common-auth
> >>> In pam.d/common-auth, the default pam_tally2 deny=0 means "accounts
> >>> are never locked because of failed authentication attempts".
> >>>
> >>> The Redfish APIs are implemented here:
> >>> https://github.com/openbmc/bmcweb/blob/master/redfish-core/lib/account
> >>> _service.hpp Specifically, PATCHing /redfish/v1/AccountService/
> >>> property AccountLockoutDurationor AccountLockoutThreshold invokes a
> >>> D-Bus method which ultimately modifies the pam.d/common-auth config
> >>> file above.
> >>>
> >>> Note that downstream projects may typically want to modify these
> >>> default settings.
> >>>
> >>> 3. I don't think you mean this: There is a current code review for a
> >>> BMCWeb enhancement to allow the BMC admin to control the idle session
> >>> SessionTimeout property.  The minimum is 30 seconds. See
> >>> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/36016
> >>>
> >>> - Joseph
> >>>
> >>>> Thanks!
> >>>>
> >>>> àNeil
> >>>>
>