iptables and combining additional rule sources

iptables and combining additional rule sources

[Jeffrey Walton]

Hi Everyone,

We are having trouble with our MediaWiki installation on a low-end VM.
The VM is servicing a lot of spam traffic, and it is driving cpu usage
up to about 80%. The 404's appear to be more expensive then the 200's.
GoDaddy wrote to us and told us they were going to suspend our service
if we don't get cpu usage down.

I experimented with several Apache and MediaWiki plugins and I have a
design I like. The plugin scans the URL, detects the problematic URLs,
and sends the ip address to a privileged out-of-proc proxy to update
iptables. The proxy is privileged and can update iptables rules. It
also maintains a database to remove the host after 45 days.

The problem I am having is, adding the new information to the existing
iptables rules in /etc/sysconfig/iptables. I want to write my rules to
a separate file and then tell /etc/sysconfig/iptables to include it at
the correct position.

I read the iptables(8), iptables-save(8) and iptables-restore(8) man
pages, but I don't see how to combine the different sources.

How do I tell iptables to include a second external source at a
specific location?

# iptables --version
iptables v1.4.21

Thanks in advance.

=========================

Here is an example of /etc/sysconfig/iptables with the position I want
to insert the MediaWiki ban rules.

# cat /etc/sysconfig/iptables
*nat
:PREROUTING ACCEPT [4276:232374]
:POSTROUTING ACCEPT [270:136514]
:OUTPUT ACCEPT [270:136514]
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [269:205262]

-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT

### I want to insert rules here ###
*include my-mediawiki-rules

### Back to normal rules ###
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
...

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 888 bytes --]

On Fri, 24 Apr 2020 18:28:21 -0400, Jeffrey Walton said:

> The problem I am having is, adding the new information to the existing
> iptables rules in /etc/sysconfig/iptables. I want to write my rules to
> a separate file and then tell /etc/sysconfig/iptables to include it at
> the correct position.
>
> I read the iptables(8), iptables-save(8) and iptables-restore(8) man
> pages, but I don't see how to combine the different sources.
>
> How do I tell iptables to include a second external source at a
> specific location?

Turn the problem on its side....

#!/bin/bash
cat /etc/iptables.header /etc/iptables.newstuff /etc/iptables.trailer > /etc/sysconfig/iptables
iptables-restore < /etc/sysconfig/iptables

(basically the solution I did for an NFS server, where 'newstuff' and /etc/exports were
both machine-generated by a perl script that read a config file of authorized clients.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Keh-Ming Luoh]

[-- Attachment #1.1: Type: text/plain, Size: 2407 bytes --]

Assuming these IP address are treated the same way in your iptables rule,
ipset may help to make it simpler.



On Fri, Apr 24, 2020 at 3:30 PM Jeffrey Walton <noloader@gmail.com> wrote:

> Hi Everyone,
>
> We are having trouble with our MediaWiki installation on a low-end VM.
> The VM is servicing a lot of spam traffic, and it is driving cpu usage
> up to about 80%. The 404's appear to be more expensive then the 200's.
> GoDaddy wrote to us and told us they were going to suspend our service
> if we don't get cpu usage down.
>
> I experimented with several Apache and MediaWiki plugins and I have a
> design I like. The plugin scans the URL, detects the problematic URLs,
> and sends the ip address to a privileged out-of-proc proxy to update
> iptables. The proxy is privileged and can update iptables rules. It
> also maintains a database to remove the host after 45 days.
>
> The problem I am having is, adding the new information to the existing
> iptables rules in /etc/sysconfig/iptables. I want to write my rules to
> a separate file and then tell /etc/sysconfig/iptables to include it at
> the correct position.
>
> I read the iptables(8), iptables-save(8) and iptables-restore(8) man
> pages, but I don't see how to combine the different sources.
>
> How do I tell iptables to include a second external source at a
> specific location?
>
> # iptables --version
> iptables v1.4.21
>
> Thanks in advance.
>
> =========================
>
> Here is an example of /etc/sysconfig/iptables with the position I want
> to insert the MediaWiki ban rules.
>
> # cat /etc/sysconfig/iptables
> *nat
> :PREROUTING ACCEPT [4276:232374]
> :POSTROUTING ACCEPT [270:136514]
> :OUTPUT ACCEPT [270:136514]
> COMMIT
>
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [269:205262]
>
> -A INPUT -p icmp -j ACCEPT
> -A INPUT -i lo -j ACCEPT
>
> ### I want to insert rules here ###
> *include my-mediawiki-rules
>
> ### Back to normal rules ###
> -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
> -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
> ...
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies@kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

[-- Attachment #1.2: Type: text/html, Size: 3113 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Jeffrey Walton]

On Fri, Apr 24, 2020 at 11:07 PM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> On Fri, 24 Apr 2020 18:28:21 -0400, Jeffrey Walton said:
>
> > The problem I am having is, adding the new information to the existing
> > iptables rules in /etc/sysconfig/iptables. I want to write my rules to
> > a separate file and then tell /etc/sysconfig/iptables to include it at
> > the correct position.
> >
> > I read the iptables(8), iptables-save(8) and iptables-restore(8) man
> > pages, but I don't see how to combine the different sources.
> >
> > How do I tell iptables to include a second external source at a
> > specific location?
>
> Turn the problem on its side....
>
> #!/bin/bash
> cat /etc/iptables.header /etc/iptables.newstuff /etc/iptables.trailer > /etc/sysconfig/iptables
> iptables-restore < /etc/sysconfig/iptables
>
> (basically the solution I did for an NFS server, where 'newstuff' and /etc/exports were
> both machine-generated by a perl script that read a config file of authorized clients.

That's a good idea. I think that may work better for some data sets.

One last question... Should I create my own target - say mediawiki -
and append my rules to it? That may simplify things:

* Header, newstuff and trailer is fixed
* newstuff just jumps to mediawiki target
* if mediawiki does not ban, then control returns to trailer

Then, my out-of-proc service just keeps adding to mediawiki target. I
don't need to write files in this case. I'll just keep adding to the
running config.

Jeff

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Jeffrey Walton]

On Fri, Apr 24, 2020 at 11:32 PM Keh-Ming Luoh <kmluoh@gmail.com> wrote:
>
> Assuming these IP address are treated the same way in your iptables rule, ipset may help to make it simpler.

Thanks Keh-Ming.

Yeah, I was looking for that earlier.

Jeff

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 612 bytes --]

On Sat, 25 Apr 2020 02:55:08 -0400, Jeffrey Walton said:

> One last question... Should I create my own target - say mediawiki -
> and append my rules to it? That may simplify things:
>
> * Header, newstuff and trailer is fixed
> * newstuff just jumps to mediawiki target
> * if mediawiki does not ban, then control returns to trailer
>
> Then, my out-of-proc service just keeps adding to mediawiki target. I
> don't need to write files in this case. I'll just keep adding to the
> running config.

That totally fails if your machine reboots, because there's no memory of
what the rules were before the reboot.


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Jeffrey Walton]

On Sat, Apr 25, 2020 at 12:53 PM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> On Sat, 25 Apr 2020 02:55:08 -0400, Jeffrey Walton said:
>
> > One last question... Should I create my own target - say mediawiki -
> > and append my rules to it? That may simplify things:
> >
> > * Header, newstuff and trailer is fixed
> > * newstuff just jumps to mediawiki target
> > * if mediawiki does not ban, then control returns to trailer
> >
> > Then, my out-of-proc service just keeps adding to mediawiki target. I
> > don't need to write files in this case. I'll just keep adding to the
> > running config.
>
> That totally fails if your machine reboots, because there's no memory of
> what the rules were before the reboot.

Yeah, I'm OK with that. Dropping the database means the code
simplifies _a lot_. I can remove the administrivia, connection
information and all the code for inserts and deletes.

Looking at the logs, these folks are aggressive. It looks like (to me)
the wiki installation will see most spammers in the first 5 or 10
minutes of starting up. After initial startup the system reaches
stability rather quickly.

I think it is a good tradeoff.

Jeff

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Aruna Hewapathirane]

[-- Attachment #1.1: Type: text/plain, Size: 1211 bytes --]

<snip>
> The problem I am having is, adding the new information to the existing
> iptables rules in /etc/sysconfig/iptables. I want to write my rules to
> a separate file and then tell /etc/sysconfig/iptables to include it at
> the correct position.
>
> I read the iptables(8), iptables-save(8) and iptables-restore(8) man
> pages, but I don't see how to combine the different sources.
>
> How do I tell iptables to include a second external source at a
> specific location?
<snip>

1 - Get the iptables rules list with the line numbers enabled
     $ iptables -nL –line-numbers

2 - Look up the line number you want to use and insert your rule.
     ( I am inserting a rule at line number 10, the existing rule will
shift down)
     $ sudo /sbin/iptables -I INPUT 10 -s 202.54.1.1 -j DROP -m comment
--comment "DROP spam IP address - "

3 - Save the rules to a file in etc so you can reload them at the next
reboot
     $ sudo /sbin/iptables-save > /etc/iptables.local

4 - To make the rules persistent, add the following rule to your
/etc/rc.local file)
     sudo /sbin/iptables-restore < /etc/iptables.local

If your system reboots you will not loose the rules now.

Aruna

[-- Attachment #1.2: Type: text/html, Size: 2834 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: iptables and combining additional rule sources

[Thorondir]

On 2020-04-25 00:28, Jeffrey Walton wrote:
> Hi Everyone,
> 
> We are having trouble with our MediaWiki installation on a low-end VM.
> The VM is servicing a lot of spam traffic, and it is driving cpu usage
> up to about 80%. The 404's appear to be more expensive then the 200's.
> GoDaddy wrote to us and told us they were going to suspend our service
> if we don't get cpu usage down.
> 
> I experimented with several Apache and MediaWiki plugins and I have a
> design I like. The plugin scans the URL, detects the problematic URLs,
> and sends the ip address to a privileged out-of-proc proxy to update
> iptables. The proxy is privileged and can update iptables rules. It
> also maintains a database to remove the host after 45 days.

Hi Jeffrey,

have you looked into Fail2Ban? It seems to do what you need, but
real-time.

Kind regards,
Thorondir

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies