performance questions

performance questions

[LC Bruzenak]

I was looking at some strace results from a process using the
audit_log_user_message call and I think I see how I can eliminate some
ioctls and /proc/self lookups by setting the hostname/tty parameters to
non-NULL pointers pointing to NULL values.

But the exename is another story. It does a lookup each time. We have
persistent processes each of which submit 100Ks (on the way to 1Ms) of
audit_log_user_message events daily, so it would make a difference.

I was thinking about a patch to store off the exename statically if one
isn't already in the pipeline. Let me know; I'll submit something if
not.

The other question is on the auditd side. IIUC on each event the
write_to_log function is checking the logfile size. Seems to me that we
could limit the fstat checks to say one every ten events or so. Any
problems there?

Thx,
LCB 

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: performance questions

[Steve Grubb]

On Thursday, September 29, 2011 11:33:09 AM LC Bruzenak wrote:
> I was looking at some strace results from a process using the
> audit_log_user_message call and I think I see how I can eliminate some
> ioctls and /proc/self lookups by setting the hostname/tty parameters to
> non-NULL pointers pointing to NULL values.
> 
> But the exename is another story. It does a lookup each time. We have
> persistent processes each of which submit 100Ks (on the way to 1Ms) of
> audit_log_user_message events daily, so it would make a difference.
> 
> I was thinking about a patch to store off the exename statically if one
> isn't already in the pipeline. Let me know; I'll submit something if
> not.

You might try this:

diff -urp audit-2.1.4.orig/lib/audit_logging.c audit-2.1.4/lib/audit_logging.c
--- audit-2.1.4.orig/lib/audit_logging.c	2011-09-06 14:17:06.000000000 -0400
+++ audit-2.1.4/lib/audit_logging.c	2011-09-30 09:08:50.000000000 -0400
@@ -240,7 +240,7 @@ int audit_log_user_message(int audit_fd,
 {
 	char buf[MAX_AUDIT_MESSAGE_LENGTH];
 	char addrbuf[INET6_ADDRSTRLEN];
-	char exename[PATH_MAX*2];
+	static char exename[PATH_MAX*2]="";
 	char ttyname[TTY_PATH];
 	const char *success;
 	int ret;
@@ -262,7 +262,8 @@ int audit_log_user_message(int audit_fd,
 	else
 		strncat(addrbuf, addr, sizeof(addrbuf)-1);
 
-	_get_exename(exename, sizeof(exename));
+	if (exename[0] == 0)
+		_get_exename(exename, sizeof(exename));
 	if (tty == NULL) 
 		tty = _get_tty(ttyname, TTY_PATH);
 	else if (*tty == 0)


> The other question is on the auditd side. IIUC on each event the
> write_to_log function is checking the logfile size. Seems to me that we
> could limit the fstat checks to say one every ten events or so. Any
> problems there?

We can probably use the return value of fprintf() +1 (for the NULL byte) and 
just keep the running total in memory.

-Steve

Re: performance questions

[LC Bruzenak]

On Fri, 2011-09-30 at 09:20 -0400, Steve Grubb wrote:
> On Thursday, September 29, 2011 11:33:09 AM LC Bruzenak wrote:
...
> 
> You might try this:
...
>  
> -	_get_exename(exename, sizeof(exename));
> +	if (exename[0] == 0)
> +		_get_exename(exename, sizeof(exename));
>  	if (tty == NULL) 
>  		tty = _get_tty(ttyname, TTY_PATH);
>  	else if (*tty == 0)

Well, we could (and then it would work like the others) but we really
want to store the exename I think. Isn't that what becomes
"exe=<EXEPATH>" in the event?

> 
> We can probably use the return value of fprintf() +1 (for the NULL byte) and 
> just keep the running total in memory.

Oh, right. That would be more precise. Good idea!

Since we're looking, what about the fstatfs in check_disk_space? Any
thoughts on that one?

Thanks Steve!
LCB

-- 
LC (Lenny) Bruzenak
lenny@magitekltd.com

Re: performance questions

[Steve Grubb]

On Friday, September 30, 2011 10:20:43 AM LC Bruzenak wrote:
> On Fri, 2011-09-30 at 09:20 -0400, Steve Grubb wrote:
> > On Thursday, September 29, 2011 11:33:09 AM LC Bruzenak wrote:
> ...
> 
> > You might try this:
> ...
> 
> > -	_get_exename(exename, sizeof(exename));
> > +	if (exename[0] == 0)
> > +		_get_exename(exename, sizeof(exename));
> > 
> >  	if (tty == NULL)
> >  	
> >  		tty = _get_tty(ttyname, TTY_PATH);
> >  	
> >  	else if (*tty == 0)
> 
> Well, we could (and then it would work like the others) but we really
> want to store the exename I think. Isn't that what becomes
> "exe=<EXEPATH>" in the event?

It does. You can strace it. :)

 
> > We can probably use the return value of fprintf() +1 (for the NULL byte)
> > and just keep the running total in memory.
> 
> Oh, right. That would be more precise. Good idea!
> 
> Since we're looking, what about the fstatfs in check_disk_space? Any
> thoughts on that one?

Probably can't get rid of that one. Many times people don't separate the audit 
directory to its own partition. So, we wind up sharing space with /var/log/messages 
which anyone can write to. Even if we had it exclusively, sometimes there is a cron 
job that might come and grab files for archiving in which case an internal count would 
be off.

-Steve

Performance questions.

[Tomasz Kusmierz]

Gentleman, 

I don’t want this to sound too pretentious but when running ceph, under certain loads it curiously feels like watching a sci-fi movie from 80’s where computers where spitting out letter after letter.

I’ve just sat in front of terminal watching 
rm -rf /certain_folder 
taking ~23 minutes to delete 88100 files on cephfs. I’ve seriously seen file after file being deleted (hence the reference to movies like alien etc)

So this is just a simple “test box” which runs a single node ceph installation with 4 osd. There is _nothing_ else running on this cephfs … machine is stable for 30 minutes before running the command … It just seems very odd. I use this server (at my home) to test stuff before I put it into production. 

Worst part is that in production on system with 3 node X 4 osd where everything sits in RDB (not cephfs) I see pretty much the same behaviour … writes take way to long than they should, latencies are horrendous and when for example CCTV drops 5GB of footage during scheduled process - everything grind to a halt and most of other VM’s are nearly inaccessible, CCTV can’t even store any file on it’s own FS.

I do run everything more or less “out of the box”. Even on home setup when I pump large file into cephfs (other pc -> samba -> server folder -> mounted cephfs) every 2gb everything just halts and ceph tries to “catch up” because there is no network IO but disks are chugging away.

Is there anything a poor schmuck can do to get most out of their hardware ? I’ve tried going down the route of enabling async for messenger with:
ms_type = async
but that seem to give zero benefit.


ps. lot of documentation seems dated, specially that bluestor is just “mentioned” few times …  

performance questions

[Filka Michal]

Hi all,

I did some performance tests focused on routing.

Traffic is generated using a traffic generator which has two interfaces
connected to two different switches. Router is connected to those
switches and routes traffic from one generator's IP to other one.
Everything works over 1Gbps Ethernet using metallic and optical
interconnection.

Case A:
The Generator created two flows. First was icmp traffic (echo request)
with packet size 64B. Average performance with zero packet loss was 28%
(cca 400.000 tx fps). Second flow was udp traffic with packet size 64B.
Traffic decreased to average performance round 23% (cca 340.000 tx fps).

Case B:
Then I decided to create a module which allows bypassing nefilter for
some traffic and study changes in performance. The module registers
netfilter callbacks at the beginning of PREROUTING and POSTROUTING
chains. It stops executing of other netfilter hooks for particular
traffic (returns NF_STOLEN (in this case runs okfn too) or NF_STOP - I
tried both). I run both tests as described earlier. First test (icmp
traffic) increased performance to 30% but the second (udp traffic)
decreased to 20%.

I need a help in explaining following questions.
1) Does anyone have any idea why is there such difference in performance
in case A? In my opinion protocol above ip (icmp or udp in this case)
shouldn't make difference and the performance should be almost
identical.

2) What could cause performance decrease of udp traffic in case B? the
result is worst than in Case A for udp traffic but better for icmp
traffic and it looks strange to me.

Thanks for your help.
Michal Filka