* [KORG] Taking hera down to deal with the load issues
@ 2011-08-16 23:27 J.H.
2011-08-17 8:05 ` [kernel.org users] " J.H.
0 siblings, 1 reply; 6+ messages in thread
From: J.H. @ 2011-08-16 23:27 UTC (permalink / raw)
To: linux-kernel, users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hey Everyone,
Ok so since I'm back from vacation, and things don't seem to be getting
better, I'm going to take hera/master offline in about 30 minutes or so
(5pm PDT or Tue Aug 17 0:0:0 UTC 2011). This is going to involve a base
OS upgrade, and a full on new kernel.
This should get the loads back into what we are expecting, and give me a
chance to more thoroughly investigate the panics from last week. If
you've got questions or comments, please give me a shout as soon as
possible.
- - John 'Warthog9' Hawley
Chief Kernel.org Administrator
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk5K/PkACgkQ/E3kyWU9dieV8wCcCUmHmIYqeqK0eBUt7f3ne07J
994An3MrErlAjOLx/zkpB6T1dNsHVLtC
=CSPj
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [kernel.org users] [KORG] Taking hera down to deal with the load issues
2011-08-16 23:27 [KORG] Taking hera down to deal with the load issues J.H.
@ 2011-08-17 8:05 ` J.H.
2011-08-17 18:37 ` Jan Kara
2011-08-19 5:14 ` David Miller
0 siblings, 2 replies; 6+ messages in thread
From: J.H. @ 2011-08-17 8:05 UTC (permalink / raw)
To: linux-kernel, users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Sorry about the delay, the upgrade took longer than expected and the
initial kernel didn't actually solve the load issue completely. I've
since installed a very slightly modified build of the initial kernel and
had to reboot hera twice more (both to the same kernel, but I again ran
`lsof -p <pid>` and had Linux panic on me. A feature that has very
consistently bitten me on a number of kernels and a number of systems
lately).
This second kernel is at least, so far, acting a lot closer to what I'd
expect from hera, but the disk cache is still filling. I'm going to
keep an eye on it for a bit longer tonight. As a note, the only
difference between the first 3.1-rc1 kernel and the second, that is
currently running, was the explicit disabling of IMA and recompiling the
kernel.
I'm going to give the all clear at this point, I won't be doing any more
reboots without an additional e-mail and notice. If you see any
problems or issue, or you catch that the loads are spiking up into the
stratosphere, please let me know.
- - John 'Warthog9' Hawley
On 08/16/2011 04:27 PM, J.H. wrote:
> Hey Everyone,
>
> Ok so since I'm back from vacation, and things don't seem to be getting
> better, I'm going to take hera/master offline in about 30 minutes or so
> (5pm PDT or Tue Aug 17 0:0:0 UTC 2011). This is going to involve a base
> OS upgrade, and a full on new kernel.
>
> This should get the loads back into what we are expecting, and give me a
> chance to more thoroughly investigate the panics from last week. If
> you've got questions or comments, please give me a shout as soon as
> possible.
>
> - John 'Warthog9' Hawley
> Chief Kernel.org Administrator
_______________________________________________
Users mailing list
Users@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk5LdmQACgkQ/E3kyWU9dic7rgCcDuBBmmfmNeyqs/zC1t9cByzy
5J0AnRcfO4701ix0HAr+KVmNKVedl38O
=FQPl
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [kernel.org users] [KORG] Taking hera down to deal with the load issues
2011-08-17 8:05 ` [kernel.org users] " J.H.
@ 2011-08-17 18:37 ` Jan Kara
2011-08-18 18:52 ` J.H.
2011-08-19 5:14 ` David Miller
1 sibling, 1 reply; 6+ messages in thread
From: Jan Kara @ 2011-08-17 18:37 UTC (permalink / raw)
To: J.H.; +Cc: linux-kernel, users
Hi,
On Wed 17-08-11 01:05:56, J.H. wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> This second kernel is at least, so far, acting a lot closer to what I'd
> expect from hera, but the disk cache is still filling. I'm going to
> keep an eye on it for a bit longer tonight. As a note, the only
> difference between the first 3.1-rc1 kernel and the second, that is
> currently running, was the explicit disabling of IMA and recompiling the
> kernel.
What do you exactly mean by "disk cache is still filling"? 3.1-rc1 has
quite some changes in writeback code so it's possible something went
wrong...
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [kernel.org users] [KORG] Taking hera down to deal with the load issues
2011-08-17 18:37 ` Jan Kara
@ 2011-08-18 18:52 ` J.H.
0 siblings, 0 replies; 6+ messages in thread
From: J.H. @ 2011-08-18 18:52 UTC (permalink / raw)
To: Jan Kara; +Cc: linux-kernel, users
On 08/17/2011 11:37 AM, Jan Kara wrote:
> Hi,
>
> On Wed 17-08-11 01:05:56, J.H. wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> This second kernel is at least, so far, acting a lot closer to what I'd
>> expect from hera, but the disk cache is still filling. I'm going to
>> keep an eye on it for a bit longer tonight. As a note, the only
>> difference between the first 3.1-rc1 kernel and the second, that is
>> currently running, was the explicit disabling of IMA and recompiling the
>> kernel.
> What do you exactly mean by "disk cache is still filling"? 3.1-rc1 has
> quite some changes in writeback code so it's possible something went
> wrong...
Mostly that it takes time to read the bits off of the spinning rust and
have it cached into memory. Not specifically that there's something
wrong with the writeback, I'm at least not seeing one I believe.
- John 'Warthog9' Hawley
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [kernel.org users] [KORG] Taking hera down to deal with the load issues
2011-08-17 8:05 ` [kernel.org users] " J.H.
2011-08-17 18:37 ` Jan Kara
@ 2011-08-19 5:14 ` David Miller
2011-08-19 11:08 ` Mimi Zohar
1 sibling, 1 reply; 6+ messages in thread
From: David Miller @ 2011-08-19 5:14 UTC (permalink / raw)
To: warthog9; +Cc: linux-kernel, users
From: "J.H." <warthog9@kernel.org>
Date: Wed, 17 Aug 2011 01:05:56 -0700
> This second kernel is at least, so far, acting a lot closer to what I'd
> expect from hera, but the disk cache is still filling. I'm going to
> keep an eye on it for a bit longer tonight. As a note, the only
> difference between the first 3.1-rc1 kernel and the second, that is
> currently running, was the explicit disabling of IMA and recompiling the
> kernel.
I wonder if the IMA bits are generating crypto hashes for every file
modification done on the machine.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [kernel.org users] [KORG] Taking hera down to deal with the load issues
2011-08-19 5:14 ` David Miller
@ 2011-08-19 11:08 ` Mimi Zohar
0 siblings, 0 replies; 6+ messages in thread
From: Mimi Zohar @ 2011-08-19 11:08 UTC (permalink / raw)
To: David Miller; +Cc: warthog9, linux-kernel, users
On Thu, 2011-08-18 at 22:14 -0700, David Miller wrote:
> From: "J.H." <warthog9@kernel.org>
> Date: Wed, 17 Aug 2011 01:05:56 -0700
>
> > This second kernel is at least, so far, acting a lot closer to what I'd
> > expect from hera, but the disk cache is still filling. I'm going to
> > keep an eye on it for a bit longer tonight. As a note, the only
> > difference between the first 3.1-rc1 kernel and the second, that is
> > currently running, was the explicit disabling of IMA and recompiling the
> > kernel.
>
> I wonder if the IMA bits are generating crypto hashes for every file
> modification done on the machine.
Possibly, but IMA doesn't recalculate the hash for every modification.
Without the boot command line parameter 'ima_tcb', nothing should be
measured. With 'ima_tcb', only files accessed by root are measured and
then only re-measured, if flushed from the inode cache or has been
marked changed, which occurs on __fput(), not every write.
thanks,
Mimi
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-08-19 11:08 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-08-16 23:27 [KORG] Taking hera down to deal with the load issues J.H.
2011-08-17 8:05 ` [kernel.org users] " J.H.
2011-08-17 18:37 ` Jan Kara
2011-08-18 18:52 ` J.H.
2011-08-19 5:14 ` David Miller
2011-08-19 11:08 ` Mimi Zohar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).