Re: kernel BUG at mm/rmap.c:483!

Re: kernel BUG at mm/rmap.c:483!

[Nick Warne]

> But not all cases could be accounted in that way.  If you
> report back that memtest86 ran cleanly...

Hugh,

Nothing to do with the 'problem' in this thread, but an aside that is perhaps 
relevant.

On my main gateway, I couldn't get any kernel greater than 2.6.4 to run 
without an 'oops' after x amount of time.  It was always swapd or memory oops 
that caused it.

I ran memtest86 a few times with no errors - reaseated everything, new fans 
etc. etc.  No go.

I upgraded memory - all 4 sticks - over Christmas, and after a few weeks 
uptime, tried 2.4.10 again.

I have had no problems since - so perhaps I did have bad memory (it was old).  
But all tests never showed anything untoward.

I was always suspicious why my 2.6.4 build ran OK, but newer builds always 
failed.  Could it be a subtle fault in memory whilst building kernels that 
does it?

Nick
-- 
"When you're chewing on life's gristle,
Don't grumble, Give a whistle..."

Re: kernel BUG at mm/rmap.c:483!

[Hugh Dickins]

On Wed, 23 Feb 2005, Nick Warne wrote:
> 
> I upgraded memory - all 4 sticks - over Christmas, and after a few weeks 
> uptime, tried 2.4.10 again.
> 
> I have had no problems since - so perhaps I did have bad memory (it was old).
> But all tests never showed anything untoward.
> 
> I was always suspicious why my 2.6.4 build ran OK, but newer builds always 
> failed.  Could it be a subtle fault in memory whilst building kernels that 
> does it?

Perhaps, though I don't remember hearing of any example of that.

I think what typically happens, on a build the size of the kernel,
is that one of the compilations collapses with a SIGSEGV because some
pointer within gcc gets corrupted by the bad memory, so the build
fails to complete; rather than completing with a bad vmlinux output.

A more likely cause for what you saw, if the bad memory is low down
or high up (and what I mean by high may change: wli made an important
change to memory allocation ordering in 2.6.8 which would affect it),
is that one kernel's image or system initialization will happen to
allocate the bad memory to something scarcely used, where another
may allocate it to something vital.

But please don't place too much weight on my idle speculations,
others have a better appreciation of these issues.

Hugh

Re: kernel BUG at mm/rmap.c:483!

[Arjan van de Ven]

On Mon, 2005-02-28 at 11:43 +0100, Giacomo A. Catenazzi wrote:
> Arjan van de Ven wrote:
> 
> >>but what what's the
> >>penalty of preventing microcode from loading? a performance
> >>hit?
> > 
> > 
> > not even that; in theory a few cpu bugs may have been fixed. Nobody
> > really knows since there's no changelog for the microcode..
> 
> You can see the processor bugs in intel website, i.e.:
> ftp://download.intel.com/design/Xeon/specupdt/24967847.pdf
> 
> The following sentence (IMHO) meens that bug is corrected in microcode:
> "Workaround: It is possible for the BIOS to contain a workaround
> for this erratum."

yeah but it doesn't say in which microcode. Eg it's not possible to find
out what a specific microcode update changes over the one the bios
already put in...

Re: kernel BUG at mm/rmap.c:483!

[Giacomo A. Catenazzi]

Arjan van de Ven wrote:

>>but what what's the
>>penalty of preventing microcode from loading? a performance
>>hit?
> 
> 
> not even that; in theory a few cpu bugs may have been fixed. Nobody
> really knows since there's no changelog for the microcode..

You can see the processor bugs in intel website, i.e.:
ftp://download.intel.com/design/Xeon/specupdt/24967847.pdf

The following sentence (IMHO) meens that bug is corrected in microcode:
"Workaround: It is possible for the BIOS to contain a workaround
for this erratum."

ciao
	cate

Re: kernel BUG at mm/rmap.c:483!

[Horst von Brand]

Hugh Dickins <hugh@veritas.com> said:
> On Wed, 23 Feb 2005, Ammar T. Al-Sayegh wrote:
> > I recently installed Fedora RC3 on a new server.
> > The kernel is 2.6.10-1.741_FC3smp.

> I can't really speak for Fedora RC3 kernels,
> perhaps there's some special patch in there that happens to
> trigger it for you, but certainly there have been occasional
> other reports of this BUG with vanilla kernel.org kernels.

That kernel is outdated, current is 2.6.10-1.766_FC3. Before reporting any
bugs, update everything! And in case of problems with vendor kernels, it is
more useful for everybody involved to report to the distribution.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: kernel BUG at mm/rmap.c:483!

[Arjan van de Ven]

> I guess I can no longer monitor the processor temperature and
> such after preventing i2c from loading, 

yup

> but what what's the
> penalty of preventing microcode from loading? a performance
> hit?

not even that; in theory a few cpu bugs may have been fixed. Nobody
really knows since there's no changelog for the microcode..

Re: kernel BUG at mm/rmap.c:483!

[Ammar T. Al-Sayegh]

>>  really; it was supposed to do that already
>> > 
>> >> i2c_dev                13249  0 
>> >> i2c_core               24513  1 i2c_dev
>> > 
>> > try for fun to not use i2c for a while
>> > 
>> >> microcode              11489  0 
>> > same for microcode... try removing that so that the microcode of your
>> > system doesn't get updated at boot
>> 
>> What do these two modules do in particular? and how can I disable
>> them so that they don't get reloaded during boot time? do I need
>> to disable both i2c_dev and i2c_core or just one of them?
> 
> i2c is used to directly talk to motherboard hardware such as temperature
> sensors. I've seen cases of certain chipset bugs leading to cacheline
> corruption when stuff talked to the slow i2c bus and did other stuff in
> parallel. 
> 
> microcode changes the microcode of the cpu (a part of your cpu is
> actually written in "software" that can be updated); however updating
> this behind the back of the bios might not always be a good idea. (but I
> have no hard proof of any failures due to this)
> 
> As for how to disable these.. you could just rename the respective .ko
> files to .notko or something....

Done. Following is my new loaded module list:

[root ~]# lsmod 
Module                  Size  Used by
ip_conntrack_ftp       76145  0 
md5                     8001  1 
ipv6                  236769  38 
autofs4                21829  0 
sunrpc                135077  1 
ipt_REJECT             10561  2 
ipt_state               5825  79 
ip_conntrack           45317  2 ip_conntrack_ftp,ipt_state
iptable_filter          7489  1 
ip_tables              20929  3 ipt_REJECT,ipt_state,iptable_filter
dm_mod                 57925  0 
video                  19653  0 
button                 10577  0 
battery                13253  0 
ac                      8773  0 
uhci_hcd               33497  0 
ehci_hcd               33737  0 
e1000                  84629  0 
floppy                 56913  0 
ext3                  117961  6 
jbd                    57177  1 ext3
3w_xxxx                30561  0 
ata_piix               12485  7 
libata                 44101  1 ata_piix
sd_mod                 20545  9 
scsi_mod              116033  3 3w_xxxx,libata,sd_mod

Looks better now?

I guess I can no longer monitor the processor temperature and
such after preventing i2c from loading, but what what's the
penalty of preventing microcode from loading? a performance
hit?

I will be testing memory as suggested by Hugh Dickins as well.
Hopefully, your trick or Hugh's suggestion will help revealing
the source of the problem, if not the kernel itself.


-ammar

Re: kernel BUG at mm/rmap.c:483!

[Arjan van de Ven]

>  really; it was supposed to do that already
> > 
> >> i2c_dev                13249  0 
> >> i2c_core               24513  1 i2c_dev
> > 
> > try for fun to not use i2c for a while
> > 
> >> microcode              11489  0 
> > same for microcode... try removing that so that the microcode of your
> > system doesn't get updated at boot
> 
> What do these two modules do in particular? and how can I disable
> them so that they don't get reloaded during boot time? do I need
> to disable both i2c_dev and i2c_core or just one of them?

i2c is used to directly talk to motherboard hardware such as temperature
sensors. I've seen cases of certain chipset bugs leading to cacheline
corruption when stuff talked to the slow i2c bus and did other stuff in
parallel. 

microcode changes the microcode of the cpu (a part of your cpu is
actually written in "software" that can be updated); however updating
this behind the back of the bios might not always be a good idea. (but I
have no hard proof of any failures due to this)

As for how to disable these.. you could just rename the respective .ko
files to .notko or something....

Re: kernel BUG at mm/rmap.c:483!

[Hugh Dickins]

On Wed, 23 Feb 2005, Ammar T. Al-Sayegh wrote:
> ----- Original Message ----- From: "Hugh Dickins" <hugh@veritas.com>
> > though quite possibly you cannot afford
> > such experiments on this server, and will revert to 2.4 for now.
> 
> The problem is that my server is already in production
> mode. I'm running great portion of my business on it,
> where there is very little tolerance for downtime.

I feared as much.

> Because the server is located in a remote datacenter,
> every time it goes down it takes several hours to have
> someone sent up there to manually reboot it for a hefty
> emergency fee. So this bug has already cost me a lot of
> money, and I'm worried that it will cost me a lot of my
> clients as well if it persists.

I'm very sorry for that.

> Remote hands are rather expensive, so it will cost me
> $100/hr to have someone runs memtest86 on my server
> since I can't perform it remotely. I'll do it though
> since that's your recommendation for the time being.
> Hope it will not take more than an hour to run the
> test, and hope it turns out as bad memory modules as
> you expect because I hate to downgrade after all the
> time and money I expended on the upgrade.

One hour will be enough if it does find a problem in that time,
worth a shot; but not enough to give confidence in the memory
if it does not find one, 12 hours better.  I actually wonder
whether rmap.c:483 is the best memory tester (serious answer
would be, in some cases yes, but not in all).

Do let me know.  If I can find time to rejig the debug patch
against your kernel, it would itself keep your server running,
replacing the BUG_ON by printks and safety.  But without knowing
what it will report, I can't judge how satisfactory that would
be (and it's unlikely to lead us to the final answer in one go).

Hugh

Re: kernel BUG at mm/rmap.c:483!

[Ammar T. Al-Sayegh]

----- Original Message ----- 
From: "Arjan van de Ven" <arjan@infradead.org>
To: "Ammar T. Al-Sayegh" <ammar@kunet.com>
Cc: <linux-kernel@vger.kernel.org>
Sent: Wednesday, February 23, 2005 5:01 PM
Subject: Re: kernel BUG at mm/rmap.c:483!


> On Wed, 2005-02-23 at 16:45 -0500, Ammar T. Al-Sayegh wrote:
>> > On Wed, 2005-02-23 at 15:41 -0500, Ammar T. Al-Sayegh wrote:
>> >> Hi All,
>> >> 
>> >> I recently installed Fedora RC3 on a new server.
>> >> The kernel is 2.6.10-1.741_FC3smp. The server
>> >> crashes every few days. When I examine /var/log/messages,
>> >> I find the following line just before the crash:
>> >> 
>> >> Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
>> >> Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
>> >> 
>> >> No further debug lines are given to diagnose the
>> >> source of the 
>> > no oops at all?
>> 
>> No. Is there a way to enable the kernel to give more
>> diagnostic debug output next time this error happens?
> 
> not really; it was supposed to do that already
> 
>> i2c_dev                13249  0 
>> i2c_core               24513  1 i2c_dev
> 
> try for fun to not use i2c for a while
> 
>> microcode              11489  0 
> same for microcode... try removing that so that the microcode of your
> system doesn't get updated at boot

What do these two modules do in particular? and how can I disable
them so that they don't get reloaded during boot time? do I need
to disable both i2c_dev and i2c_core or just one of them?

Thanks.


-ammar

Re: kernel BUG at mm/rmap.c:483!

[Ammar T. Al-Sayegh]

----- Original Message ----- 
From: "Hugh Dickins" <hugh@veritas.com>
To: "Ammar T. Al-Sayegh" <ammar@kunet.com>
Cc: <linux-kernel@vger.kernel.org>
Sent: Wednesday, February 23, 2005 4:31 PM
Subject: Re: kernel BUG at mm/rmap.c:483!


> On Wed, 23 Feb 2005, Ammar T. Al-Sayegh wrote:
>> 
>> Any suggestion on what else I can do to mitigate this
>> problem?
> 
> The first thing to do is to give memtest86 a good (say
> overnight) run.  Many of the rmap.c BUG reporters have
> subsequently found memtest86 failures, and we believe those
> instances are accounted for by bad memory.  And if that's so
> in your case, you don't really want to be running 2.4 on it.
> 
> But not all cases could be accounted in that way.  If you
> report back that memtest86 ran cleanly, then I'll have to
> rework a debug patch against your Fedora RC3 kernel to try
> to give us more info - though quite possibly you cannot afford
> such experiments on this server, and will revert to 2.4 for now.

The problem is that my server is already in production
mode. I'm running great portion of my business on it,
where there is very little tolerance for downtime.
Because the server is located in a remote datacenter,
every time it goes down it takes several hours to have
someone sent up there to manually reboot it for a hefty
emergency fee. So this bug has already cost me a lot of
money, and I'm worried that it will cost me a lot of my
clients as well if it persists.

Remote hands are rather expensive, so it will cost me
$100/hr to have someone runs memtest86 on my server
since I can't perform it remotely. I'll do it though
since that's your recommendation for the time being.
Hope it will not take more than an hour to run the
test, and hope it turns out as bad memory modules as
you expect because I hate to downgrade after all the
time and money I expended on the upgrade.


-ammar

Re: kernel BUG at mm/rmap.c:483!

[Arjan van de Ven]

On Wed, 2005-02-23 at 16:45 -0500, Ammar T. Al-Sayegh wrote:
> > On Wed, 2005-02-23 at 15:41 -0500, Ammar T. Al-Sayegh wrote:
> >> Hi All,
> >> 
> >> I recently installed Fedora RC3 on a new server.
> >> The kernel is 2.6.10-1.741_FC3smp. The server
> >> crashes every few days. When I examine /var/log/messages,
> >> I find the following line just before the crash:
> >> 
> >> Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
> >> Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
> >> 
> >> No further debug lines are given to diagnose the
> >> source of the 
> > no oops at all?
> 
> No. Is there a way to enable the kernel to give more
> diagnostic debug output next time this error happens?

not really; it was supposed to do that already

> i2c_dev                13249  0 
> i2c_core               24513  1 i2c_dev

try for fun to not use i2c for a while

> microcode              11489  0 
same for microcode... try removing that so that the microcode of your
system doesn't get updated at boot

Re: kernel BUG at mm/rmap.c:483!

[Ammar T. Al-Sayegh]

> On Wed, 2005-02-23 at 15:41 -0500, Ammar T. Al-Sayegh wrote:
>> Hi All,
>> 
>> I recently installed Fedora RC3 on a new server.
>> The kernel is 2.6.10-1.741_FC3smp. The server
>> crashes every few days. When I examine /var/log/messages,
>> I find the following line just before the crash:
>> 
>> Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
>> Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
>> 
>> No further debug lines are given to diagnose the
>> source of the 
> no oops at all?

No. Is there a way to enable the kernel to give more
diagnostic debug output next time this error happens?

Is there a way to at least make the server reboot itself
next time the kernel is alerted to this problem before
crashing? When the server is rebooted, it works fine for
few more days before encountering this problem and
crashing again.

 
> which modules are you using?

[root ~]# lsmod 
Module                  Size  Used by
ip_conntrack_ftp       76145  0 
md5                     8001  1 
ipv6                  236769  78 
autofs4                21829  0 
i2c_dev                13249  0 
i2c_core               24513  1 i2c_dev
sunrpc                135077  1 
ipt_REJECT             10561  2 
ipt_state               5825  79 
ip_conntrack           45317  2 ip_conntrack_ftp,ipt_state
iptable_filter          7489  1 
ip_tables              20929  3 ipt_REJECT,ipt_state,iptable_filter
microcode              11489  0 
dm_mod                 57925  0 
video                  19653  0 
button                 10577  0 
battery                13253  0 
ac                      8773  0 
uhci_hcd               33497  0 
ehci_hcd               33737  0 
e1000                  84629  0 
floppy                 56913  0 
ext3                  117961  6 
jbd                    57177  1 ext3
3w_xxxx                30561  0 
ata_piix               12485  7 
libata                 44101  1 ata_piix
sd_mod                 20545  9 
scsi_mod              116033  3 3w_xxxx,libata,sd_mod

Any clue?


-ammar

Re: kernel BUG at mm/rmap.c:483!

[Hugh Dickins]

On Wed, 23 Feb 2005, Ammar T. Al-Sayegh wrote:
> 
> I recently installed Fedora RC3 on a new server.
> The kernel is 2.6.10-1.741_FC3smp.

I can't really speak for Fedora RC3 kernels,
perhaps there's some special patch in there that happens to
trigger it for you, but certainly there have been occasional
other reports of this BUG with vanilla kernel.org kernels.

> The server
> crashes every few days. When I examine /var/log/messages,
> I find the following line just before the crash:
> 
> Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
> Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
> 
> No further debug lines are given to diagnose the
> source of the problem.

It's odd that you get no more lines, but it doesn't really
matter in this case.  Sadly, the debug info accompanying this
BUG has done very little to shed light on its causes (and it's
on my todo list to change it to something less of a hindrance).

> I have been using kernel 2.4 for few years now without
> any problem. This is the first time I see this problem
> with kernel 2.6. I'm not sure if this is related to
> the kernel itself, the new hardware, or some other
> installed software. I'm thinking about downgrading to
> kernel 2.4. Do you think this will resolve this issue?

Downgrading to 2.4 will most certainly stop that particular
BUG, since 2.4 has no equivalent.  But it won't necessarily
fix the underlying issue.

> Any suggestion on what else I can do to mitigate this
> problem?

The first thing to do is to give memtest86 a good (say
overnight) run.  Many of the rmap.c BUG reporters have
subsequently found memtest86 failures, and we believe those
instances are accounted for by bad memory.  And if that's so
in your case, you don't really want to be running 2.4 on it.

But not all cases could be accounted in that way.  If you
report back that memtest86 ran cleanly, then I'll have to
rework a debug patch against your Fedora RC3 kernel to try
to give us more info - though quite possibly you cannot afford
such experiments on this server, and will revert to 2.4 for now.

Hugh

Re: kernel BUG at mm/rmap.c:483!

[Arjan van de Ven]

On Wed, 2005-02-23 at 15:41 -0500, Ammar T. Al-Sayegh wrote:
> Hi All,
> 
> I recently installed Fedora RC3 on a new server.
> The kernel is 2.6.10-1.741_FC3smp. The server
> crashes every few days. When I examine /var/log/messages,
> I find the following line just before the crash:
> 
> Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
> Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!
> 
> No further debug lines are given to diagnose the
> source of the 
no oops at all?

which modules are you using?

kernel BUG at mm/rmap.c:483!

[Ammar T. Al-Sayegh]

Hi All,

I recently installed Fedora RC3 on a new server.
The kernel is 2.6.10-1.741_FC3smp. The server
crashes every few days. When I examine /var/log/messages,
I find the following line just before the crash:

Feb 22 23:50:35 hostname kernel: ------------[ cut here ]------------
Feb 22 23:50:35 hostname kernel: kernel BUG at mm/rmap.c:483!

No further debug lines are given to diagnose the
source of the problem.

I have been using kernel 2.4 for few years now without
any problem. This is the first time I see this problem
with kernel 2.6. I'm not sure if this is related to
the kernel itself, the new hardware, or some other
installed software. I'm thinking about downgrading to
kernel 2.4. Do you think this will resolve this issue?
Any suggestion on what else I can do to mitigate this
problem?

Thanks.


-ammar