Re: VM: Bad swap entry 0044cb00

Re: VM: Bad swap entry 0044cb00

[Christian Borntraeger]

After some debug if think that the function which was called is:

zap_page_range

which inlines: zap_pmd_range
which inlines: zap_pte_range
which inlines: free_pte
which calls: swap_free
Adress 30d00 is part of the call to swap_free.



--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507

Re: VM: Bad swap entry 0044cb00

[Christian Borntraeger]

> But keep yesterday's testing patch in too, in case you find
> problems still (as Adrian Bunk reported on 2.4.9-ac2).

Hi Hugh,

as we still have to stuck with 2.4.7 I applied your patch, but the problem
isn´t fully solved. The error message still appears.
In some minutes I am going to vacation so for further questions, write to
Martin Schwidefsky. (schwidefsky@de.ibm.com) and cc Ulrich Weigand
(Ulrich.Weigand@de.ibm.com).

Sorry.

Bytheway now the messages were:

Out of Memory: Killed process 7090 (_18411Y45_s).
swap_free from 80030d04: Bad swap entry 0a9ca600
Out of Memory: Killed process 7841 (_18411Y46_s).
Out of Memory: Killed process 7861 (_18411Y47_s).
Out of Memory: Killed process 7993 (_18411Y48_s).
Out of Memory: Killed process 8020 (_18411Y47_s).
Out of Memory: Killed process 8153 (_18411Y48_s).
Out of Memory: Killed process 8260 (_18411Y85_s).
Out of Memory: Killed process 8345 (_18411Y86_s).
Out of Memory: Killed process 8355 (_18411Y85_s).
Out of Memory: Killed process 8414 (_18411Y87_s).
Out of Memory: Killed process 8453 (_18411Y86_s).
Out of Memory: Killed process 8478 (_18411Y8_10_s).
Out of Memory: Killed process 8480 (_18411Y45_s).
Out of Memory: Killed process 8545 (_18411Y87_s).
Out of Memory: Killed process 8586 (_18411Y46_s).
Out of Memory: Killed process 8645 (_18423Y11_s).

30d04 seems to be the same call to swap_free as before.

Re: VM: Bad swap entry 0044cb00

[Christian Borntraeger]

Hi Hugh,

System.map:

000307c0 T copy_page_range
00030a9c T zap_page_range
00030da4 t follow_page
00030e70 T map_user_kiobuf

dmesg:

Out of Memory: Killed process 6667 (_18411Y45_s).
swap_free from 80030d00: Bad swap entry 0a2c2600
Out of Memory: Killed process 7374 (_18411Y46_s).
Out of Memory: Killed process 7401 (_18411Y85_s).
swap_free from 80030d00: Bad swap entry 0a0b0600
Out of Memory: Killed process 7589 (_18411Y86_s).
Out of Memory: Killed process 7611 (_18411Y47_s).
Out of Memory: Killed process 7778 (_18411Y48_s).
Out of Memory: Killed process 7791 (_18411Y87_s).
Out of Memory: Killed process 7804 (_18411Y45_s).
Out of Memory: Killed process 7935 (_18411Y8_10_s).
Out of Memory: Killed process 7957 (_18411Y46_s).
Out of Memory: Killed process 8077 (_18411Y85_s).
Out of Memory: Killed process 8154 (_18411Y47_s).
Out of Memory: Killed process 8154 (_18411Y47_s).

I will make do a listing of the kernel (S/390-assembler) of zap_page_range
and of possible find out the line of the C-Code.


Note: Don´t care about the 8 in front of the adress. That is
S/390-specific.

Thanks

--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507

Re: VM: Bad swap entry 0044cb00

[Hugh Dickins]

On Wed, 29 Aug 2001, Christian Borntraeger wrote:
> 00030a9c T zap_page_range
> 00030da4 t follow_page
> 
> Out of Memory: Killed process 6667 (_18411Y45_s).
> swap_free from 80030d00: Bad swap entry 0a2c2600
> Out of Memory: Killed process 7374 (_18411Y46_s).
> Out of Memory: Killed process 7401 (_18411Y85_s).
> swap_free from 80030d00: Bad swap entry 0a0b0600
> 
> I will make do a listing of the kernel (S/390-assembler) of
> zap_page_range and if possible find out the line of the C-Code.

No need to look for that line, swap_free() is indeed called from
free_pte(), nested several inlines down within zap_page_range().

That was very helpful info, thank you: it helped to concentrate me
to realize my own stupidity.  You made it perfectly clear that you
are using 2.4.7, therefore (unless you already applied it by hand)
you don't have the very relevant 2.4.8 patch to do_swap_page() by
Jeremy Linton and Linus.

If you'd prefer not to upgrade to 2.4.8 or 2.4.9 right now, please
apply patch below and test again.  It should fix some of your "Out
of Memory" kills (which may have been mistaken, see first pte_same
test below, from Jeremy), and hopefully stop your "Bad swap entry"s
(see second pte_same test below, from Linus): if not present but
not same, old code would have inserted wrong page in page table, and
swap_freed it, leading to error from the later swap_free in free_pte().

But keep yesterday's testing patch in too, in case you find
problems still (as Adrian Bunk reported on 2.4.9-ac2).

Hugh

--- linux-2.4.7/mm/memory.c	Thu Jul 12 17:05:22 2001
+++ linux/mm/memory.c	Wed Aug 29 14:19:39 2001
@@ -1091,9 +1091,10 @@
  */
 static int do_swap_page(struct mm_struct * mm,
 	struct vm_area_struct * vma, unsigned long address,
-	pte_t * page_table, swp_entry_t entry, int write_access)
+	pte_t * page_table, pte_t orig_pte, int write_access)
 {
 	struct page *page;
+	swp_entry_t entry = pte_to_swp_entry(orig_pte);
 	pte_t pte;
 
 	spin_unlock(&mm->page_table_lock);
@@ -1105,7 +1106,11 @@
 		unlock_kernel();
 		if (!page) {
 			spin_lock(&mm->page_table_lock);
-			return -1;
+			/*
+			 * Back out if somebody else faulted in this pte while
+			 * we released the page table lock.
+			 */
+			return pte_same(*page_table, orig_pte) ? -1 : 1;
 		}
 	}
 
@@ -1121,7 +1126,7 @@
 	 * released the page table lock.
 	 */
 	spin_lock(&mm->page_table_lock);
-	if (pte_present(*page_table)) {
+	if (!pte_same(*page_table, orig_pte)) {
 		UnlockPage(page);
 		page_cache_release(page);
 		return 1;
@@ -1284,7 +1289,7 @@
 		 */
 		if (pte_none(entry))
 			return do_no_page(mm, vma, address, write_access, pte);
-		return do_swap_page(mm, vma, address, pte, pte_to_swp_entry(entry), write_access);
+		return do_swap_page(mm, vma, address, pte, entry, write_access);
 	}
 
 	if (write_access) {

Re: VM: Bad swap entry 0044cb00

[Christian Borntraeger]

>If you're still getting such messages, please let me know and
>I'll send a test patch to make the message more informative.
>
>Is this an SMP machine?
>
>Hugh

I am also interested in getting this patch as I face the same problem on an
S/390, but with Kernel 2.4.7
It is a SMP machine.
I don´t know, if you can get informations about this issue from non-intel
platforms, but we should try.

--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507

Re: VM: Bad swap entry 0044cb00

[Hugh Dickins]

On Tue, 28 Aug 2001, Christian Borntraeger wrote:
> 
> I am also interested in getting this patch as I face the same problem on an
> S/390, but with Kernel 2.4.7
> It is a SMP machine.
> I don´t know, if you can get informations about this issue from non-intel
> platforms, but we should try.

Please try the (untested) patch below, which I hope will identify the
failing function, giving the return address back to its caller - please
mail me the message(s), and your System.map entries immediately before
and immediately after the "from" address reported.  Patch is based on
2.4.7, but should apply okay (at different offsets) to anything since.

Thanks,
Hugh

--- 247/mm/swapfile.c	Wed Jul  4 19:50:38 2001
+++ 247tmp/mm/swapfile.c	Tue Aug 28 15:37:26 2001
@@ -191,12 +191,12 @@
 	printk("swap_free: offset exceeds max\n");
 	goto out;
 bad_free:
-	printk("VM: Bad swap entry %08lx\n", entry.val);
+	printk("swap_free from %p: Bad swap entry %08lx\n", __builtin_return_address(0), entry.val);
 	goto out;
 bad_count:
 	swap_device_unlock(p);
 	swap_list_unlock();
-	printk(KERN_ERR "VM: Bad count %hd current count %hd\n", count, p->swap_map[offset]);
+	printk(KERN_ERR "swap_free: Bad count %hd current count %hd\n", count, p->swap_map[offset]);
 	goto out;
 }
 
@@ -867,7 +867,7 @@
 	else {
 		static int overflow = 0;
 		if (overflow++ < 5)
-			printk("VM: swap entry overflow\n");
+			printk("swap_dup: swap entry overflow\n");
 		p->swap_map[offset] = SWAP_MAP_MAX;
 	}
 	swap_device_unlock(p);
@@ -876,13 +876,13 @@
 	return result;
 
 bad_file:
-	printk("Bad swap file entry %08lx\n", entry.val);
+	printk("swap_dup: Bad swap file entry %08lx\n", entry.val);
 	goto out;
 bad_offset:
-	printk("Bad swap offset entry %08lx\n", entry.val);
+	printk("swap_dup: Bad swap offset entry %08lx\n", entry.val);
 	goto out;
 bad_unused:
-	printk("Unused swap offset entry in swap_dup %08lx\n", entry.val);
+	printk("swap_dup from %p: Unused swap offset entry %08lx\n", __builtin_return_address(0), entry.val);
 	goto out;
 }
 
@@ -917,13 +917,13 @@
 	printk(KERN_ERR "swap_count: null entry!\n");
 	goto out;
 bad_file:
-	printk("Bad swap file entry %08lx\n", entry.val);
+	printk("swap_count: Bad swap file entry %08lx\n", entry.val);
 	goto out;
 bad_offset:
-	printk("Bad swap offset entry %08lx\n", entry.val);
+	printk("swap_count: Bad swap offset entry %08lx\n", entry.val);
 	goto out;
 bad_unused:
-	printk("Unused swap offset entry in swap_count %08lx\n", entry.val);
+	printk("swap_count from %p: Unused swap offset entry in swap_count %08lx\n", __builtin_return_address(0), entry.val);
 	goto out;
 }
 
@@ -938,7 +938,7 @@
 
 	type = SWP_TYPE(entry);
 	if (type >= nr_swapfiles) {
-		printk("Internal error: bad swap-device\n");
+		printk("rw_swap_page: Internal error: bad swap-device\n");
 		return;
 	}
 
@@ -949,7 +949,7 @@
 		return;
 	}
 	if (p->swap_map && !p->swap_map[*offset]) {
-		printk("VM: Bad swap entry %08lx\n", entry.val);
+		printk("rw_swap_page from %p: Bad swap entry %08lx\n", __builtin_return_address(0), entry.val);
 		return;
 	}
 	if (!(p->flags & SWP_USED)) {

re: VM: Bad swap entry 0044cb00

[Christian Borntraeger]

Hello,

I faced the same problem with the VM System on an S/390-System and Kernel
2.4.7 + S/390 patches. As there is hardware ECC-checking it is not a memory
problem.
I can reproduce the problem If I put load on the system (256 MB RAM, 512 MB
SWAP on a file, 3CPUs).

To reproduce this behaviour I run a lot of programs which consume all the
heap memory.
The C++ program uses the new function until the OOM killer is activated.

I start several of this programs with nohup prog &

This behaviour startet with 2.4.7.
With all previous Kernel versions since 2.4.0 there were a kernel BUG
messages during this test. The messages changed regularly with every new
kernel version, unfortunately we were not able to track this problem down.
Kernel 2.2 runs fine.

--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507

re: VM: Bad swap entry 0044cb00

[Christian Borntraeger]

Hello,

I faced the same problem with the VM System on an S/390-System and Kernel
2.4.7 + S/390 patches. As there is hardware ECC-checking it is
__definitely__ not a memory problem.
I can reproduce the problem If I put load on the system (256 MB RAM, 512 MB
SWAP on a file, 3CPUs).

To reproduce this behaviour I run a lot of programs which consume all the
heap memory.
The C++ program uses the new function until the OOM killer is activated.

I start several of this programs with nohup prog &

This behaviour startet with 2.4.7.
With all previous Kernel versions since 2.4.0 there were a kernel BUG
messages during this test. The messages changed regularly with every new
kernel version, unfortunately we were not able to track this problem down.
Kernel 2.2 runs fine.

--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507

VM: Bad swap entry 0044cb00

[Adrian Bunk]

Hi,

I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
the following message in my syslog file (I've never seen something like
this before):

Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00


What does this mean (my machine seems to run fine)?


TIA
Adrian

Re: VM: Bad swap entry 0044cb00

[Marcelo Tosatti]

On Tue, 28 Aug 2001, Adrian Bunk wrote:

> Hi,
> 
> I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
> the following message in my syslog file (I've never seen something like
> this before):
> 
> Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00
> 
> 
> What does this mean (my machine seems to run fine)?

Did you turned any swap file/partition off? (with swapoff) 

Re: VM: Bad swap entry 0044cb00

[Adrian Bunk]

On Mon, 27 Aug 2001, Marcelo Tosatti wrote:

> > I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
> > the following message in my syslog file (I've never seen something like
> > this before):
> >
> > Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00
> >
> >
> > What does this mean (my machine seems to run fine)?
>
> Did you turned any swap file/partition off? (with swapoff)

No.

cu
Adrian

-- 

Get my GPG key: finger bunk@debian.org | gpg --import

Fingerprint: B29C E71E FE19 6755 5C8A  84D4 99FC EA98 4F12 B400

Re: VM: Bad swap entry 0044cb00

[John Levon]

On Mon, Aug 27, 2001 at 07:46:56PM -0300, Marcelo Tosatti wrote:

> > I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
> > the following message in my syslog file (I've never seen something like
> > this before):
> > 
> > Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00
> > 
> > 
> > What does this mean (my machine seems to run fine)?
> 
> Did you turned any swap file/partition off? (with swapoff) 

fwiw, this seems to be a common characteristic of hardware problems with
2.4 kernels. I've had bad RAM (discovered by memtest86) which was producing
this error without any swapoff. Once it only occurred after an entire night
of kernel compiles (memtest86 found it easily however)

regards
john

-- 
"Premature generalization is the root of all evil."
	- Karl Fogel

Re: VM: Bad swap entry 0044cb00

[Marcelo Tosatti]

On Tue, 28 Aug 2001, John Levon wrote:

> On Mon, Aug 27, 2001 at 07:46:56PM -0300, Marcelo Tosatti wrote:
> 
> > > I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
> > > the following message in my syslog file (I've never seen something like
> > > this before):
> > > 
> > > Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00
> > > 
> > > 
> > > What does this mean (my machine seems to run fine)?
> > 
> > Did you turned any swap file/partition off? (with swapoff) 
> 
> fwiw, this seems to be a common characteristic of hardware problems with
> 2.4 kernels. I've had bad RAM (discovered by memtest86) which was producing
> this error without any swapoff. Once it only occurred after an entire night
> of kernel compiles (memtest86 found it easily however)

It may well be a memory problem (Andrian, can you run memtest86 so we can
confirm that). 

OTOH, Hugh Dickins has changed the swapin code lately to fix similar
problems and there is one user still reporting SIGSEGV's under stress
testing. 

Re: VM: Bad swap entry 0044cb00

[Alan Cox]

> fwiw, this seems to be a common characteristic of hardware problems with
> 2.4 kernels. I've had bad RAM (discovered by memtest86) which was producing
> this error without any swapoff. Once it only occurred after an entire night
> of kernel compiles (memtest86 found it easily however)

Worth testing maybe, but the -ac tree does have memory changes although not
between 8ac12 and 9ac2.

Re: VM: Bad swap entry 0044cb00

[Adrian Bunk]

On Tue, 28 Aug 2001, John Levon wrote:

> fwiw, this seems to be a common characteristic of hardware problems with
> 2.4 kernels. I've had bad RAM (discovered by memtest86) which was producing
> this error without any swapoff. Once it only occurred after an entire night
> of kernel compiles (memtest86 found it easily however)

memtest86 didn't find any errors in my RAM.

> regards
> john

cu
Adrian

-- 

Get my GPG key: finger bunk@debian.org | gpg --import

Fingerprint: B29C E71E FE19 6755 5C8A  84D4 99FC EA98 4F12 B400

Re: VM: Bad swap entry 0044cb00

[Alan Cox]

> I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
> the following message in my syslog file (I've never seen something like
> this before):
> 
> Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00
> What does this mean (my machine seems to run fine)?

It I suspect means that Rik still has work to do. 

Re: VM: Bad swap entry 0044cb00

[Hugh Dickins]

On Tue, 28 Aug 2001, Adrian Bunk wrote:
> 
> I upgraded my kernel from 2.4.8ac10 to 2.4.9ac2 some hours ago and I found
> the following message in my syslog file (I've never seen something like
> this before):
> 
> Aug 27 22:40:46 r063144 kernel: VM: Bad swap entry 0044cb00
> 
> What does this mean (my machine seems to run fine)?

If you only get such a message occasionally, it probably indicates
some race in the swapin code; probably not a new problem, but one now
made more visible by Rik's vm_swap_full swap deletion (as Alan hinted).
The race may well be benign.

But I'm guessing: I ought to understand this, but I don't.

The message I would expect you to get occasionally is the equivalent
message from swap_duplicate(), but that would be differently worded
("Unused swap offset entry in swap_dup 0044cb00").  The one you see
either comes from __swap_free() or from get_swaphandle_info():
I wonder which?

If you're still getting such messages, please let me know and
I'll send a test patch to make the message more informative.

Is this an SMP machine?

Hugh

Re: VM: Bad swap entry 0044cb00

[Adrian Bunk]

On Tue, 28 Aug 2001, Hugh Dickins wrote:

>...
> If you're still getting such messages, please let me know and
> I'll send a test patch to make the message more informative.

I'll try to reproduce it (it didn't appear again).

> Is this an SMP machine?

No.

> Hugh

cu
Adrian

-- 

Get my GPG key: finger bunk@debian.org | gpg --import

Fingerprint: B29C E71E FE19 6755 5C8A  84D4 99FC EA98 4F12 B400