From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759342Ab1FAR3k (ORCPT ); Wed, 1 Jun 2011 13:29:40 -0400 Received: from smtp107.prem.mail.ac4.yahoo.com ([76.13.13.46]:30867 "HELO smtp107.prem.mail.ac4.yahoo.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1759055Ab1FAR0Q (ORCPT ); Wed, 1 Jun 2011 13:26:16 -0400 X-Yahoo-SMTP: _Dag8S.swBC1p4FJKLCXbs8NQzyse1SYSgnAbY0- X-YMail-OSG: ASVIiJcVM1k_9YyHJhNvU5TOrJn6Kk2NXeK5BexhK9zUv48 lkRUlIjYE4s1N_8XVE0Wh67eMo71pUMNg8Dqs9ScHobdF5sU922Aol0fS583 cZ.vvtX6Vyh6FHR2smuC7JS.Ay5nb4RMhLNlFp6vFEiVXRwmmua6K976xLqT VfCqUrCWDh0n6nTBFwQGDSpQYoBFdS6QuBTIWe2MLYSAsf3tv9e1GTyRvxw6 5w987Ud5ToWo6diuRpytwtEk0FYg7wxUfZrMahd_MpavDnlwnb1vK1h47Hr6 L6kXVPHugY_INTlVCyEnGNtmG3gFQ31Rr6oP.aVgp63ILG5sj X-Yahoo-Newman-Property: ymail-3 Message-Id: <20110601172614.725685218@linux.com> User-Agent: quilt/0.48-1 Date: Wed, 01 Jun 2011 12:25:48 -0500 From: Christoph Lameter To: Pekka Enberg Cc: David Rientjes Cc: Eric Dumazet Cc: "H. Peter Anvin" Cc: linux-kernel@vger.kernel.org Cc: Thomas Gleixner Subject: [slubllv7 05/17] mm: Rearrange struct page References: <20110601172543.437240675@linux.com> Content-Disposition: inline; filename=resort_struct_page Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We need to be able to use cmpxchg_double on the freelist and object count field in struct page. Rearrange the fields in struct page according to doubleword entities so that the freelist pointer comes before the counters. Do the rearranging with a future in mind where we use more doubleword atomics to avoid locking of updates to flags/mapping or lru pointers. Create another union to allow access to counters in struct page as a single unsigned long value. The doublewords must be properly aligned for cmpxchg_double to work. Sadly this increases the size of page struct by one word on some architectures. But as a resultpage structs are now cacheline aligned on x86_64. Signed-off-by: Christoph Lameter --- include/linux/mm_types.h | 89 +++++++++++++++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 29 deletions(-) Index: linux-2.6/include/linux/mm_types.h =================================================================== --- linux-2.6.orig/include/linux/mm_types.h 2011-05-31 09:46:41.912987862 -0500 +++ linux-2.6/include/linux/mm_types.h 2011-05-31 09:46:44.282987846 -0500 @@ -30,52 +30,74 @@ struct address_space; * moment. Note that we have no way to track which tasks are using * a page, though if it is a pagecache page, rmap structures can tell us * who is mapping it. + * + * The objects in struct page are organized in double word blocks in + * order to allows us to use atomic double word operations on portions + * of struct page. That is currently only used by slub but the arrangement + * allows the use of atomic double word operations on the flags/mapping + * and lru list pointers also. */ struct page { + /* First double word block */ unsigned long flags; /* Atomic flags, some possibly * updated asynchronously */ - atomic_t _count; /* Usage count, see below. */ + struct address_space *mapping; /* If low bit clear, points to + * inode address_space, or NULL. + * If page mapped as anonymous + * memory, low bit is set, and + * it points to anon_vma object: + * see PAGE_MAPPING_ANON below. + */ + /* Second double word */ union { - atomic_t _mapcount; /* Count of ptes mapped in mms, - * to show when page is mapped - * & limit reverse map searches. + struct { + pgoff_t index; /* Our offset within mapping. */ + atomic_t _mapcount; /* Count of ptes mapped in mms, + * to show when page is mapped + * & limit reverse map searches. + */ + atomic_t _count; /* Usage count, see below. */ + }; + + struct { /* SLUB cmpxchg_double area */ + void *freelist; + union { + unsigned long counters; + struct { + unsigned inuse:16; + unsigned objects:15; + unsigned frozen:1; + /* + * Kernel may make use of this field even when slub + * uses the rest of the double word! */ - struct { /* SLUB */ - unsigned inuse:16; - unsigned objects:15; - unsigned frozen:1; + atomic_t _count; + }; + }; }; }; + + /* Third double word block */ + struct list_head lru; /* Pageout list, eg. active_list + * protected by zone->lru_lock ! + */ + + /* Remainder is not double word aligned */ union { - struct { - unsigned long private; /* Mapping-private opaque data: + unsigned long private; /* Mapping-private opaque data: * usually used for buffer_heads * if PagePrivate set; used for * swp_entry_t if PageSwapCache; * indicates order in the buddy * system if PG_buddy is set. */ - struct address_space *mapping; /* If low bit clear, points to - * inode address_space, or NULL. - * If page mapped as anonymous - * memory, low bit is set, and - * it points to anon_vma object: - * see PAGE_MAPPING_ANON below. - */ - }; #if USE_SPLIT_PTLOCKS - spinlock_t ptl; + spinlock_t ptl; #endif - struct kmem_cache *slab; /* SLUB: Pointer to slab */ - struct page *first_page; /* Compound tail pages */ + struct kmem_cache *slab; /* SLUB: Pointer to slab */ + struct page *first_page; /* Compound tail pages */ }; - union { - pgoff_t index; /* Our offset within mapping. */ - void *freelist; /* SLUB: freelist req. slab lock */ - }; - struct list_head lru; /* Pageout list, eg. active_list - * protected by zone->lru_lock ! - */ + /* * On machines where all RAM is mapped into kernel address space, * we can simply calculate the virtual address. On machines with @@ -101,7 +123,16 @@ struct page { */ void *shadow; #endif -}; +} +/* + * If another subsystem starts using the double word pairing for atomic + * operations on struct page then it must change the #if to ensure + * proper alignment of the page struct. + */ +#if defined(CONFIG_SLUB) && defined(CONFIG_CMPXCHG_LOCAL) + __attribute__((__aligned__(2*sizeof(unsigned long)))) +#endif +; typedef unsigned long __nocast vm_flags_t;