linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH V5r1 0/6] staging: ramster: multi-machine memory capacity management
@ 2012-02-15 15:54 Dan Magenheimer
  2012-02-15 15:54 ` [PATCH 1/6] staging: ramster: cluster/messaging foundation Dan Magenheimer
                   ` (5 more replies)
  0 siblings, 6 replies; 10+ messages in thread
From: Dan Magenheimer @ 2012-02-15 15:54 UTC (permalink / raw)
  To: devel, linux-kernel, gregkh, linux-mm, dan.magenheimer

[Note: identical to V5 but rebased to staging-next and patchset threaded,
 due to gregkh request... sorry for the noise and thanks for your patience!]

HIGH LEVEL OVERVIEW

RAMster implements peer-to-peer transcendent memory, allowing a "cluster" of
kernels to dynamically pool their RAM so that a RAM-hungry workload on one
machine can temporarily and transparently utilize RAM on another machine which
is presumably idle or running a non-RAM-hungry workload.  Other than the
already-merged cleancache patchset and the not-yet-merged frontswap patchset,
no core kernel changes are currently required.

(Note that, unlike previous public descriptions of RAMster, this implementation
does NOT require synchronous "gets" or core networking changes. As of V5,
it also co-exists with ocfs2.)

RAMster combines a clustering and messaging foundation based on the ocfs2
cluster layer with the in-kernel compression implementation of zcache, and
adds code to glue them together.  When a page is "put" to RAMster, it is
compressed and stored locally.  Periodically, a thread will "remotify" these
pages by sending them via messages to a remote machine.  When the page is
later needed as indicated by a page fault, a "get" is issued.  If the data
is local, it is uncompressed and the fault is resolved.  If the data is
remote, a message is sent to fetch the data and the faulting thread sleeps;
when the data arrives, the thread awakens, the data is decompressed and
the fault is resolved.

As of V5, clusters up to eight nodes are supported; each node can remotify
pages to one specified node, so clusters can be configured as clients to
a "memory server".  Some simple policy is in place that will need to be
refined over time.  Larger clusters and fault-resistant protocols can also
be added over time.

A git branch containing these patches built on linux-3.2 can be found at:
git://oss.oracle.com/git/djm/tmem.git #ramster-v5
Note that that tree also includes frontswap-v11 and "WasActive" patches

A HOW-TO is available at:
http://oss.oracle.com/projects/tmem/dist/files/RAMster/HOWTO-v5-120214

v4->v5: support multi-node clusters (up to 8 nodes)
v4->v5: add settable to choose remotify target node for memory server config
v4->v5: incorporate ocfs2 cluster layer directly to allow co-exist with ocfs2
v4->v5: incorporate xvmalloc directly to avoid upstream zsmalloc conflicts
v4->v5: support ramster-tools (instead of ocfs2-tools) in userland
v3->v4: rebase to 3.2 (including updates in zcache)
v3->v4: fix a couple of bad memory leaks to get cleancache fully working
v3->v4: fix preemption calls to remove dependency on CONFIG_PREEMPT_NONE
v3->v4: various cleanup
v2->v3: documentation and commit message changes required [gregkh]

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>


---

Diffstat:

 drivers/staging/Kconfig                            |    2 +
 drivers/staging/Makefile                           |    1 +
 drivers/staging/ramster/Kconfig                    |   13 +
 drivers/staging/ramster/Makefile                   |    1 +
 drivers/staging/ramster/TODO                       |   13 +
 drivers/staging/ramster/cluster/Makefile           |    3 +
 drivers/staging/ramster/cluster/heartbeat.c        |  464 +++
 drivers/staging/ramster/cluster/heartbeat.h        |   87 +
 drivers/staging/ramster/cluster/masklog.c          |  155 +
 drivers/staging/ramster/cluster/masklog.h          |  220 ++
 drivers/staging/ramster/cluster/nodemanager.c      |  992 ++++++
 drivers/staging/ramster/cluster/nodemanager.h      |   88 +
 .../staging/ramster/cluster/ramster_nodemanager.h  |   39 +
 drivers/staging/ramster/cluster/tcp.c              | 2256 +++++++++++++
 drivers/staging/ramster/cluster/tcp.h              |  159 +
 drivers/staging/ramster/cluster/tcp_internal.h     |  248 ++
 drivers/staging/ramster/r2net.c                    |  401 +++
 drivers/staging/ramster/ramster.h                  |  118 +
 drivers/staging/ramster/tmem.c                     |  851 +++++
 drivers/staging/ramster/tmem.h                     |  244 ++
 drivers/staging/ramster/xvmalloc.c                 |  510 +++
 drivers/staging/ramster/xvmalloc.h                 |   30 +
 drivers/staging/ramster/xvmalloc_int.h             |   95 +
 drivers/staging/ramster/zcache-main.c              | 3320 ++++++++++++++++++++
 drivers/staging/ramster/zcache.h                   |   22 +
 25 files changed, 10332 insertions(+), 0 deletions(-)

^ permalink raw reply	[flat|nested] 10+ messages in thread
* [PATCH 6/6] staging: ramster: enable as staging driver
@ 2012-02-14 23:36 Dan Magenheimer
  0 siblings, 0 replies; 10+ messages in thread
From: Dan Magenheimer @ 2012-02-14 23:36 UTC (permalink / raw)
  To: devel, linux-kernel, gregkh, linux-mm, ngupta, konrad.wilk,
	kurt.hackel, sjenning, chris.mason, dan.magenheimer

RAMster implements peer-to-peer transcendent memory, allowing a "cluster"
of kernels to dynamically pool their RAM.

Enable build of ramster as a staging driver

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
---
 drivers/staging/Kconfig  |    2 ++
 drivers/staging/Makefile |    1 +
 2 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/staging/Kconfig b/drivers/staging/Kconfig
index 25cdff3..5f7b5eb 100644
--- a/drivers/staging/Kconfig
+++ b/drivers/staging/Kconfig
@@ -132,4 +132,6 @@ source "drivers/staging/nvec/Kconfig"
 
 source "drivers/staging/media/Kconfig"
 
+source "drivers/staging/ramster/Kconfig"
+
 endif # STAGING
diff --git a/drivers/staging/Makefile b/drivers/staging/Makefile
index a25f3f2..3a7b2f6 100644
--- a/drivers/staging/Makefile
+++ b/drivers/staging/Makefile
@@ -57,3 +57,4 @@ obj-$(CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4)	+= ste_rmi4/
 obj-$(CONFIG_DRM_PSB)		+= gma500/
 obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_MFD_NVEC)		+= nvec/
+obj-$(CONFIG_RAMSTER)		+= ramster/
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread
* [PATCH 6/6] staging: ramster: enable as staging driver
@ 2012-01-30 22:39 Dan Magenheimer
  0 siblings, 0 replies; 10+ messages in thread
From: Dan Magenheimer @ 2012-01-30 22:39 UTC (permalink / raw)
  To: devel, linux-kernel, gregkh, linux-mm, ngupta, konrad.wilk,
	kurt.hackel, sjenning, chris.mason, dan.magenheimer

Enable build of ramster as a staging driver

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
---
 drivers/staging/Kconfig  |    2 ++
 drivers/staging/Makefile |    1 +
 2 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/staging/Kconfig b/drivers/staging/Kconfig
index 25cdff3..5f7b5eb 100644
--- a/drivers/staging/Kconfig
+++ b/drivers/staging/Kconfig
@@ -132,4 +132,6 @@ source "drivers/staging/nvec/Kconfig"
 
 source "drivers/staging/media/Kconfig"
 
+source "drivers/staging/ramster/Kconfig"
+
 endif # STAGING
diff --git a/drivers/staging/Makefile b/drivers/staging/Makefile
index a25f3f2..3a7b2f6 100644
--- a/drivers/staging/Makefile
+++ b/drivers/staging/Makefile
@@ -57,3 +57,4 @@ obj-$(CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4)	+= ste_rmi4/
 obj-$(CONFIG_DRM_PSB)		+= gma500/
 obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_MFD_NVEC)		+= nvec/
+obj-$(CONFIG_RAMSTER)		+= ramster/
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread
* [PATCH 6/6] staging: ramster: enable as staging driver
@ 2011-12-27 18:50 Dan Magenheimer
  0 siblings, 0 replies; 10+ messages in thread
From: Dan Magenheimer @ 2011-12-27 18:50 UTC (permalink / raw)
  To: devel, linux-kernel, linux-mm, ngupta, konrad.wilk, kurt.hackel,
	sjenning, chris.mason, dan.magenheimer

Enable build of ramster as a staging driver

Signed-off-by: Dan Magenheimer <dan.magenheimer@oracle.com>
---
 drivers/staging/Kconfig  |    2 ++
 drivers/staging/Makefile |    1 +
 2 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/drivers/staging/Kconfig b/drivers/staging/Kconfig
index 06c9081..28819c3 100644
--- a/drivers/staging/Kconfig
+++ b/drivers/staging/Kconfig
@@ -150,4 +150,6 @@ source "drivers/staging/mei/Kconfig"
 
 source "drivers/staging/nvec/Kconfig"
 
+source "drivers/staging/ramster/Kconfig"
+
 endif # STAGING
diff --git a/drivers/staging/Makefile b/drivers/staging/Makefile
index f3c5e33..f1e6ec0 100644
--- a/drivers/staging/Makefile
+++ b/drivers/staging/Makefile
@@ -66,3 +66,4 @@ obj-$(CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4)	+= ste_rmi4/
 obj-$(CONFIG_DRM_PSB)		+= gma500/
 obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_MFD_NVEC)		+= nvec/
+obj-$(CONFIG_RAMSTER)		+= ramster/
-- 
1.7.1


^ permalink raw reply related	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2012-02-15 15:55 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-02-15 15:54 [PATCH V5r1 0/6] staging: ramster: multi-machine memory capacity management Dan Magenheimer
2012-02-15 15:54 ` [PATCH 1/6] staging: ramster: cluster/messaging foundation Dan Magenheimer
2012-02-15 15:54 ` [PATCH 2/6] staging: ramster: local compression + tmem Dan Magenheimer
2012-02-15 15:54 ` [PATCH 3/6] staging: ramster: xvmalloc allocation files Dan Magenheimer
2012-02-15 15:54 ` [PATCH 4/6] staging: ramster: ramster-specific changes to zcache/tmem Dan Magenheimer
2012-02-15 15:54 ` [PATCH 5/6] staging: ramster: ramster-specific new files Dan Magenheimer
2012-02-15 15:54 ` [PATCH 6/6] staging: ramster: enable as staging driver Dan Magenheimer
  -- strict thread matches above, loose matches on Subject: below --
2012-02-14 23:36 Dan Magenheimer
2012-01-30 22:39 Dan Magenheimer
2011-12-27 18:50 Dan Magenheimer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).