[PATCH 0 of 9] xenpaging fixes for xen-unstable

[PATCH 0 of 9] xenpaging fixes for xen-unstable

[Olaf Hering]

The following series implements growing and shrinking of the paging
file.  A xenstore watch is used the retreive the new number of pages to
page-out. This number is compared to the current amount of paged-out
pages and, depending on the difference, new pages are written to disk or
written back into the guest.

Please review and apply.

Olaf


 tools/xenpaging/Makefile         |    4 
 tools/xenpaging/file_ops.c       |    1 
 tools/xenpaging/policy_default.c |   11 -
 tools/xenpaging/xenpaging.c      |  231 +++++++++++++++++++++++++++------------
 tools/xenpaging/xenpaging.h      |    8 -
 5 files changed, 175 insertions(+), 80 deletions(-)

[PATCH 1 of 9] xenpaging: install into LIBEXEC dir

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067221 -7200
# Node ID cd07fbf47c11055a30ba21e904bf5537c50b044b
# Parent  e90438f6e6d1585a71b18784a99c162b5d95f390
xenpaging: install into LIBEXEC dir

In preparation of upcoming libxl integration,
move xenpaging binary from /usr/sbin/ to /usr/lib/xen/bin/

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r e90438f6e6d1 -r cd07fbf47c11 tools/xenpaging/Makefile
--- a/tools/xenpaging/Makefile
+++ b/tools/xenpaging/Makefile
@@ -24,8 +24,8 @@ xenpaging: $(OBJS)
 
 install: all
 	$(INSTALL_DIR) $(DESTDIR)/var/lib/xen/xenpaging
-	$(INSTALL_DIR) $(DESTDIR)$(SBINDIR)
-	$(INSTALL_PROG) $(IBINS) $(DESTDIR)$(SBINDIR)
+	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC)
+	$(INSTALL_PROG) $(IBINS) $(DESTDIR)$(LIBEXEC)
 
 clean:
 	rm -f *.o *~ $(DEPS) xen TAGS $(IBINS) $(LIB)

[PATCH 2 of 9] xenpaging: remove filename from comment

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067223 -7200
# Node ID eb968e957ee4789a7a06c5f5683eb4b0e49324cd
# Parent  cd07fbf47c11055a30ba21e904bf5537c50b044b
xenpaging: remove filename from comment

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r cd07fbf47c11 -r eb968e957ee4 tools/xenpaging/file_ops.c
--- a/tools/xenpaging/file_ops.c
+++ b/tools/xenpaging/file_ops.c
@@ -1,5 +1,4 @@
 /******************************************************************************
- * tools/xenpaging/file_ops.c
  *
  * Common file operations.
  *
diff -r cd07fbf47c11 -r eb968e957ee4 tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -1,5 +1,4 @@
 /******************************************************************************
- * tools/xenpaging/policy.c
  *
  * Xen domain paging default policy.
  *
diff -r cd07fbf47c11 -r eb968e957ee4 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -1,5 +1,4 @@
 /******************************************************************************
- * tools/xenpaging/xenpaging.c
  *
  * Domain paging. 
  * Copyright (c) 2009 by Citrix Systems, Inc. (Patrick Colp)

[PATCH 3 of 9] xenpaging: update xenpaging_init

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067225 -7200
# Node ID 1e5697849aa62c08651a613901dcf054ee652ea4
# Parent  eb968e957ee4789a7a06c5f5683eb4b0e49324cd
xenpaging: update xenpaging_init

Move comment about xc_handle to the right place.
Allocate paging early and use calloc.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r eb968e957ee4 -r 1e5697849aa6 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -169,18 +169,21 @@ static xenpaging_t *xenpaging_init(domid
     char *p;
     int rc;
 
+    /* Allocate memory */
+    paging = calloc(1, sizeof(xenpaging_t));
+    if ( !paging )
+        goto err;
+
     if ( getenv("XENPAGING_DEBUG") )
         dbg = (xentoollog_logger *)xtl_createlogger_stdiostream(stderr, XTL_DEBUG, 0);
-    xch = xc_interface_open(dbg, NULL, 0);
+
+    /* Open connection to xen */
+    paging->xc_handle = xch = xc_interface_open(dbg, NULL, 0);
     if ( !xch )
-        goto err_iface;
+        goto err;
 
     DPRINTF("xenpaging init\n");
 
-    /* Allocate memory */
-    paging = malloc(sizeof(xenpaging_t));
-    memset(paging, 0, sizeof(xenpaging_t));
-
     /* Open connection to xenstore */
     paging->xs_handle = xs_open(0);
     if ( paging->xs_handle == NULL )
@@ -204,9 +207,6 @@ static xenpaging_t *xenpaging_init(domid
          DPRINTF("Setting policy mru_size to %d\n", paging->policy_mru_size);
     }
 
-    /* Open connection to xen */
-    paging->xc_handle = xch;
-
     /* Set domain id */
     paging->mem_event.domain_id = domain_id;
 
@@ -319,7 +319,8 @@ static xenpaging_t *xenpaging_init(domid
     {
         if ( paging->xs_handle )
             xs_close(paging->xs_handle);
-        xc_interface_close(xch);
+        if ( xch )
+            xc_interface_close(xch);
         if ( paging->mem_event.shared_page )
         {
             munlock(paging->mem_event.shared_page, PAGE_SIZE);
@@ -337,7 +338,6 @@ static xenpaging_t *xenpaging_init(domid
         free(paging);
     }
 
- err_iface: 
     return NULL;
 }

[PATCH 4 of 9] xenpaging: remove xc_dominfo_t from paging_t

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067227 -7200
# Node ID 2ff9871c02a1712d55addc2b30e4f7694e5b7ebe
# Parent  1e5697849aa62c08651a613901dcf054ee652ea4
xenpaging: remove xc_dominfo_t from paging_t

Remove xc_dominfo_t from paging_t, record only max_pages.
This value is used to setup internal data structures.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 1e5697849aa6 -r 2ff9871c02a1 tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -41,17 +41,17 @@ int policy_init(xenpaging_t *paging)
     int i;
     int rc = -ENOMEM;
 
+    max_pages = paging->max_pages;
+
     /* Allocate bitmap for pages not to page out */
-    bitmap = bitmap_alloc(paging->domain_info->max_pages);
+    bitmap = bitmap_alloc(max_pages);
     if ( !bitmap )
         goto out;
     /* Allocate bitmap to track unusable pages */
-    unconsumed = bitmap_alloc(paging->domain_info->max_pages);
+    unconsumed = bitmap_alloc(max_pages);
     if ( !unconsumed )
         goto out;
 
-    max_pages = paging->domain_info->max_pages;
-
     /* Initialise MRU list of paged in pages */
     if ( paging->policy_mru_size > 0 )
         mru_size = paging->policy_mru_size;
diff -r 1e5697849aa6 -r 2ff9871c02a1 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -164,6 +164,7 @@ static void *init_page(void)
 static xenpaging_t *xenpaging_init(domid_t domain_id, int num_pages)
 {
     xenpaging_t *paging;
+    xc_domaininfo_t domain_info;
     xc_interface *xch;
     xentoollog_logger *dbg = NULL;
     char *p;
@@ -273,33 +274,29 @@ static xenpaging_t *xenpaging_init(domid
     paging->mem_event.port = rc;
 
     /* Get domaininfo */
-    paging->domain_info = malloc(sizeof(xc_domaininfo_t));
-    if ( paging->domain_info == NULL )
-    {
-        ERROR("Error allocating memory for domain info");
-        goto err;
-    }
-
     rc = xc_domain_getinfolist(xch, paging->mem_event.domain_id, 1,
-                               paging->domain_info);
+                               &domain_info);
     if ( rc != 1 )
     {
         ERROR("Error getting domain info");
         goto err;
     }
 
+    /* Record number of max_pages */
+    paging->max_pages = domain_info.max_pages;
+
     /* Allocate bitmap for tracking pages that have been paged out */
-    paging->bitmap = bitmap_alloc(paging->domain_info->max_pages);
+    paging->bitmap = bitmap_alloc(paging->max_pages);
     if ( !paging->bitmap )
     {
         ERROR("Error allocating bitmap");
         goto err;
     }
-    DPRINTF("max_pages = %"PRIx64"\n", paging->domain_info->max_pages);
+    DPRINTF("max_pages = %d\n", paging->max_pages);
 
-    if ( num_pages < 0 || num_pages > paging->domain_info->max_pages )
+    if ( num_pages < 0 || num_pages > paging->max_pages )
     {
-        num_pages = paging->domain_info->max_pages;
+        num_pages = paging->max_pages;
         DPRINTF("setting num_pages to %d\n", num_pages);
     }
     paging->num_pages = num_pages;
@@ -334,7 +331,6 @@ static xenpaging_t *xenpaging_init(domid
         }
 
         free(paging->bitmap);
-        free(paging->domain_info);
         free(paging);
     }
 
@@ -764,7 +760,7 @@ int main(int argc, char *argv[])
         if ( interrupted == SIGTERM || interrupted == SIGINT )
         {
             int num = 0;
-            for ( i = 0; i < paging->domain_info->max_pages; i++ )
+            for ( i = 0; i < paging->max_pages; i++ )
             {
                 if ( test_bit(i, paging->bitmap) )
                 {
@@ -780,7 +776,7 @@ int main(int argc, char *argv[])
              */
             if ( num )
                 page_in_trigger();
-            else if ( i == paging->domain_info->max_pages )
+            else if ( i == paging->max_pages )
                 break;
         }
         else
diff -r 1e5697849aa6 -r 2ff9871c02a1 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -44,11 +44,11 @@ typedef struct xenpaging {
     xc_interface *xc_handle;
     struct xs_handle *xs_handle;
 
-    xc_domaininfo_t    *domain_info;
-
     unsigned long *bitmap;
 
     mem_event_t mem_event;
+    /* number of pages for which data structures were allocated */
+    int max_pages;
     int num_pages;
     int policy_mru_size;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];

[PATCH 5 of 9] xenpaging: track the number of paged-out pages

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067229 -7200
# Node ID 6beca8cbc2c92900859712f8738db17084bcebdb
# Parent  2ff9871c02a1712d55addc2b30e4f7694e5b7ebe
xenpaging: track the number of paged-out pages

This change is required by subsequent changes.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 2ff9871c02a1 -r 6beca8cbc2c9 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -468,6 +468,9 @@ static int xenpaging_evict_page(xenpagin
     /* Notify policy of page being paged out */
     policy_notify_paged_out(victim->gfn);
 
+    /* Record number of evicted pages */
+    paging->num_paged_out++;
+
  out:
     return ret;
 }
@@ -481,8 +484,13 @@ static int xenpaging_resume_page(xenpagi
 
     /* Notify policy of page being paged in */
     if ( notify_policy )
+    {
         policy_notify_paged_in(rsp->gfn);
 
+       /* Record number of resumed pages */
+       paging->num_paged_out--;
+    }
+
     /* Tell Xen page is ready */
     ret = xc_mem_paging_resume(paging->xc_handle, paging->mem_event.domain_id,
                                rsp->gfn);
diff -r 2ff9871c02a1 -r 6beca8cbc2c9 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -49,6 +49,7 @@ typedef struct xenpaging {
     mem_event_t mem_event;
     /* number of pages for which data structures were allocated */
     int max_pages;
+    int num_paged_out;
     int num_pages;
     int policy_mru_size;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];

[PATCH 6 of 9] xenpaging: add evict_pages function

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067230 -7200
# Node ID 3a3a5979b799d948802183d10d65894ee84a872f
# Parent  6beca8cbc2c92900859712f8738db17084bcebdb
xenpaging: add evict_pages function

Add new function to evict a couple of pages.
Allocate all possible slots in a paging file to allow growing and
shrinking of the number of paged-out pages. Adjust other places to
iterate over all slots.

This change is required by subsequent patches.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 6beca8cbc2c9 -r 3a3a5979b799 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -597,6 +597,30 @@ static int evict_victim(xenpaging_t *pag
     return ret;
 }
 
+/* Evict a couple of pages and write them to a free slot in the paging file */
+static int evict_pages(xenpaging_t *paging, int fd, xenpaging_victim_t *victims, int num_pages)
+{
+    xc_interface *xch = paging->xc_handle;
+    int rc, slot, num = 0;
+
+    for ( slot = 0; slot < paging->max_pages && num < num_pages; slot++ )
+    {
+        /* Slot is allocated */
+        if ( victims[slot].gfn != INVALID_MFN )
+            continue;
+
+        rc = evict_victim(paging, &victims[slot], fd, slot);
+        if ( rc == -ENOSPC )
+            break;
+        if ( rc == -EINTR )
+            break;
+        if ( num && num % 100 == 0 )
+            DPRINTF("%d pages evicted\n", num);
+        num++;
+    }
+    return num;
+}
+
 int main(int argc, char *argv[])
 {
     struct sigaction act;
@@ -639,7 +663,12 @@ int main(int argc, char *argv[])
         return 2;
     }
 
-    victims = calloc(paging->num_pages, sizeof(xenpaging_victim_t));
+    /* Allocate upper limit of pages to allow growing and shrinking */
+    victims = calloc(paging->max_pages, sizeof(xenpaging_victim_t));
+
+    /* Mark all slots as unallocated */
+    for ( i = 0; i < paging->max_pages; i++ )
+        victims[i].gfn = INVALID_MFN;
 
     /* ensure that if we get a signal, we'll do cleanup, then exit */
     act.sa_handler = close_handler;
@@ -653,18 +682,7 @@ int main(int argc, char *argv[])
     /* listen for page-in events to stop pager */
     create_page_in_thread(paging);
 
-    /* Evict pages */
-    for ( i = 0; i < paging->num_pages; i++ )
-    {
-        rc = evict_victim(paging, &victims[i], fd, i);
-        if ( rc == -ENOSPC )
-            break;
-        if ( rc == -EINTR )
-            break;
-        if ( i % 100 == 0 )
-            DPRINTF("%d pages evicted\n", i);
-    }
-
+    i = evict_pages(paging, fd, victims, paging->num_pages);
     DPRINTF("%d pages evicted. Done.\n", i);
 
     /* Swap pages in and out */
@@ -690,13 +708,13 @@ int main(int argc, char *argv[])
             if ( test_and_clear_bit(req.gfn, paging->bitmap) )
             {
                 /* Find where in the paging file to read from */
-                for ( i = 0; i < paging->num_pages; i++ )
+                for ( i = 0; i < paging->max_pages; i++ )
                 {
                     if ( victims[i].gfn == req.gfn )
                         break;
                 }
     
-                if ( i >= paging->num_pages )
+                if ( i >= paging->max_pages )
                 {
                     DPRINTF("Couldn't find page %"PRIx64"\n", req.gfn);
                     goto out;

[PATCH 7 of 9] xenpaging: add resume_pages function

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067232 -7200
# Node ID 6987aa2dde4e93481f1599735ed1a26defb6d6b9
# Parent  3a3a5979b799d948802183d10d65894ee84a872f
xenpaging: add resume_pages function

Add new function to resume a couple of pages.
This change is required by subsequent patches.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 3a3a5979b799 -r 6987aa2dde4e tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -554,6 +554,27 @@ static int xenpaging_populate_page(xenpa
     return ret;
 }
 
+/* Trigger a page-in for a couple of pages */
+static void resume_pages(xenpaging_t *paging, int num_pages)
+{
+    xc_interface *xch = paging->xc_handle;
+    int i, num = 0;
+
+    for ( i = 0; i < paging->max_pages && num < num_pages; i++ )
+    {
+        if ( test_bit(i, paging->bitmap) )
+        {
+            paging->pagein_queue[num] = i;
+            num++;
+            if ( num == XENPAGING_PAGEIN_QUEUE_SIZE )
+                break;
+        }
+    }
+    /* num may be less than num_pages, caller has to try again */
+    if ( num )
+        page_in_trigger();
+}
+
 static int evict_victim(xenpaging_t *paging,
                         xenpaging_victim_t *victim, int fd, int i)
 {
@@ -785,25 +806,12 @@ int main(int argc, char *argv[])
         /* Write all pages back into the guest */
         if ( interrupted == SIGTERM || interrupted == SIGINT )
         {
-            int num = 0;
-            for ( i = 0; i < paging->max_pages; i++ )
-            {
-                if ( test_bit(i, paging->bitmap) )
-                {
-                    paging->pagein_queue[num] = i;
-                    num++;
-                    if ( num == XENPAGING_PAGEIN_QUEUE_SIZE )
-                        break;
-                }
-            }
-            /*
-             * One more round if there are still pages to process.
-             * If no more pages to process, exit loop.
-             */
-            if ( num )
-                page_in_trigger();
-            else if ( i == paging->max_pages )
+            /* If no more pages to process, exit loop. */
+            if ( !paging->num_paged_out )
                 break;
+            
+            /* One more round if there are still pages to process. */
+            resume_pages(paging, paging->num_paged_out);
         }
         else
         {

[PATCH 8 of 9] xenpaging: compare both token and path when checking for @releaseDomain event

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067234 -7200
# Node ID 1addb9e81178ea164ef516f340df2607fddcb930
# Parent  6987aa2dde4e93481f1599735ed1a26defb6d6b9
xenpaging: compare both token and path when checking for @releaseDomain event

Subsequent patches will use xenstored to store the numbers of pages
xenpaging is suppose to page-out. A domain_id value could be
misinterpreted as number of pages. Compare both path and token to
recognize the @releaseDomain event.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 6987aa2dde4e -r 1addb9e81178 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -101,7 +101,7 @@ static int xenpaging_wait_for_event_or_t
         vec = xs_read_watch(paging->xs_handle, &num);
         if ( vec )
         {
-            if ( strcmp(vec[XS_WATCH_TOKEN], watch_token) == 0 )
+            if ( strcmp(vec[XS_WATCH_PATH], "@releaseDomain") == 0 && strcmp(vec[XS_WATCH_TOKEN], watch_token) == 0 )
             {
                 /* If our guest disappeared, set interrupt flag and fall through */
                 if ( xs_is_domain_introduced(paging->xs_handle, paging->mem_event.domain_id) == false )

[PATCH 9 of 9] xenpaging: watch the domains /xenpaging/num_pages xenstore value

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1316067236 -7200
# Node ID ad903b34e0997e864539e36b92dde234f9ffbd55
# Parent  1addb9e81178ea164ef516f340df2607fddcb930
xenpaging: watch the domains /xenpaging/num_pages xenstore value

Subsequent patches will use xenstored to store the numbers of pages
xenpaging is suppose to page-out.
Remove num_pages and use target_pages instead.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 1addb9e81178 -r ad903b34e099 tools/xenpaging/policy_default.c
--- a/tools/xenpaging/policy_default.c
+++ b/tools/xenpaging/policy_default.c
@@ -70,7 +70,7 @@ int policy_init(xenpaging_t *paging)
 
     /* Start in the middle to avoid paging during BIOS startup */
     current_gfn = max_pages / 2;
-    current_gfn -= paging->num_pages / 2;
+    current_gfn -= paging->target_pages / 2;
 
     rc = 0;
  out:
diff -r 1addb9e81178 -r ad903b34e099 tools/xenpaging/xenpaging.c
--- a/tools/xenpaging/xenpaging.c
+++ b/tools/xenpaging/xenpaging.c
@@ -35,6 +35,9 @@
 #include "policy.h"
 #include "xenpaging.h"
 
+/* Defines number of pages to page-out */
+#define WATCH_TARGETPAGES "/xenpaging/num_pages"
+static char *watch_targetpages;
 static char watch_token[16];
 static char filename[80];
 static int interrupted;
@@ -72,7 +75,7 @@ static int xenpaging_wait_for_event_or_t
 {
     xc_interface *xch = paging->xc_handle;
     xc_evtchn *xce = paging->mem_event.xce_handle;
-    char **vec;
+    char **vec, *val;
     unsigned int num;
     struct pollfd fd[2];
     int port;
@@ -101,6 +104,7 @@ static int xenpaging_wait_for_event_or_t
         vec = xs_read_watch(paging->xs_handle, &num);
         if ( vec )
         {
+            DPRINTF("path '%s' token '%s'\n", vec[XS_WATCH_PATH], vec[XS_WATCH_TOKEN]);
             if ( strcmp(vec[XS_WATCH_PATH], "@releaseDomain") == 0 && strcmp(vec[XS_WATCH_TOKEN], watch_token) == 0 )
             {
                 /* If our guest disappeared, set interrupt flag and fall through */
@@ -111,6 +115,22 @@ static int xenpaging_wait_for_event_or_t
                     rc = 0;
                 }
             }
+            else if ( strcmp(vec[XS_WATCH_PATH], watch_targetpages) == 0 )
+            {
+                int ret, target_pages;
+                val = xs_read(paging->xs_handle, XBT_NULL, vec[XS_WATCH_PATH], NULL);
+                if (val)
+                {
+                    ret = sscanf(val, "%d", &target_pages);
+                    if ( ret > 0 )
+                    {
+                        if ( target_pages < 0 || target_pages > paging->max_pages )
+                            target_pages = paging->max_pages;
+                        paging->target_pages = target_pages;
+                        DPRINTF("new target_pages %d\n", target_pages);
+                    }
+                }
+            }
             free(vec);
         }
     }
@@ -167,8 +187,8 @@ static xenpaging_t *xenpaging_init(domid
     xc_domaininfo_t domain_info;
     xc_interface *xch;
     xentoollog_logger *dbg = NULL;
-    char *p;
-    int rc;
+    char *p, *dom_path = NULL, *watchdir = WATCH_TARGETPAGES;
+    int rc, len;
 
     /* Allocate memory */
     paging = calloc(1, sizeof(xenpaging_t));
@@ -201,6 +221,28 @@ static xenpaging_t *xenpaging_init(domid
         goto err;
     }
 
+    /* watch guests xenpaging directory */
+    dom_path = xs_get_domain_path(paging->xs_handle, domain_id);
+    if ( !dom_path )
+    {
+        ERROR("Could not find domain path\n");
+        goto err;
+    }
+    len = strlen(dom_path) + strlen(watchdir) + 1;
+    watch_targetpages = malloc(len);
+    if ( !watch_targetpages )
+    {
+        ERROR("Could not alloc domain path\n");
+        goto err;
+    }
+    snprintf(watch_targetpages, len, "%s%s", dom_path, watchdir);
+    DPRINTF("watching '%s'\n", watch_targetpages);
+    if ( xs_watch(paging->xs_handle, watch_targetpages, "") == false )
+    {
+        ERROR("Could not bind to xenpaging watch\n");
+        goto err;
+    }
+
     p = getenv("XENPAGING_POLICY_MRU_SIZE");
     if ( p && *p )
     {
@@ -299,7 +341,7 @@ static xenpaging_t *xenpaging_init(domid
         num_pages = paging->max_pages;
         DPRINTF("setting num_pages to %d\n", num_pages);
     }
-    paging->num_pages = num_pages;
+    paging->target_pages = num_pages;
 
     /* Initialise policy */
     rc = policy_init(paging);
@@ -330,6 +372,8 @@ static xenpaging_t *xenpaging_init(domid
             free(paging->mem_event.ring_page);
         }
 
+        free(dom_path);
+        free(watch_targetpages);
         free(paging->bitmap);
         free(paging);
     }
@@ -345,6 +389,9 @@ static int xenpaging_teardown(xenpaging_
     if ( paging == NULL )
         return 0;
 
+    xs_unwatch(paging->xs_handle, watch_targetpages, "");
+    xs_unwatch(paging->xs_handle, "@releaseDomain", watch_token);
+
     xch = paging->xc_handle;
     paging->xc_handle = NULL;
     /* Tear down domain paging in Xen */
@@ -649,6 +696,7 @@ int main(int argc, char *argv[])
     xenpaging_victim_t *victims;
     mem_event_request_t req;
     mem_event_response_t rsp;
+    int num, prev_num = 0;
     int i;
     int rc = -1;
     int rc1;
@@ -673,7 +721,7 @@ int main(int argc, char *argv[])
     }
     xch = paging->xc_handle;
 
-    DPRINTF("starting %s %u %d\n", argv[0], paging->mem_event.domain_id, paging->num_pages);
+    DPRINTF("starting %s %u %d\n", argv[0], paging->mem_event.domain_id, paging->target_pages);
 
     /* Open file */
     sprintf(filename, "page_cache_%u", paging->mem_event.domain_id);
@@ -703,9 +751,6 @@ int main(int argc, char *argv[])
     /* listen for page-in events to stop pager */
     create_page_in_thread(paging);
 
-    i = evict_pages(paging, fd, victims, paging->num_pages);
-    DPRINTF("%d pages evicted. Done.\n", i);
-
     /* Swap pages in and out */
     while ( 1 )
     {
@@ -774,8 +819,6 @@ int main(int argc, char *argv[])
                  * or clear this pagefile slot on exit */
                 if ( interrupted )
                     victims[i].gfn = INVALID_MFN;
-                else
-                    evict_victim(paging, &victims[i], fd, i);
             }
             else
             {
@@ -819,6 +862,29 @@ int main(int argc, char *argv[])
             if ( interrupted )
                 break;
         }
+        if ( paging->num_paged_out < paging->target_pages )
+        {
+            num = paging->target_pages - paging->num_paged_out;
+            if ( num != prev_num )
+            {
+                DPRINTF("Need to evict %d pages to reach %d target_pages\n", num, paging->target_pages);
+                prev_num = num;
+            }
+            /* Limit the number of evicts to be able to process page-in requests */
+            if ( num > 42 )
+                num = 42;
+            evict_pages(paging, fd, victims, num);
+        }
+        else if ( paging->num_paged_out > paging->target_pages )
+        {
+            num = paging->num_paged_out - paging->target_pages;
+            if ( num != prev_num )
+            {
+                DPRINTF("Need to resume %d pages to reach %d target_pages\n", num, paging->target_pages);
+                prev_num = num;
+            }
+            resume_pages(paging, num);
+        }
     }
     DPRINTF("xenpaging got signal %d\n", interrupted);
 
diff -r 1addb9e81178 -r ad903b34e099 tools/xenpaging/xenpaging.h
--- a/tools/xenpaging/xenpaging.h
+++ b/tools/xenpaging/xenpaging.h
@@ -50,7 +50,8 @@ typedef struct xenpaging {
     /* number of pages for which data structures were allocated */
     int max_pages;
     int num_paged_out;
-    int num_pages;
+    /* number of pages xenpaging is supposed to page-out */
+    int target_pages;
     int policy_mru_size;
     unsigned long pagein_queue[XENPAGING_PAGEIN_QUEUE_SIZE];
 } xenpaging_t;

Re: [PATCH 6 of 9] xenpaging: add evict_pages function

[Ian Campbell]

On Thu, 2011-09-15 at 02:16 -0400, Olaf Hering wrote:
> # HG changeset patch
> # User Olaf Hering <olaf@aepfle.de>
> # Date 1316067230 -7200
> # Node ID 3a3a5979b799d948802183d10d65894ee84a872f
> # Parent  6beca8cbc2c92900859712f8738db17084bcebdb
> xenpaging: add evict_pages function
> 
> Add new function to evict a couple of pages.

Do you really mean "a couple" here? (that generally means exactly two).
>From the implementation I think you mean evict a batch of pages?

> Allocate all possible slots in a paging file to allow growing and
> shrinking of the number of paged-out pages. Adjust other places to
> iterate over all slots.
> 
> This change is required by subsequent patches.
> 
> Signed-off-by: Olaf Hering <olaf@aepfle.de>
> 
> diff -r 6beca8cbc2c9 -r 3a3a5979b799 tools/xenpaging/xenpaging.c
> --- a/tools/xenpaging/xenpaging.c
> +++ b/tools/xenpaging/xenpaging.c
> @@ -597,6 +597,30 @@ static int evict_victim(xenpaging_t *pag
>      return ret;
>  }
>  
> +/* Evict a couple of pages and write them to a free slot in the paging file */
> +static int evict_pages(xenpaging_t *paging, int fd, xenpaging_victim_t *victims, int num_pages)
> +{
> +    xc_interface *xch = paging->xc_handle;
> +    int rc, slot, num = 0;
> +
> +    for ( slot = 0; slot < paging->max_pages && num < num_pages; slot++ )
> +    {
> +        /* Slot is allocated */
> +        if ( victims[slot].gfn != INVALID_MFN )
> +            continue;
> +
> +        rc = evict_victim(paging, &victims[slot], fd, slot);
> +        if ( rc == -ENOSPC )
> +            break;
> +        if ( rc == -EINTR )
> +            break;
> +        if ( num && num % 100 == 0 )
> +            DPRINTF("%d pages evicted\n", num);
> +        num++;
> +    }
> +    return num;
> +}
> +
>  int main(int argc, char *argv[])
>  {
>      struct sigaction act;
> @@ -639,7 +663,12 @@ int main(int argc, char *argv[])
>          return 2;
>      }
>  
> -    victims = calloc(paging->num_pages, sizeof(xenpaging_victim_t));
> +    /* Allocate upper limit of pages to allow growing and shrinking */
> +    victims = calloc(paging->max_pages, sizeof(xenpaging_victim_t));
> +
> +    /* Mark all slots as unallocated */
> +    for ( i = 0; i < paging->max_pages; i++ )
> +        victims[i].gfn = INVALID_MFN;
>  
>      /* ensure that if we get a signal, we'll do cleanup, then exit */
>      act.sa_handler = close_handler;
> @@ -653,18 +682,7 @@ int main(int argc, char *argv[])
>      /* listen for page-in events to stop pager */
>      create_page_in_thread(paging);
>  
> -    /* Evict pages */
> -    for ( i = 0; i < paging->num_pages; i++ )
> -    {
> -        rc = evict_victim(paging, &victims[i], fd, i);
> -        if ( rc == -ENOSPC )
> -            break;
> -        if ( rc == -EINTR )
> -            break;
> -        if ( i % 100 == 0 )
> -            DPRINTF("%d pages evicted\n", i);
> -    }
> -
> +    i = evict_pages(paging, fd, victims, paging->num_pages);
>      DPRINTF("%d pages evicted. Done.\n", i);
>  
>      /* Swap pages in and out */
> @@ -690,13 +708,13 @@ int main(int argc, char *argv[])
>              if ( test_and_clear_bit(req.gfn, paging->bitmap) )
>              {
>                  /* Find where in the paging file to read from */
> -                for ( i = 0; i < paging->num_pages; i++ )
> +                for ( i = 0; i < paging->max_pages; i++ )
>                  {
>                      if ( victims[i].gfn == req.gfn )
>                          break;
>                  }
>      
> -                if ( i >= paging->num_pages )
> +                if ( i >= paging->max_pages )
>                  {
>                      DPRINTF("Couldn't find page %"PRIx64"\n", req.gfn);
>                      goto out;
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 6 of 9] xenpaging: add evict_pages function

[George Dunlap]

On Thu, Sep 15, 2011 at 9:16 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2011-09-15 at 02:16 -0400, Olaf Hering wrote:
>> # HG changeset patch
>> # User Olaf Hering <olaf@aepfle.de>
>> # Date 1316067230 -7200
>> # Node ID 3a3a5979b799d948802183d10d65894ee84a872f
>> # Parent  6beca8cbc2c92900859712f8738db17084bcebdb
>> xenpaging: add evict_pages function
>>
>> Add new function to evict a couple of pages.
>
> Do you really mean "a couple" here? (that generally means exactly two).

LIterally "couple" means two, but at least in US idiom, "a couple of
[foo]" means a small indeterminate number, usually 2-4.

In any case, a more precise description seems like a better idea -- it
looks like it takes an argument for the number of pages to evict; and
it's not adding a new function, it's pulling existing code into a
function.  So, "Pull eviction loop into a function" would probably be
a better description.

 -George

> >From the implementation I think you mean evict a batch of pages?
>
>> Allocate all possible slots in a paging file to allow growing and
>> shrinking of the number of paged-out pages. Adjust other places to
>> iterate over all slots.
>>
>> This change is required by subsequent patches.
>>
>> Signed-off-by: Olaf Hering <olaf@aepfle.de>
>>
>> diff -r 6beca8cbc2c9 -r 3a3a5979b799 tools/xenpaging/xenpaging.c
>> --- a/tools/xenpaging/xenpaging.c
>> +++ b/tools/xenpaging/xenpaging.c
>> @@ -597,6 +597,30 @@ static int evict_victim(xenpaging_t *pag
>>      return ret;
>>  }
>>
>> +/* Evict a couple of pages and write them to a free slot in the paging file */
>> +static int evict_pages(xenpaging_t *paging, int fd, xenpaging_victim_t *victims, int num_pages)
>> +{
>> +    xc_interface *xch = paging->xc_handle;
>> +    int rc, slot, num = 0;
>> +
>> +    for ( slot = 0; slot < paging->max_pages && num < num_pages; slot++ )
>> +    {
>> +        /* Slot is allocated */
>> +        if ( victims[slot].gfn != INVALID_MFN )
>> +            continue;
>> +
>> +        rc = evict_victim(paging, &victims[slot], fd, slot);
>> +        if ( rc == -ENOSPC )
>> +            break;
>> +        if ( rc == -EINTR )
>> +            break;
>> +        if ( num && num % 100 == 0 )
>> +            DPRINTF("%d pages evicted\n", num);
>> +        num++;
>> +    }
>> +    return num;
>> +}
>> +
>>  int main(int argc, char *argv[])
>>  {
>>      struct sigaction act;
>> @@ -639,7 +663,12 @@ int main(int argc, char *argv[])
>>          return 2;
>>      }
>>
>> -    victims = calloc(paging->num_pages, sizeof(xenpaging_victim_t));
>> +    /* Allocate upper limit of pages to allow growing and shrinking */
>> +    victims = calloc(paging->max_pages, sizeof(xenpaging_victim_t));
>> +
>> +    /* Mark all slots as unallocated */
>> +    for ( i = 0; i < paging->max_pages; i++ )
>> +        victims[i].gfn = INVALID_MFN;
>>
>>      /* ensure that if we get a signal, we'll do cleanup, then exit */
>>      act.sa_handler = close_handler;
>> @@ -653,18 +682,7 @@ int main(int argc, char *argv[])
>>      /* listen for page-in events to stop pager */
>>      create_page_in_thread(paging);
>>
>> -    /* Evict pages */
>> -    for ( i = 0; i < paging->num_pages; i++ )
>> -    {
>> -        rc = evict_victim(paging, &victims[i], fd, i);
>> -        if ( rc == -ENOSPC )
>> -            break;
>> -        if ( rc == -EINTR )
>> -            break;
>> -        if ( i % 100 == 0 )
>> -            DPRINTF("%d pages evicted\n", i);
>> -    }
>> -
>> +    i = evict_pages(paging, fd, victims, paging->num_pages);
>>      DPRINTF("%d pages evicted. Done.\n", i);
>>
>>      /* Swap pages in and out */
>> @@ -690,13 +708,13 @@ int main(int argc, char *argv[])
>>              if ( test_and_clear_bit(req.gfn, paging->bitmap) )
>>              {
>>                  /* Find where in the paging file to read from */
>> -                for ( i = 0; i < paging->num_pages; i++ )
>> +                for ( i = 0; i < paging->max_pages; i++ )
>>                  {
>>                      if ( victims[i].gfn == req.gfn )
>>                          break;
>>                  }
>>
>> -                if ( i >= paging->num_pages )
>> +                if ( i >= paging->max_pages )
>>                  {
>>                      DPRINTF("Couldn't find page %"PRIx64"\n", req.gfn);
>>                      goto out;
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: [PATCH 7 of 9] xenpaging: add resume_pages function

[George Dunlap]

On Thu, Sep 15, 2011 at 7:16 AM, Olaf Hering <olaf@aepfle.de> wrote:
> # HG changeset patch
> # User Olaf Hering <olaf@aepfle.de>
> # Date 1316067232 -7200
> # Node ID 6987aa2dde4e93481f1599735ed1a26defb6d6b9
> # Parent  3a3a5979b799d948802183d10d65894ee84a872f
> xenpaging: add resume_pages function
>
> Add new function to resume a couple of pages.
> This change is required by subsequent patches.

Olaf, if you end up re-submitting this patch series for any reason, I
would combine this patch and the previous patch, and make the
description something like this:

"Pull page eviction and resume loops into their own functions."  That
gives a better idea what's going on.  Since neither really makes any
functional changes, there's no need to have separate patches, IMHO.

(I don't think that in itself is worth rejecting the series for.)

 -George

>
> Signed-off-by: Olaf Hering <olaf@aepfle.de>
>
> diff -r 3a3a5979b799 -r 6987aa2dde4e tools/xenpaging/xenpaging.c
> --- a/tools/xenpaging/xenpaging.c
> +++ b/tools/xenpaging/xenpaging.c
> @@ -554,6 +554,27 @@ static int xenpaging_populate_page(xenpa
>     return ret;
>  }
>
> +/* Trigger a page-in for a couple of pages */
> +static void resume_pages(xenpaging_t *paging, int num_pages)
> +{
> +    xc_interface *xch = paging->xc_handle;
> +    int i, num = 0;
> +
> +    for ( i = 0; i < paging->max_pages && num < num_pages; i++ )
> +    {
> +        if ( test_bit(i, paging->bitmap) )
> +        {
> +            paging->pagein_queue[num] = i;
> +            num++;
> +            if ( num == XENPAGING_PAGEIN_QUEUE_SIZE )
> +                break;
> +        }
> +    }
> +    /* num may be less than num_pages, caller has to try again */
> +    if ( num )
> +        page_in_trigger();
> +}
> +
>  static int evict_victim(xenpaging_t *paging,
>                         xenpaging_victim_t *victim, int fd, int i)
>  {
> @@ -785,25 +806,12 @@ int main(int argc, char *argv[])
>         /* Write all pages back into the guest */
>         if ( interrupted == SIGTERM || interrupted == SIGINT )
>         {
> -            int num = 0;
> -            for ( i = 0; i < paging->max_pages; i++ )
> -            {
> -                if ( test_bit(i, paging->bitmap) )
> -                {
> -                    paging->pagein_queue[num] = i;
> -                    num++;
> -                    if ( num == XENPAGING_PAGEIN_QUEUE_SIZE )
> -                        break;
> -                }
> -            }
> -            /*
> -             * One more round if there are still pages to process.
> -             * If no more pages to process, exit loop.
> -             */
> -            if ( num )
> -                page_in_trigger();
> -            else if ( i == paging->max_pages )
> +            /* If no more pages to process, exit loop. */
> +            if ( !paging->num_paged_out )
>                 break;
> +
> +            /* One more round if there are still pages to process. */
> +            resume_pages(paging, paging->num_paged_out);
>         }
>         else
>         {
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: [PATCH 6 of 9] xenpaging: add evict_pages function

[Olaf Hering]

On Thu, Sep 15, George Dunlap wrote:

> On Thu, Sep 15, 2011 at 9:16 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2011-09-15 at 02:16 -0400, Olaf Hering wrote:
> >> # HG changeset patch
> >> # User Olaf Hering <olaf@aepfle.de>
> >> # Date 1316067230 -7200
> >> # Node ID 3a3a5979b799d948802183d10d65894ee84a872f
> >> # Parent  6beca8cbc2c92900859712f8738db17084bcebdb
> >> xenpaging: add evict_pages function
> >>
> >> Add new function to evict a couple of pages.
> >
> > Do you really mean "a couple" here? (that generally means exactly two).
> 
> LIterally "couple" means two, but at least in US idiom, "a couple of
> [foo]" means a small indeterminate number, usually 2-4.
> 
> In any case, a more precise description seems like a better idea -- it
> looks like it takes an argument for the number of pages to evict; and
> it's not adding a new function, it's pulling existing code into a
> function.  So, "Pull eviction loop into a function" would probably be
> a better description.


Thanks to both of you, I will improve the description.

Olaf

Re: [PATCH 6 of 9] xenpaging: add evict_pages function

[Ian Jackson]

Olaf Hering writes ("Re: [Xen-devel] [PATCH 6 of 9] xenpaging: add evict_pages function"):
> On Thu, Sep 15, George Dunlap wrote:
> > In any case, a more precise description seems like a better idea -- it
> > looks like it takes an argument for the number of pages to evict; and
> > it's not adding a new function, it's pulling existing code into a
> > function.  So, "Pull eviction loop into a function" would probably be
> > a better description.
> 
> Thanks to both of you, I will improve the description.

Thanks.  Just to be clear, I think I'm expecting a repost either of
this patch, or perhaps of the whole series ?  Ian C made a comment
about combining two of your later patches, too.

Thanks,
Ian.

Re: [PATCH 6 of 9] xenpaging: add evict_pages function

[Olaf Hering]

On Wed, Sep 28, Ian Jackson wrote:

> Olaf Hering writes ("Re: [Xen-devel] [PATCH 6 of 9] xenpaging: add evict_pages function"):
> > On Thu, Sep 15, George Dunlap wrote:
> > > In any case, a more precise description seems like a better idea -- it
> > > looks like it takes an argument for the number of pages to evict; and
> > > it's not adding a new function, it's pulling existing code into a
> > > function.  So, "Pull eviction loop into a function" would probably be
> > > a better description.
> > 
> > Thanks to both of you, I will improve the description.
> 
> Thanks.  Just to be clear, I think I'm expecting a repost either of
> this patch, or perhaps of the whole series ?  Ian C made a comment
> about combining two of your later patches, too.

Yes, I will post a new series, replacing this one.

Olaf