Building Ceph in Docker

[Mingliang LIU <mingliang.liu@salesforce.com>]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Hi,

I know we're building Ceph in Linux distributions and many of you use
vstart.sh as I do. Steps to install dependencies are easy so we don't
have to worry about prerequisites in a dev machine.

I'm thinking if it's useful to build Ceph in Docker. There are reasons
to do this, for example 1) not be able/willing to install quite a few
dependency software/library on a shared Linux, 2) to have a standard
build image for isolated and automatic testing, and 3) to build on
Mac/Windows. I know for this we can create Linux virtual machines on
Mac/ Windows, but I think container is lightweight and better.

So I attach a small script that creates a Docker image to build the
Ceph from source, and then starts a test cluster with vstart.sh in a
container. Do you think it's something useful and/or worth to have in
source tree?

When I was building Hadoop, I found its start-build-env.sh helpful,
which is basically the same idea I borrowed here.

Thanks,

Mingliang

[-- Attachment #2: 0001-Initial-docker-build-scripts-start-build-env.patch --]
[-- Type: application/octet-stream, Size: 4310 bytes --]

From 1ed259b4d4895dcadab83f6ee06c95e7eb1d5ba2 Mon Sep 17 00:00:00 2001
From: Mingliang Liu <liuml07@gmail.com>
Date: Sun, 20 Aug 2017 00:33:04 -0700
Subject: [PATCH] Initial docker build scripts start-build-env

---
 .gitignore                     |  2 ++
 dev-tools/docker/Dockerfile    | 45 ++++++++++++++++++++++++++++++++++++++++++
 dev-tools/docker/entrypoint.sh | 21 ++++++++++++++++++++
 start-build-env.sh             | 34 +++++++++++++++++++++++++++++++
 4 files changed, 102 insertions(+)
 create mode 100644 dev-tools/docker/Dockerfile
 create mode 100755 dev-tools/docker/entrypoint.sh
 create mode 100755 start-build-env.sh

diff --git a/.gitignore b/.gitignore
index 9a0cd9f0a1..d6919bd42b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -60,3 +60,5 @@ GTAGS
 
 # editor backup files etc.
 \#*\#
+
+dev-tools/docker/control
diff --git a/dev-tools/docker/Dockerfile b/dev-tools/docker/Dockerfile
new file mode 100644
index 0000000000..ab84728523
--- /dev/null
+++ b/dev-tools/docker/Dockerfile
@@ -0,0 +1,45 @@
+# Dockerfile for building a dev docker image
+
+FROM ubuntu:16.04
+
+MAINTAINER Mingliang Liu <mingliang.liu@salesforce.com>
+
+RUN apt-get update && apt-get install -y \
+    lsb-release devscripts equivs \
+    dpkg-dev gcc \
+    ccache git sudo
+
+ARG USER_ID
+ARG USER_NAME
+ARG GROUP_ID
+
+# Create sudo user with given UID/GID for proper volume permissions
+RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME} \
+ && useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME} \
+ && echo "${USER_NAME} ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/${USER_NAME} \
+ && chmod 0440 /etc/sudoers.d/${USER_NAME}
+
+# Install dependencies
+ADD control /control
+ENV DEBIAN_FRONTEND=noninteractive
+# the dependencies are vulnerable to i18n
+ENV LC_ALL=C
+# make a metapackage that expresses the build dependencies,
+# install it, rm the .deb; then uninstall the package as its
+# work is done
+RUN apt-get update \
+ && mk-build-deps --install --remove --tool="apt-get -y --no-install-recommends" /control \
+ && apt-get -y remove ceph-build-deps
+
+USER ${USER_NAME}
+
+ENV HOME /home/${USER_NAME}
+ENV PATH "/home/${USER_NAME}/ceph/build/bin:${PATH}"
+# Default is ~/.ccache, but we don't want to remove that directory when exiting container.
+ENV CCACHE_DIR /home/${USER_NAME}/ceph/build/.ccache
+
+WORKDIR /home/${USER_NAME}/ceph/build
+
+EXPOSE 41000 42000
+
+ENTRYPOINT ["../dev-tools/docker/entrypoint.sh"]
diff --git a/dev-tools/docker/entrypoint.sh b/dev-tools/docker/entrypoint.sh
new file mode 100755
index 0000000000..b11a9178ac
--- /dev/null
+++ b/dev-tools/docker/entrypoint.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+set -ex
+
+if [ ! -d $HOME/ceph ]; then
+    echo "You have to attach ceph source directory first!"
+    exit 1
+fi
+
+cd $HOME/ceph
+
+# Reuse the existing build directory
+[ -d build ] || ./do_cmake.sh
+
+cd build  # Assuming this is where you ran cmake
+make -j$(expr $(nproc) / 2) vstart
+
+# If is not Linux, we now only support memstore as the osd objectstore backend
+../src/vstart.sh -d -n -x --memstore
+
+bin/ceph -w
diff --git a/start-build-env.sh b/start-build-env.sh
new file mode 100755
index 0000000000..e1eee45771
--- /dev/null
+++ b/start-build-env.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+set -ex
+
+cd "$(dirname "$0")" # connect to root
+
+cp -f debian/control dev-tools/docker
+
+# We use ceph user with passwordless sudo access in Docer container.
+# Assign an explicit UID/GID to handle volume permissions
+USER_NAME=$USER
+if [ "$(uname -s)" == "Linux" ]; then
+  USER_ID=$(id -u "${USER_NAME}")
+  GROUP_ID=$(id -g "${USER_NAME}")
+else
+  USER_ID=1000
+  GROUP_ID=50
+fi
+
+docker build -t ceph-build \
+    --build-arg USER_NAME=${USER_NAME} \
+    --build-arg USER_ID=${USER_ID} \
+    --build-arg GROUP_ID=${GROUP_ID} \
+    dev-tools/docker
+
+# 1. We need more memory (>4GB) to avoid OOM when compiling
+#    If you are running docker in VM, please increase its memory limit first
+# 2. We publish exposed ports on host
+# 3. The source directory is from host so any changes will be reflected and persisted.
+#    You may want to add more mount options, e.g. consistency
+docker run -m 8G \
+    -p 41000:41000 -p 42000:42000 \
+    --mount type=bind,source="$(pwd)",target=/home/${USER_NAME}/ceph \
+    ceph-build
-- 
2.14.1