git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: gitster@pobox.com, sandals@crustytoothpaste.net,
	lenaic@lhuard.fr, Derrick Stolee <derrickstolee@github.com>,
	Derrick Stolee <derrickstolee@github.com>
Subject: [PATCH 1/6] maintenance: add get_random_minute()
Date: Mon, 07 Aug 2023 18:51:35 +0000	[thread overview]
Message-ID: <fefdaa9457948ee5302e7cbfaae250e0b589d752.1691434300.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.1567.git.1691434300.gitgitgadget@gmail.com>

From: Derrick Stolee <derrickstolee@github.com>

When we initially created background maintenance -- with its hourly,
daily, and weekly schedules -- we considered the effects of all clients
launching fetches to the server every hour on the hour. The worry of
DDoSing server hosts was noted, but left as something we would consider
for a future update.

As background maintenance has gained more adoption over the past three
years, our worries about DDoSing the big Git hosts has been unfounded.
Those systems, especially those serving public repositories, are already
resilient to thundering herds of much smaller scale.

However, sometimes organizations spin up specific custom server
infrastructure either in addition to or on top of their Git host. Some
of these technologies are built for a different range of scale, and can
hit concurrency limits sooner. Organizations with such custom
infrastructures are more likely to recommend tools like `scalar` which
furthers their adoption of background maintenance.

To help solve for this, create get_random_minute() as a method to help
Git select a random minute when creating schedules in the future. The
integrations with this method do not yet exist, but will follow in
future changes.

One thing that is important for testability is that we notice when we
are under a test scenario and return a predictable result. The schedules
themselves are not checked for this value, but at least one launchctl
test checks that we do not unnecessarily reboot the schedule if it has
not changed from a previous version.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 builtin/gc.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/builtin/gc.c b/builtin/gc.c
index f3942188a61..66a972bc292 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -1708,6 +1708,23 @@ static int get_schedule_cmd(const char **cmd, int *is_available)
 	return 1;
 }
 
+MAYBE_UNUSED
+static int get_random_minute(void)
+{
+	static int random_initialized = 0;
+
+	/* Use a static value when under tests. */
+	if (!getenv("GIT_TEST_MAINTENANCE_SCHEDULER"))
+		return 13;
+
+	if (!random_initialized) {
+		srand((unsigned int)getpid());
+		random_initialized = 1;
+	}
+
+	return rand() % 60;
+}
+
 static int is_launchctl_available(void)
 {
 	const char *cmd = "launchctl";
-- 
gitgitgadget


  reply	other threads:[~2023-08-07 18:51 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-08-07 18:51 [PATCH 0/6] maintenance: schedule maintenance on a random minute Derrick Stolee via GitGitGadget
2023-08-07 18:51 ` Derrick Stolee via GitGitGadget [this message]
2023-08-07 21:20   ` [PATCH 1/6] maintenance: add get_random_minute() Taylor Blau
2023-08-07 23:53     ` Junio C Hamano
2023-08-08  0:22       ` Junio C Hamano
2023-08-08 14:48         ` Taylor Blau
2023-08-08 16:34           ` Junio C Hamano
2023-08-08 16:49             ` Junio C Hamano
2023-08-08 20:01               ` Taylor Blau
2023-08-08 17:28     ` Derrick Stolee
2023-08-08 20:04       ` Taylor Blau
2023-08-09 12:17         ` Derrick Stolee
2023-08-09 18:50           ` Junio C Hamano
2023-08-09 20:34             ` Taylor Blau
2023-08-07 18:51 ` [PATCH 2/6] maintenance: use random minute in launchctl scheduler Derrick Stolee via GitGitGadget
2023-08-07 21:23   ` Taylor Blau
2023-08-07 18:51 ` [PATCH 3/6] maintenance: use random minute in Windows scheduler Derrick Stolee via GitGitGadget
2023-08-07 18:51 ` [PATCH 4/6] maintenance: use random minute in cron scheduler Derrick Stolee via GitGitGadget
2023-08-07 18:51 ` [PATCH 5/6] maintenance: swap method locations Derrick Stolee via GitGitGadget
2023-08-07 21:24   ` Taylor Blau
2023-08-07 18:51 ` [PATCH 6/6] maintenance: use random minute in systemd scheduler Derrick Stolee via GitGitGadget
2023-08-07 21:31   ` Taylor Blau
2023-08-08 13:49     ` Derrick Stolee
2023-08-08 20:05       ` Taylor Blau
2023-08-08  9:53   ` Phillip Wood
2023-08-08 13:03     ` Phillip Wood
2023-08-08 13:56     ` Derrick Stolee
2023-08-08 17:24       ` Derrick Stolee
2023-08-09 10:03         ` Phillip Wood
2023-08-08 12:08   ` Phillip Wood
2023-08-08 17:06     ` Derrick Stolee
2023-08-08 17:14       ` Derrick Stolee
2023-08-09 10:00         ` Phillip Wood
2023-08-10 20:39 ` [PATCH v2 0/8] maintenance: schedule maintenance on a random minute Derrick Stolee via GitGitGadget
2023-08-10 20:39   ` [PATCH v2 1/8] maintenance: add get_random_minute() Derrick Stolee via GitGitGadget
2023-08-10 21:25     ` Taylor Blau
2023-08-10 20:39   ` [PATCH v2 2/8] maintenance: use random minute in launchctl scheduler Derrick Stolee via GitGitGadget
2023-08-10 20:39   ` [PATCH v2 3/8] maintenance: use random minute in Windows scheduler Derrick Stolee via GitGitGadget
2023-08-10 20:39   ` [PATCH v2 4/8] maintenance: use random minute in cron scheduler Derrick Stolee via GitGitGadget
2023-08-10 20:39   ` [PATCH v2 5/8] maintenance: swap method locations Derrick Stolee via GitGitGadget
2023-08-10 20:39   ` [PATCH v2 6/8] maintenance: use random minute in systemd scheduler Derrick Stolee via GitGitGadget
2023-08-14 11:26     ` Phillip Wood
2023-08-10 20:39   ` [PATCH v2 7/8] maintenance: fix systemd schedule overlaps Derrick Stolee via GitGitGadget
2023-08-10 21:22     ` Junio C Hamano
2023-08-14 11:27     ` Phillip Wood
2023-08-10 20:39   ` [PATCH v2 8/8] maintenance: update schedule before config Derrick Stolee via GitGitGadget
2023-08-14 11:28     ` Phillip Wood

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=fefdaa9457948ee5302e7cbfaae250e0b589d752.1691434300.git.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=derrickstolee@github.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=lenaic@lhuard.fr \
    --cc=sandals@crustytoothpaste.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).