From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 216A2C43441 for ; Mon, 19 Nov 2018 19:46:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C61BA2086A for ; Mon, 19 Nov 2018 19:46:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=concurrentrt.onmicrosoft.com header.i=@concurrentrt.onmicrosoft.com header.b="MR+zO0iV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C61BA2086A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=concurrent-rt.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730272AbeKTGLi (ORCPT ); Tue, 20 Nov 2018 01:11:38 -0500 Received: from mail-eopbgr780097.outbound.protection.outlook.com ([40.107.78.97]:18592 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725843AbeKTGLi (ORCPT ); Tue, 20 Nov 2018 01:11:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=concurrentrt.onmicrosoft.com; s=selector1-concurrentrt-com02e; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xW10cHfCTL/qBIm0OjTt0Zk/K34Hk56ukZByoWr9MF8=; b=MR+zO0iVSL0uavMADSm/RqEAzq1mNAvRU7EQOt4qQ6bXVI6A8xIEnnEvL8CUL6UKOu0mPBNeNlhV+L9DfSi6KSRTX+smqvkcjD5O8EMMfzGndYZd5DXDt4Bl8ByJ8xzp61RAmAk8kJ9keFeeXdJa7HuIVltMbqFRVfLNancZgvE= Received: from DM6PR11MB2570.namprd11.prod.outlook.com (20.176.99.12) by DM6PR11MB2746.namprd11.prod.outlook.com (20.176.99.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1339.25; Mon, 19 Nov 2018 19:46:23 +0000 Received: from DM6PR11MB2570.namprd11.prod.outlook.com ([fe80::516:620c:3f40:7215]) by DM6PR11MB2570.namprd11.prod.outlook.com ([fe80::516:620c:3f40:7215%5]) with mapi id 15.20.1339.027; Mon, 19 Nov 2018 19:46:23 +0000 From: Joe Korty To: "julia@ni.com" , "tglx@linutronix.de" , "bigeasy@linutronix.de" , "rostedt@goodmis.org" , "oleg@redhat.com" , "linux-rt-users@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Joe Korty Subject: [ptrace, rt] erratic behaviour in PTRACE_SINGLESET on 4.13-rt and later. Thread-Topic: [ptrace, rt] erratic behaviour in PTRACE_SINGLESET on 4.13-rt and later. Thread-Index: AQHUgECMj/wXEoj7Wkyq0VvWdFU8Ow== Date: Mon, 19 Nov 2018 19:46:23 +0000 Message-ID: <20181119194619.GA32121@zipoli.concurrent-rt.com> Reply-To: Joe Korty Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BN6PR03CA0049.namprd03.prod.outlook.com (2603:10b6:404:4c::11) To DM6PR11MB2570.namprd11.prod.outlook.com (2603:10b6:5:c6::12) authentication-results: spf=none (sender IP is ) smtp.mailfrom=Joe.Korty@concurrent-rt.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [12.220.59.2] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM6PR11MB2746;6:goG6n2QtCo3pWWe+Yno4GWdmpA/gEIVPMmpFDHqwVjOvqrOkXeKVry/H8T4/Db858BbwZLmBDBrJ4CyGVsu+p3WOg1mBUa8l7jcBbl5Pr040L3oLF4wrRkt2xx+HQ+y5pl7XIstnWsQEqtibO0KkgkbCMbBwIV46e+A3qkpkC/eGNcfnS+0JqqQ4Z7w/o6diOvBjkxlP5oWCHOEmYb5PrHjPF+l2bweNB1E5UbstI/bhzqakjOZtxlyBRSxccYVvnmx8322XAasZxFfxffGDOhdNes83lxaXuL9UB8yG3b19HCPDHPfwPao7PZAUmTPoiJE2ktF/Gpa9a03+i+43EfI9gQhrx/4rrLQz3dBfsnjak/ZMR90h+lhjGmdKRd2omXEx8QdJZiPmqHVZs1J0y/LqN+vaGGOzamifEbpwP+4kos6VRSEu5ynealTO0Uttr5KIfRlO8Sh7u6RD5XyMUA==;5:x/6KFKNLqAQMRJsmqwvUJvJF0GrTEGud3vHdQ6bEl/e72Dy9iRvF+Tv6Kke0BcZniRdwjG6dFUh2mnrLqWQ1vrePSBbGm04zfK+peErPG+8H6W1aUFvsK7P95npj5dtQ5If3m72Ln0xIgqvMKdUFulZDIHt8tbB+PQzCr8J7+W0=;7:mCA0d5Y64aX9PUdQEl47Ps5dQzp0858EARWYy93Ef0G3CTlmx8OtriLkYOiCL41akIfqr5lhU0A4Jrs15NAW7WnOJFH+0EWf0IIfCt6NmMMapw5IfLs9oBTBIgoosR0iKPQB3bhq8bPLpkrIpPoHgQ== x-ms-office365-filtering-correlation-id: 9ab33e9b-b577-44f2-bcc3-08d64e57af30 x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390098)(7020095)(4652040)(7021145)(8989299)(5600074)(711020)(4534185)(7022145)(4603075)(4627221)(201702281549075)(8990200)(7048125)(7024125)(7027125)(7023125)(2017052603328)(7153060)(7193020);SRVR:DM6PR11MB2746; x-ms-traffictypediagnostic: DM6PR11MB2746: x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(5005006)(8121501046)(10201501046)(3002001)(3231435)(944501410)(52105112)(93006095)(93001095)(148016)(149066)(150057)(6041310)(20161123562045)(20161123558120)(2016111802025)(20161123560045)(20161123564045)(6043046)(201708071742011)(7699051)(76991095);SRVR:DM6PR11MB2746;BCL:0;PCL:0;RULEID:;SRVR:DM6PR11MB2746; x-forefront-prvs: 08617F610C x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(366004)(39840400004)(136003)(396003)(346002)(376002)(199004)(189003)(2501003)(44832011)(2900100001)(6436002)(66066001)(68736007)(81166006)(81156014)(4744004)(25786009)(6116002)(97736004)(2906002)(1076002)(476003)(71190400001)(486006)(71200400001)(6486002)(3846002)(72206003)(2201001)(7736002)(86362001)(8676002)(186003)(14454004)(3450700001)(26005)(305945005)(33656002)(386003)(6506007)(508600001)(106356001)(6512007)(110136005)(316002)(5660300001)(102836004)(99286004)(14444005)(5024004)(53936002)(8936002)(43066004)(52116002)(105586002)(256004)(21314003);DIR:OUT;SFP:1102;SCL:1;SRVR:DM6PR11MB2746;H:DM6PR11MB2570.namprd11.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: concurrent-rt.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: L3BL6xYqlYihPTluDgj+7gilrQY2w1tEvbdfs4qghtgAHdmPS5fow/dEqL9d+hbcka5nIciuxnH2wDYLo04MgbMc8v9lkVmiQFQHiStwrBkEXy0lXt5mL43FdL5TOKk85MEJyt6gnb/TCqHYSZMaFda3TG8Akm8tai6hdHoNnnDbuZ6RFmv0lx1+U93h3W+PdR45R5mHxSYCXIn+CO3LGorEUtDGkZmmTR7CF2Q6UXw5M+CmHzaim7v5Vd7WND5sTiK589Yb/RuFFYZpPX2GL5wYWbwOW6w5GPo4A3pXx0aNW2GEXP+tZXFky8Ll2qhotP1LhrR7OJMVK6W6WUyY5aZSJySlM74f5fbk+iLX51A= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <68CB111ACF90544A8F1E3A1D282BFE14@namprd11.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: concurrent-rt.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9ab33e9b-b577-44f2-bcc3-08d64e57af30 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Nov 2018 19:46:23.0844 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 38747689-e6b0-4933-86c0-1116ee3ef93e X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB2746 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Julia & the RT team, The following program might make a good addition to the rt test suite. It tests the reliability of PTRACE_SINGLESTEP. It does by default 10,000 ssteps against a simple, spinner tracee. Also by default, it spins off ten of these tracer/tracee pairs, all of which are to run concurrently. Starting with 4.13-rt, this test occasionally encounters a sstep whose waitpid returns a WIFSIGNALED (signal SIGTRAP) rather than a WIFSTOPPED. This usually happens after thousands of ssteps have executed. Having multiple tracer/tracee pairs running dramatically increases the chances of failure. The is what the test output looks like for a good run: #forks: 10 #steps: 10000 =20 forktest#0/22872: STARTING forktest#7/22879: STARTING forktest#8/22880: STARTING forktest#6/22878: STARTING forktest#5/22877: STARTING forktest#3/22875: STARTING forktest#4/22876: STARTING forktest#9/22882: STARTING forktest#2/22874: STARTING forktest#1/22873: STARTING forktest#0/22872: EXITING, no error forktest#8/22880: EXITING, no error forktest#3/22875: EXITING, no error forktest#7/22879: EXITING, no error forktest#6/22878: EXITING, no error forktest#5/22877: EXITING, no error forktest#2/22874: EXITING, no error forktest#4/22876: EXITING, no error forktest#9/22882: EXITING, no error forktest#1/22873: EXITING, no error All tests PASSED. This is what the test output looks like for a failing run: #forks: 10 #steps: 10000 =20 forktest#0/22906: STARTING forktest#1/22907: STARTING forktest#2/22909: STARTING forktest#3/22911: STARTING forktest#4/22912: STARTING forktest#5/22915: STARTING forktest#6/22916: STARTING forktest#7/22919: STARTING forktest#8/22920: STARTING forktest#9/22923: STARTING forktest#2/22909: EXITING, ERROR: wait on PTRACE_SINGLESTEP #9: wanted ST= ATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#5/22915: EXITING, no error forktest#3/22911: EXITING, ERROR: wait on PTRACE_SINGLESTEP #7953: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#0/22906: EXITING, ERROR: wait on PTRACE_SINGLESTEP #5072: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#9/22923: EXITING, ERROR: wait on PTRACE_SINGLESTEP #7992: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#4/22912: EXITING, ERROR: wait on PTRACE_SINGLESTEP #9923: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#1/22907: EXITING, ERROR: wait on PTRACE_SINGLESTEP #7723: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#7/22919: EXITING, ERROR: wait on PTRACE_SINGLESTEP #5036: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#8/22920: EXITING, ERROR: wait on PTRACE_SINGLESTEP #4943: wanted= STATE_STOPPED, saw STATE_SIGNALED instead (and saw signo 5 too) forktest#6/22916: EXITING, no error One or more tests FAILED. Here are the observations from my testing so far: - It has never failed when confined to a single cpu. - It has never failed on !rt kernels. - It has never failed on any kernel prior to 4.13. - More failures than what chance would dictate happen near the end of a test run (ie, if a test of 10,000=20 steps is run, the failure will be at the 9,xxx'th step, if 100,000 steps are run, the failure will be at the 9x,xxx'th step). The above results are from kernels 4.{4,9,11,13,14,19}-rt and some !rt's of these as well. I have yet to see or hear of this bug, if it is a bug, giving anyone a problem in a debug session. It might not even be a bug but merely expected behaviour. And of course there is the possibility of a misuse of ptrace/waitpid in my test program. Its API, after all, is rather convoluted. Regards, Joe /* * Have a tracer do a bunch of PTRACE_SINGLESTEPs against * a tracee as fast as possible. Create several of these * tracer/tracee pairs and see if they can be made to * interfere with each other. * * Usage: * ssdd nforks niters * Where: * nforks - number of tracer/tracee pairs to fork off. * default 10. * niters - number of PTRACE_SINGLESTEP iterations to * do before declaring success, for each tracer/ * tracee pair set up. Default 10,000. * * The tracee waits on each PTRACE_SINGLESTEP with a waitpid(2) * and checks that waitpid's return values for correctness. */ #include #include #include #include #include #include #include #include #include #include /* do_wait return values */ #define STATE_EXITED 1 #define STATE_STOPPED 2 #define STATE_SIGNALED 3 #define STATE_UNKNOWN 4 #define STATE_ECHILD 5 #define STATE_EXITED_TSIG 6 /* exited with termination signal */ #define STATE_EXITED_ERRSTAT 7 /* exited with non-zero status */ char *state_name[] =3D { [STATE_EXITED] =3D "STATE_EXITED", [STATE_STOPPED] =3D "STATE_STOPPED", [STATE_SIGNALED] =3D "STATE_SIGNALED", [STATE_UNKNOWN] =3D "STATE_UNKNOWN", [STATE_ECHILD] =3D "STATE_ECHILD", [STATE_EXITED_TSIG] =3D "STATE_EXITED_TSIG", [STATE_EXITED_ERRSTAT] =3D "STATE_EXITED_ERRSTAT" }; const char *get_state_name(int state) { if (state < STATE_EXITED || state > STATE_EXITED_ERRSTAT) return "?"; return state_name[state]; } #define unused __attribute__((unused)) static int got_sigchld; static int do_wait(pid_t *wait_pid, int *ret_sig) { int status, child_status; *ret_sig =3D -1; /* initially mark 'nothing returned' */ while (1) { status =3D waitpid(-1, &child_status, WUNTRACED | __WALL); if (status =3D=3D -1) { if (errno =3D=3D EINTR) continue; if (errno =3D=3D ECHILD) { *wait_pid =3D (pid_t)0; return STATE_ECHILD; } printf("do_wait/%d: EXITING, ERROR: " "waitpid() returned errno %d\n", getpid(), errno); exit(1); } break; } *wait_pid =3D (pid_t)status; if (WIFEXITED(child_status)) { if (WIFSIGNALED(child_status)) return STATE_EXITED_TSIG; if (WEXITSTATUS(child_status)) return STATE_EXITED_ERRSTAT; return STATE_EXITED; } if (WIFSTOPPED(child_status)) { *ret_sig =3D WSTOPSIG(child_status); return STATE_STOPPED; } if (WIFSIGNALED(child_status)) { *ret_sig =3D WTERMSIG(child_status); return STATE_SIGNALED; } return STATE_UNKNOWN; } int check_sigchld(void) { int i; /* * The signal is asynchronous so give it some * time to arrive. */ for (i =3D 0; i < 10 && !got_sigchld; i++) usleep(1000); /* 10 msecs */ for (i =3D 0; i < 10 && !got_sigchld; i++) usleep(2000); /* 20 + 10 =3D 30 msecs */ for (i =3D 0; i < 10 && !got_sigchld; i++) usleep(4000); /* 40 + 30 =3D 70 msecs */ for (i =3D 0; i < 10 && !got_sigchld; i++) usleep(8000); /* 80 + 40 =3D 150 msecs */ for (i =3D 0; i < 10 && !got_sigchld; i++) usleep(16000); /* 160 + 150 =3D 310 msecs */ for (i =3D 0; i < 10 && !got_sigchld; i++) usleep(32000); /* 320 + 310 =3D 630 msecs */ return got_sigchld; } pid_t parent; int nforks =3D 10; int nsteps =3D 10000; static void sigchld(int sig, unused siginfo_t * info, unused void *arg) { got_sigchld =3D 1; } static void child_process(void) { unused volatile int i; /* wait for ptrace attach */ usleep(100000); while (1) i =3D 0; } static int forktests(int testid) { int i, status, ret_sig; long pstatus; pid_t child, wait_pid; struct sigaction act, oact; parent =3D getpid(); printf("forktest#%d/%d: STARTING\n", testid, parent); child =3D fork(); if (child =3D=3D -1) { printf("forktest#%d/%d: EXITING, ERROR: " "fork returned errno %d\n", testid, parent, errno); exit(1); } if (!child) child_process(); act.sa_sigaction =3D sigchld; sigemptyset(&act.sa_mask); act.sa_flags =3D SA_SIGINFO; status =3D sigaction(SIGCHLD, &act, &oact); if (status) { printf("forktest#%d/%d: EXITING, ERROR: " "sigaction returned %d, errno %d\n", testid, parent, status, errno); exit(1); } /* give both our child and parent time to set things up */ usleep(125000); /* * Attach to the child. */ pstatus =3D ptrace(PTRACE_ATTACH, child, NULL, NULL); if (pstatus =3D=3D ~0l) { printf("forktest#%d/%d: EXITING, ERROR: " "attach failed. errno %d\n", testid, getpid(), errno); exit(1); } /* * The attach should cause the child to receive a signal. */ status =3D do_wait(&wait_pid, &ret_sig); if (wait_pid !=3D child) { printf("forktest#%d/%d: EXITING, ERROR: " "attach: Unexpected wait pid %d\n", testid, getpid(), wait_pid); exit(1); } if (status !=3D STATE_STOPPED) { printf("forktest#%d/%d: EXITING, ERROR: " "attach: wait on PTRACE_ATTACH returned %d " "[%s, wanted STATE_STOPPED], signo %d\n", testid, getpid(), status, get_state_name(status), ret_sig); exit(1); } else if (!check_sigchld()) { printf("forktest#%d/%d: EXITING, ERROR: " "wait on PTRACE_ATTACH saw a SIGCHLD count of %d, should be 1\n", testid, getpid(), got_sigchld); exit(1); } got_sigchld =3D 0; /* * Generate 'nsteps' PTRACE_SINGLESTEPs, make sure they all actually step * the tracee. */ for (i =3D 0; i < nsteps; i++) { pstatus =3D ptrace(PTRACE_SINGLESTEP, child, NULL, NULL); if (pstatus) { printf("forktest#%d/%d: EXITING, ERROR: " "PTRACE_SINGLESTEP #%d: returned status %ld, " "errno %d, signo %d\n", testid, getpid(), i, pstatus, errno, ret_sig); exit(1); } status =3D do_wait(&wait_pid, &ret_sig); if (wait_pid !=3D child) { printf("forktest#%d/%d: EXITING, ERROR: " "wait on PTRACE_SINGLESTEP #%d: returned wrong pid %d, " "expected %d\n", testid, getpid(), i, wait_pid, child); exit(1); } if (status !=3D STATE_STOPPED) { printf("forktest#%d/%d: EXITING, ERROR: " "wait on PTRACE_SINGLESTEP #%d: wanted STATE_STOPPED, " "saw %s instead (and saw signo %d too)\n", testid, getpid(), i, get_state_name(status), ret_sig); exit(1); } if (ret_sig !=3D SIGTRAP) { printf("forktest#%d/%d: EXITING, ERROR: " "wait on PTRACE_SINGLESTEP #%d: returned signal %d, " "wanted SIGTRAP\n", testid, getpid(), i, ret_sig); exit(1); } if (!check_sigchld()) { printf("forktest#%d/%d: EXITING, ERROR: " "wait on PTRACE_SINGLESTEP #%d: no SIGCHLD seen " "(signal count =3D=3D 0), signo %d\n", testid, getpid(), i, ret_sig); exit(1); } got_sigchld =3D 0; } /* * We are all done, kill the tracee and wait for it to die. We test * the killing results as much as the above attach and sstep results, * though failure here really isn't the point of this test. */ status =3D kill(child, SIGKILL); if (status) { printf("forktest#%d/%d: EXITING, ERROR: " "kill of child %d returned %d\n", testid, getpid(), child, errno); exit(1); } status =3D do_wait(&wait_pid, &ret_sig); if (wait_pid !=3D child) { printf("forktest#%d/%d: EXITING, ERROR: " "kill: Unexpected wait pid %d\n", testid, getpid(), wait_pid); exit(1); } if (status !=3D STATE_SIGNALED) { printf("forktest#%d/%d: EXITING, ERROR: " "kill: Unexpected child, do_wait status %d " "[%s, wanted STATE_SIGNALED]\n", testid, getpid(), status, get_state_name(status)); exit(1); } if (ret_sig !=3D SIGKILL) { printf("forktest#%d/%d: EXITING, ERROR: " "kill: Unexpected child signal %d\n", testid, getpid(), ret_sig); exit(1); } printf("forktest#%d/%d: EXITING, no error\n", testid, parent); exit(0); } int main(int argc, char **argv) { int i, ret_sig, status; pid_t child =3D 0, wait_pid; int error =3D 0; setbuf(stdout, NULL); argc--, argv++; if (argc) { nforks =3D atoi(*argv); argc--, argv++; if (argc) nsteps =3D atoi(*argv); } printf("#forks: %d\n", nforks); printf("#steps: %d\n", nsteps); printf("\n"); for (i =3D 0; i < nforks; i++) { child =3D fork(); if (child =3D=3D -1) { printf("main: fork returned errno %d\n", errno); exit(1); } if (!child) forktests(i); } for (i =3D 0; i < nforks; i++) { status =3D do_wait(&wait_pid, &ret_sig); if (status !=3D STATE_EXITED) { if (0) printf("main/%d: ERROR: " "forktest#%d unexpected do_wait status %d " "[%s, wanted STATE_EXITED]\n", getpid(), wait_pid, status, get_state_name(status)); error =3D 1; } } printf("%s.\n", error ? "One or more tests FAILED" : "All tests PASSED"); exit(error); }