Test Server & Test Runner

Current Limitations / TODO

Every memory limit is currently ignored. The memory limit for programs needs to be set via linux cgroups.

sometimes the change dirs are not cleaned up
we change the dirs every changeDirIntervalInS s so in case some run is not fully cleaned up the whole dir is deleted...

Test Server This is the server that handles the requests and provides the Api.
(Also Called Server)

Test Runner This is the program that gets executed from the Test Server in order to test the users program.
(Also called Runner)

default workflow

• The Server receives a request
• Check if all required parameters are set
• Check if max parallel tests are reached
  • true: throw error (send back error)
• Start a transaction (create a row in the db)
• Get the correct command to execute from the database (if needed)
• Execute the test
  • create user test dir inside the root dir (see constants.php > $rootDirNameToUseString)
  • Write all required files to disk (if needed)
  • Execute the Test Runner with the command from the database and the parameters (if needed)
  • Remove written files from disk (if needed) and user test dir
• close/end the transaction
• Send back the result

request object (json)

syntax

The request is a json object so the request should be from mime type application/json.

• command: (string) the command to call/execute
• requestDistinctionString: (string, only letters, numbers, underscores --> should be a valid file/dir name) a number to help the this server to separate two requests in case they arrive at the same time (and microtime() would return the same value)
• plang: (string) the programming language to use (depends on the database table see database scheme / table /commands)
• mainFileNameWithExtension: (string) the file with the main class / entry point
• testContents: (array of objects) an array of tests
  • id (int) is the id of the test to identify the test (and the related result)
  • content (string/multi line string) the test content to use (depends on the used command)
  • assets (array) the files (assets) for the test
    • fileName (string) the file name for the asset.
      • note that the file name must not contain the following characters: (whitespace), :
      • also all files need to have an extension (x.y)
    • fileContent (string) the content of the asset (base64 encoded)
    • hash (string/null) the md5 hash of the content (can be null if not calculated)
      • but the test runner will not accept null as hash (for now)
• timeoutInMs: (int) the timeout in ms to wait after terminating the users program
• memoryLimitInKb: (int) the max memory in kb the users program can use
• maxDiskSpaceInKb: (int) the max disk space in kb the users program can write to
• allFiles: (array) the files to use

example

{
    "plang": "java",
    "mainFileNameWithExtension": "Main.java",
    "allFiles": [
        {
            "fileName": "Main.java",
            "fileContent": "public class Main {\npublic static void main(String[] args) {\nSystem.out.println(\"echo\");\n}\n}"
        }
    ],
    "tests": [
        {
            "command": "blackBoxTest",
            "testContents": [
                {
                    "id": 0,
                    "content": "> Hello World\n!",
                    "assets": []
                }
            ],
            "timeoutInMs": 1000,
            "memoryLimitInKb": 1000,
            "maxDiskSpaceInKb": 1000
        }
    ]
}

response array (json)

Syntax

A json array representing the responses. If an errors occurs before or after the test has run then the data object will be empty.
In case some server error happened and no test could be executed then just one object is returned:

{
    "responseCode": 0,
    "message": "some error"
}

If the tests could be executed or to code to execute the tests is reached then the response will be an array for with an response object for every test

• responseCode: (int) is the response code from the server. if we run multiple tests this is the general response code (see Response Codes from the test server)
• message: (string) an additional message (can be empty). if we run multiple tests this is the general response message
• data: (object) the response data

[
    {
        "responseCode": 0,
        "message": "",
        "data": [{
                "testServerCode": 0,
                "testServerMessage": "ok",

                "id": 0,
                "passed": true,
                "hasCompiled": true,
                "programExitCode": null,
                "timeForCompiling": null,
                "timeForUserProgram": null,
                "runnerVersion": "1.0.0",
                "testResultCode": 0,
                "protocol": "can be multiline \n\r string",
                "testServerVersion": "1.0.0",
            }
        ]
    }
]

The Data Object

these fields are used because we can run many test and every test could have a different server error code and message... so this is equal to responseCode and message

• testServerCode: the server return code (see Response Codes from the test server)
• testServerMessage: the server message in case of an server error

other fields

• id: (int) the test id from the request

• passed: (bool) true: the test was successful, false: not

• hasCompiled: (bool) true: test has compiled (if the test needs compilation, if not always true), false: not [actually this can be determined by checking testResultCode === (compile error return code from the test runner)]

• testResultCode: (int) the exit code of the test runner see Test Runner Return Codes

• programExitCode: (int | null) the exit code of the users program, null if the user program was not (fully) executed e.g. timeout hit

• timeForCompiling: (int | null) the time for compiling the user program or null if not compiled

• timeForUserProgram: (int | null) the time for executing the user program or null if not (fully) executed

• runnerVersion: (string | null) the test-runner version or null if the test-runner was not executed

• protocol: (string) a string representing the output (could contain the error message. New line chars are encoded with \\r\\n (because json encode automaticall encodes \r\n to \\n\\n)

• testServerVersion: (string) the test-server version

Response Codes from the test server

• 0 - all ok (no test server error), this has nothing to do with the test result itself (e.g. if a test fails then this can be 0)
• 1 - max parallel tests reached, try later again
• 2 - argument invalid or missing
• 3 - some error from the db or the db response
• 4 - some file system issue
• 100 - internal server error

For explanation about the different test types see below

Protocols

The Test-Runner uses \n new lines

All protocols consists out of 2 parts

the first part is the header part and contains some misc data about the test run e.g. test-runner version...

the seconds part is the real conversion protocol between the test-runner and the user program

the two parts are separated by a ---\n line

the header part including the ---\n must be removed from the protocol!

Protocol Header part

the header part has the same schema for all tests

every information is on a new line and is a tuple where key and value is separated by :

there are the following information possible (the tuple keys) (the keys are not case sensitive!)

• exit (number) the user program exit code or null if it din't finish e.g. timeout hit
• runnerVersion (string) the test runner version
• timeForCompiling (int) the time for compiling in ms or null if compiling was not done or finished
• timeForUserProgram (int) the time for running the user program or null if the execution was not done or finished

after the header part the line ---\n is followed, indicating the end of the header part

all values can be null!

the order of the tuples is not important

note that the tuple key and values must not contain the separator char : or the \n char or ---\n

exit:0
runnerVersion:1.0.0
timeForCompiling:null
timeForUserProgram:null
---

exit:null
runnerVersion:1.0.0
timeForCompiling:1000
timeForUserProgram:null
---

Protocol Output Compile Test

empty
Or
when a compile error occurred then the output of the compiler

Protocol Output for Just Run Test

Actually the program is run without any input (to stdin)...

all lines (meaning different by the first character):

• '$' are the arguments passed to the users program, starting with a $ sign (only if arguments were present)
• '>' indicating a line that was read from the users program
• 'EOT' indicates that the test ended here (written after the program terminated)
  this is useful when some output is received after the test has finished

Protocol Output Regex Test

1. Line: the used mode e.g. used mode: blacklist
2. Line: the used connector e.g. used connector: &&
   the next lines contains the test results for every file

Protocol Output Black Box Test

1 Line can be the expected user program exit code but only if the user program actually ran and finished

• 'e' is the expected user program exit code followed by a number (e.g. e0)

all other lines

• '$' are the arguments passed to the users program, starting with a $ sign (only if arguments were present) all other lines (meaning different by the first character):
• '<' indicating a line that was inputted to the users program
• '>' indicating a line that was read from the users program
• '>*' indicating a line that was read from the users program but matched anything (matches every user program output) but needs at least the empty string System.out.println("")
• 'EOT' indicates that the test ended here (written after the program terminated)
  this is useful when some output is received after the test has finished
• 'error:' output from the error stream (from the user program)
• 'expected:' indicates the expected output of the user program (this normally followed after a > when the output was wrong)

OR
when a compile error occurred then the output of the compiler

Protocol Output Compare files Test

EOT
ccompare [fileName1WithExtension] [fileName2WithExtension] (ignoreCase) (ignoreLeadingWhitespaces) (ignoreTrailingWhitespaces) (ignoreAllEmptyOrWhitespacesOnlyLines)
c[result]

where [X] is the user program exit code (must be the first line)

where EOT indicates the end of the user program (usually before all ccompares)

where [result] is one of the following string

• ok the files were equal line by line with the given options
• notfound1 the file 1 was not found (probably a test error)
• notfound2 the file 2 was not found
• expected indicates the expected line (line from file1) when the lines mismatched
• actual indicates the actual line (line from file2) when the lines mismatched

when a line mismatched we have the following output

ccompare [fileName1WithExtension] [fileName2WithExtension] (ignoreCase) (ignoreLeadingWhitespaces) (ignoreTrailingWhitespaces) (ignoreAllEmptyOrWhitespacesOnlyLines)
cactual [line nr] [line]
cexpected [line nr] [line]

if the user file ended before the expected file then the output is

ccompare [fileName1WithExtension] [fileName2WithExtension] (ignoreCase) (ignoreLeadingWhitespaces) (ignoreTrailingWhitespaces) (ignoreAllEmptyOrWhitespacesOnlyLines)
cactual [line nr] eof
cexpected [line nr] [line]

Test inputs

For the test input protocol see the section "Test Runner Interface"

Adding a new programming language

To add a new programming language (plang) you need to add the appropriate commands for the plang in the plang table.
The plang is then known to the test server and can be used (if you use e.g. syndrom client-server then that server needs to be configured separately to handle/know the new plang).

database scheme / table /commands

The Server needs a table where the commands are stored.
A command (actually commands) is used to compile (also syntax check) and execute a users program. Normally this is the shell/cmd command to execute a compiler or interpreter against a given file (or files).

In the dir backupDb are some examples that can be imported to a MySql Db.

PLang Table

The database needs to contain a table to store this commands (for config see section config.json file, you can specify the table name there but the table needs to exists already). The table needs to have the following scheme

internalName (primary key) | compile | exec | extensions --- | --- | --- | --- | --- | --- | --- (string) | (string) | (string) | (json Array)

• internalName - the internal name of the programming language (this is the primary key)
• compile - the command (shell/cmd) to execute the compiler (or do a syntax check for most interpreted languages)
• exec - the command (shell/cmd) to execute the users program (some languages requires virtual machines to execute the intermediate code)
• extensions - the extensions of the source files (used to identify source code files so that asset files can be ignored) (a json array so e.g. ["cpp", "h"])

The commands will probably require the users files (code). This files can be accessed by special strings (the special strings will be replaces by the value) (all absolute paths includes the file names)

Ref 1: This is also part of the Runners Interface
So the test runner must understand this special strings in the compile and exec command!

• #1 (closed) - contains the absolute directory path where all user files will be stored
  (without trailing directory separator)

• #2 - contains the main file name

• #3 - contains the main file name with extension

• #4 - contains the main file absolute path

• #5 - contains the main file absolute path with extension

• #6 - all other files (all user files excluding the main file) names

• #7 - all other files (all user files excluding the main file) names with extensions

• #8 - all other files (all user files excluding the main file) absolute paths (with file name but without extension)

• #9 - all other files (all user files excluding the main file) absolute paths with extension

• #50 - contains the timeout in ms

• #51 - contains the memory limit in kb

• #52 - contains the disk space limit in kb (the progrma can write to)

(So #1 (closed)/#6 would produce the same result as #8 when the operating system (os) is unix, escaped: "#1 (closed)/#6" = "#8")

Notes

• all files need an extension else they are ignored
• if the main file has no extension then code 100 is returned and the program is not compiled/executed
• any compile and exec commands will be interpret by the test server (the unix user who executes the test server) so when you e.g. set compile to \"echo\" \"$PATH\" then $PATH will be resolved by the unix user and the test runner gets /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games...

Example (windows)

Example (linux)

Reminder do not forget to escape evey path and arguments!

The compile and execute commands will be wrapped by "..." (double quotes). In order to escap paths use \"...path...\" (for bash shell) (see the command examples)

Transaction table

Columns

• id - the transaction id (this is a microtime timestamp and with optional request distinction number to help separate two requests in case they arrive at the same time and microtime() returns the same time)
• rootDirName - the current root dir name (of the current iteration) (this is a string of a timestamp)
• testType - the test type

Example

where '1480443681' ist the seconds-part if microtime() timestamp '0-04701800' is the microtime part of microtime() and '1' ist the request distinction number

Config Table

Columns (the same as in the local config except the db specific settings)

• id (we should have a primary key) (number) the number does not matter
• workingDirFullPath
• hardTimeoutInMs
• hardMemoryLimitInKb
• hardMaxDiskSpaceLimitInKb
• maxParallelTests
• maxNumberOfTestsWithOneRequest
• runner (in the db the " don't need to be escaped)
• showTestRunnerDebugOutput (0/1)
• maxLinesToRead
• maxErrLinesToRead
• maxLinesToWrite
• environmentVars

Example

config.json file

This file is used to configure the server.
This file is required! You will find the file in the directory /syndromtests/TestServer/

First the config.json file is read. If useConfigFromDb is set to true then the config from db is read and the settings are overwritten

Syntax

A json object with the following properties (order does not matter)

• dbServer : (string) the server name/ip address of the server where the database is stored

• dbName : (string) the name of the db where the required table is stored

• dbTableName : (string) the table name where the commands are stored

• dbUser : (string) the name of the database user to use for quering the database

• dbPw : (string) the password of the database user

• dbTransactionTableName : (string) the table used for the transaction (to ensure the maxParallelTests)

• workingDirFullPath : (string) the path to the directory where the user programs will be tested/saved/executed

• hardTimeoutInMs : (int) the global timeout for all programming languages (used to create if we can create a new root directory)

• hardMemoryLimitInKb - the hard memory limit in kb

• hardMaxDiskSpaceLimitInKb - the hard max disk space limit in kb

• maxParallelTests: (int) the max amount of parallel running tests, if more tests are requested they are rejected with an busy error (use this with the hard limits in the db to calculate the max ressource usage), set this to 0 or a negative value to no limit the amount of parallel tests

• maxNumberOfTestsWithOneRequest: (int) the number of tests that can be run with one request (only the first X tests will be executed ... the other will be ignored), this is used to limit the time a user can occupy the test server

• runner : (string) the shell/cmd command to execute in order to start the Runner (paths needs to be properly escaped) (you might need to use absolute paths)
  For Java you probably want UTF8 encoding... there is a command line argument for that, add
  -Dfile.encoding=UTF8
  after the java executable (else the user program would be called with utf8 encoding but the test runner cannot handle UTF8) also do not forget to correctly set the compile and exec commands in the PLang Table (add the -Dfile.encoding=UTF8 there too)!!!

• showTestRunnerDebugOutput: (int) 0/1, 1: show some internal debug output from the test runner for debugging (sent back as a part of the test response)

• maxLinesToRead: (int) the total (inclusive) number of lines to read from the user program (-x or 0 for unlimited)

• maxErrLinesToRead: (int) the total (inclusive) number of lines to read from the user program error stream (stderr) (-x or 0 for unlimited)

• maxLinesToWrite: (int) the total (inclusive) number of lines to write to the user program (-x or 0 for unlimited)

• useConfigFromDb: (bool) true: use the config in the database table, false: use the local config file, the dbConfig* (and dbConfigTableName) settings are used to connect the the mysql db and read the config settings

• dbConfigServer: (string) -''- but for the config table

• dbConfigName : (string) -''- but for the config table

• dbConfigUser : (string) -''- but for the config table

• dbConfigPw : (string) -''- but for the config table

• dbConfigTableName: (string) the table name with the configuration (the dbServer and dbName options are used)

• environmentVars : (string, \r\n or \n separated, key value pairs are separated by = OR NULL) the environment variables for the test runner. All environment vars are cleared and only these are set (even if this is the empty string). The special delimiters \n and = must not be used inside keys or values. The test runner is called and these environment variables are set. You can use "path"" to escape paths or other strings It is recommended that you at least specify the following: USER, HOME ,PATH

  • USER is the executing user e.g. yapextester
  • HOME is the home path for the executing user e.g. /home/yapextester
  • LANG the lang to use (e.g. for default formatting) e.g. de_DE.UTF-8
  • PATH are the paths that unix/windows should include when searching for e.g. programs /usr/local/bin:/usr/bin:/bin:/ Note that this is for linux, windows handles environment variables differently (e.g. other multi value separator than :) The test server splits in \n and uses these lines as input for proc_open
  • If the value is NULL then the environment variables from the php process are used. However, e.g. fast process manager (fpm) for php will discard environment variables by default. So the result might be different than expected.

If the test server cannot connect to the db then an error (db issue) is returned (so the local config is NOT used!)

If the config table has more than 1 row, the first row is taken!

Example

{
    "workingDirFullPath": "C:\\Users\\theju\\syndromTetsts",
    "dbServer": "localhost",
    "dbName": "syndromTestDb",
    "dbUser": "root",
    "dbPw": "admin",
    "dbTableName": "plangs",
    "dbTransactionTableName": "transactions",
    "maxParallelTests": 50,
    "hardTimeoutInMs": 4000,
    "hardMemoryLimitInKb": 2000,
    "hardMaxDiskSpaceLimitInKb": 2000,
    "maxNumberOfTestsWithOneRequest": 20,
    "runner": "\"C:\\Program Files (x86)\\JetBrains\\IntelliJ IDEA 14.1.4\\jdk1.8.0_65\\bin\\java\" -cp \"C:\\Users\\theju\\Documents\\WebProjects\\SyndromeAll\\DefaultTestRunner\\out\\production\\DefaultTestRunner\" Main",
    
    "environmentVars": "USER=yapextester\nHOME=/home/yapextester\nLANG=de_DE.UTF-8\nPATH=/usr/local/bin:/usr/bin:/bin:/"
}

Test Methods

Here is the list of the known test methods. Depending on the method the Runner will receive different command line arguments where the first arg is always the mode the Runner should run.

arg[0] = test type method (string) = compile | executeBlackBox | regex | justrun

Black Box Test

method name: blackBoxTest

arg[1] = the absolute directory path containing all files (including the main file)

arg[2] = the main file name to use

arg[3] = the compile command to execute

arg[4] = the execute command to run the test case

arg[5] = time limit (int) in MS time limit = time the program can run (hard limit) in MS e.g. 1000

arg[6] = memory limit (int) in KB memory limit = the max memory the program can use in KB .e.g. 1000 kb

arg[7] = disk limit (int) in KB disk limit = the max disk space the test can write to

arg[8] = contains a list of extensions of the soruce files for the p lang (differentiate beteen source files and assets). This is a "," separated list e.g. "cpp, h"

arg[9] = true to let the runner compile the sources, false to skip compilation (when running multiple tests skipping increases performance)

arg[10] = max lines to read from the user program (inclusive) (int) e.g. 100
negative or 0 is treated as unlimited

arg[11] = max lines to read from the user program stderr (inclusive) (int) e.g. 100
negative or 0 is treated as unlimited

arg[12] = max lines to write to the user program (inclusive) (int) e.g. 100
negative or 0 is treated as unlimited

arg[13] = a unique session id (e.g. used with firejail to manage overlay fs names)

arg[14] = "true" to enable debug output (this will leak internal paths!!) or "false" to disable debug output. The debug output will be normal lines starting with " [Test-Runner]"

Compile Test or Syntaxtest

method name: compileTest

arg[1] = the directory containing all files (the main file)

arg[2] = the main file name to use

arg[3] = the compile command to execute

arg[4] = contains a list of extensions of the soruce files for the p lang (differentiate beteen source files and assets). This is a "," separated list e.g. "cpp, h"

arg[5] = a unique session id

arg[6] = "true" to enable debug output (this will leak internal paths!!) or "false" to disable debug output. The debug output will be normal lines starting with " [Test-Runner]"

Regex Test

method name: regexTest

arg[1] = the directory containing all files (the main file)

arg[2] = the main file name to use

arg[3] = "true" to enable debug output (this will leak internal paths!!) or "false" to disable debug output. The debug output will be normal lines starting with " [Test-Runner]"

Compare Files Test

arg[1] = the absolute directory path containing all files (including the main file)

arg[2] = the main file name to use

arg[3] = the compile command to execute

arg[4] = the execute command to run the test case

arg[5] = time limit (int) in MS time limit = time the program can run (hard limit) in MS e.g. 1000

arg[6] = memory limit (int) in KB memory limit = the max memory the program can use in KB .e.g. 1000 kb

arg[7] = disk limit (int) in KB disk limit = the max disk space the test can write to

arg[8] = contains a list of extensions of the soruce files for the p lang (differentiate beteen source files and assets). This is a "," separated list e.g. "cpp, h"

arg[9] = true to let the runner compile the sources, false to skip compilation (when running multiple tests skipping increases performance)

arg[10] = max lines to read from the user program (inclusive) (int) e.g. 100
negative or 0 is treated as unlimited

arg[11] = max lines to read from the user program stderr (inclusive) (int) e.g. 100
negative or 0 is treated as unlimited

arg[12] = max lines to write to the user program (inclusive) (int) e.g. 100
negative or 0 is treated as unlimited

arg[13] = a unique session id (e.g. used with firejail to manage overlay fs names)

arg[14] = "true" to enable debug output (this will leak internal paths!!) or "false" to disable debug output. The debug output will be normal lines starting with " [Test-Runner]"

arg[15] = "file1:hash1 file2:hash2" a list of file:hash pairs separated with a single whitespace.
These are the files for the test (the assets) with the pre calculated hash or null (if not pre calculated). The test runner decides of the hash needs to be calculated or not (only files that should be compared need to be hashed).

For all tests

The actual test content is passed in via stdin because it might be too long for a console argument.

Just Run Test

method name: justRun

the arguments are the same as for the black box test

Test Runner Interface

The (Test) Runner executes the command that was passed to it from the (Test) Server. It is normally used to execute the users program and sandbox the execution.

Important

the test runner needs to replace every 0x00 character (\u0000 / (char)0) from the test protocol with another symbol else some databases will not be able to store the protocol

The replacement symbol should be ☐ (\u2610 or 0xE2 0x98 0x90 or ballot box) because this looks like a encoding error (and this is almost correct)

als the test runner needs to replace every \033 character (\033 or \u001B) into \033 (for java \\\\033) from the user program and the test input so that the frontend can display it properly (e.g. colors)

for java you can e.g. use

outPut = output.replaceAll("\033", "\\\\033");
trimmedTestLine = trimmedTestLine.replaceAll("\033", "\\\\033");

compile/execute commands #x args (see Ref 1)

Test Content

The test content depends on the used test method.

Compile Test

The compile test do not need any test content (therefore the test content will be ignored). It only checks if the given file(s) can be compiled

Black Box Test

Test content is an array of strings, the lines are NOT trimmed so all spaces are respected

If the first line with text starts with a '$' sign then the following text in this string are the console arguments that should be injected when the user program is called (escaping is the via "..." like in the normal shell/cmd).

• A line starting with a '<' in meant to be written to the user programs stdin (input stream), so input text starts at position 2 (position 1 is the <).
• A line starting with a '>' is expected to be outputted from the users program (stdout, output stream), so input text starts at position 2 (position 1 is the >).
• A line containing just a '>*'' is used as a wildcard that matches every output except no output (so the user program needs to output at least one character)
• A line starting with a 'e' interpreted as the expected program exit code, e.g. e5 would expect the user program to return 5

Note: Both e and

All other lines are ignored.

After every Test Protocol line the stderr stream is read (1 line) and added to the conversation if there was content. This not ensures that the stderr output is in the correct order or line!

After the program has terminated the trailing stderror lines are added to the conversation.
After this the trailing normal output lines are added to the conversation.

Just Run Test

Just as the black box test but without any input (except for console arguments).
The output from the user program is read and added to the protocol but nothing is written to the user program (except for console arguments).

Example

$ 2 Arg2 "Arg mit Leerzeichen"

<Eingabe 1
>Ausgabe 1

<Eingabe 2
<Eingabe 3

>Ausgabe 3
>Ausgabe 4

Compare files Test

All black-box-test commands are supported here.

But the compare files test lines must be the last in the test content (after all black-box-test commands). This is because they are executed after the user program has finished.

a line has the following pattern

ccompare [fileName1WithExtension] [fileName2WithExtension] (1) (2) (4) (8)

the file names can be separated with / for directories but must not include the text .. else the test is ignored (with an error information)

• options within () are optional if they are not set they are false
• the order of the options is arbitrary
• whitespaces are mandatory

The files can have the same name!

the files are compared line by line. a line might be separated by \n or \r\n

options

• 1: ignoreCase: while comparing lines the case is ignored (e.g. all to lower case)
• 2: ignoreLeadingWhitespaces: while comparing we ignore leading whitespace characters (whitespace and tab) in the user file line
• 4: ignoreTrailingWhitespaces: while comparing we ignore trailing whitespace characters (whitespace and tab) in the user file line
• 8: ignoreAllEmptyOrWhitespacesOnlyLines: while comparing we empty lines (whitespace and tab) in the user file line

all other lines are ignored

note that only the file 1 lines are compared with the user file (file 2). Thus the user file can have more lines, they are ignored

Because the files can have the same name the test runner must rename the file 1 (from the tutor) to some unique identifier (that cannot be guessed).

However the user could explore the files in the directory and may find the renamed file 1 and overwrite file 1 with empty content and the test would always be successful... To prevent this we calculate a hash (md5) from the file and after the user program finished we check if the md5 still matches else we fail the test with an error message

To speed things up we get the md5 hash for the files in the request. It's only necessary to calculate the md5 hash for files in the compare command. Other files are not required to be checked (for now).

Regex Test

The content of a regex test a rule on each line (empty lines are ignored).

There are 2 special properties all other properties are regular expressions.

special properties

• mode - (string | blacklist | whitelist) the used mode
  when blacklist is used then the regular expression(s) must not be contained in the file(s) (then the test will be successful),
  when whitelist is ued then the file(s) must contain the regular expression(s) (then the test will be successful)
  the default value is blacklist (if no mode is provided)
• connector - (string | && | ||) the connector to connect the regular expressions for every file
  the default value is && (if no connector is provided)

All other properties follow the pattern fileNameWithExtension : regex.
Where regex is an regular expression. A regular expression can be connected with other regular expressions via the && or || operator e.g. "/import/ || /package/". Because of this the regular expressions cannot contain the strings $$ and || strings

Instead of the fileNameWithExtension the expression * can be used which matches all files.

every fileNameWithExtension needs to be unique

The regex flags i and m are supported ... see examples

Examples

mode:blacklist
connector:&&
Main.java:/import
*:/^lang/

Main.java   :   /import/mi
*           :   /^lang/im

mode:blacklist
connector:||
Main.java:/([A-Z])\w+/ || /[0-9]/
Bruch.java:/import/i

Return-Codes

from the Test-Runner (testResultCode)

The Runner needs to return result code (exit code) to indicate the result of the test run.

• 0 - test was successful

• 1 - output mismatched

• 2 - some error occurred during execution of the users program (e.g. numberformat exception, any non 0 exit code will result in this)

• 3 - timeout hit / some read thread interrupted

• 4 - memory limit hit

• 5 - disk limit hit

• 6 - compare files line mismatched

• 50 - compile error

• 51 - test content read timeout

• 52 - exit mismatched, note that output mismatched has a higher priority (output mismatched is displayed of we have both errors), this is only generated if the test contained a check for the exit code

• 53 - error during compare files test

• 54 - some asset test file was changed or deleted by the user program (hash changed), the test protocol will contain some more information about the file

• 100 - some error occurred before or after executing the users program

• 101 - unknown test method (or unsupported)

Mac phpmyadmin

on mac make sure apache is running via sudo apachectl start