Skip to content
Snippets Groups Projects
Commit a109a0dc authored by Janis Daniel Dähne's avatar Janis Daniel Dähne
Browse files

added wildcard function for black box test output

parent abedc486
Branches
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
out/production/DefaultTestRunner/protocol.txt deleted 100644 → 0
+ 0
− 124
View file @ abedc486
protocol:
compile and execute cmd:
$1 - containing dir path
$2 - main file name
$3 - main file name with extension
$4 - main file absolute path
$5 - main file absolute path with extension
-- only if command = execute X (where X = black box test,) NOT FOR regex test, compile(syntax test)
$6 - timeout in ms
$7 - memory limit in kb
$8 - disk space limit the progrma can write to in kb
the compile/execute command needs to run in the shell/bash...
so the path quoting may be required...
you can assume that the command is like string that will be wrapped by "..." so you need to escape " (double quotes)
inside that are inside the wrapped command (e.g. paths or other arguments)
to escape a argument use \"arg\"
the arguments (event the $x) are not automatically escaped (e.g. whitspaced) because some commands may need
some special formatting e.g. "$4.exe\"
IT IS RECOMMENDED THAT EVERY ARGUMENT IS WRAPPED BY \"arg\"!
----- EXAMPLES ---
so e.g. to a compile command for JAVA:
\"C:\Program Files (x86)\JetBrains\IntelliJ IDEA 14.1.4\jdk1.8.0_65\bin\javac\" \"$5\"
and a execute command:
\"C:\Program Files (x86)\JetBrains\IntelliJ IDEA 14.1.4\jdk1.8.0_65\bin\java\" \"-cp\" \"$1\" \"$2\"
for C# it could be:
compile:
\"C:\Program Files (x86)\Mono\bin\mcs.bat\" \"$5\"
execute:
\"C:\Program Files (x86)\Mono\bin\mono\" \"$4.exe\"
or for PYTHON:
compile:
\"python\" \"-m\" \"py_compile\" \"$5\"
execute:
\"python\" \"$5\"
----------------------------------------------
arg[0] = test type (string)
method = compile | executeBlackBox | regex
compile: just compiles the given main file
executeBlackBox: compiles & executes a test case
regex: tests a regex expression against a file
---------------------------------- when method = regex
arg[1] = the directory containing all files (the main file)
arg[2] = the main file name to use
---------------------------------- when method = compile
arg[1] = the directory containing all files (the main file)
arg[2] = the main file name to use
arg[3] = the compile command to execute
---------------------------------- when method = executeBlackBox
arg[1] = the directory containing all files (the main file)
arg[2] = the main file name to use
arg[3] = the compile command to execute
arg[4] = the execute command to run the test case
arg[5] = time limit (int) in MS
time limit = time th program can run (hard limit) in MS e.g. 1000
arg[6] = memory limit (int) in KB
memory limit = the max memory the program can use in KB .e.g. 1000 kb
arg[7] = disk limit (int) in KB
disk limit = the max disk space the test can write to
-- the test content (see test protocol)
the test content is passed in via stdin because it might be too long for a console argument
---- test protocol is normally:
| IF the first line with text starts with a $ sign then the following strings
| (separated by a whitespace) are the arguments, a argument can contain whitespaces if its wrapped by
| "..."
|
< input line written to the process
> expected out line (red from the process)
RETURN VALUE:
0 - all ok (conversation is in stdout)
1 - output mismatch
2 - some error (during program execution e.g. numberformat exception....)
3 - timeout hit (conversation is in stdout)
4 - memory limit hit (conversation is in stdout)
5 - disk limit hit (conversation is in stdout)
50 - compile error
60 - unknown plang was requested : this should be returned by the php test server rather from this program!
conversation:
< input line that was written to the process
> the line the process has outputted (expected: ... if it was not the expected result)
error: an error occurred during the execution of the program
100 - some error (see stdout for error message, before or after program execution)
101 - no method matched...
\ No newline at end of file
src/Main.java
+ 50
− 10
View file @ a109a0dc
......@@ -6,6 +6,18 @@ import java.util.concurrent.*;
public class Main {
//use stdin for test content (1 line)
//TODO limit memory... currently we could use cgroups but problem with memswap (no permission and file not exists)
//this would be static and could not be changed for every test run
/*
firejail can use cgroup
sudo cgcreate -a appinventor -t appinventor -g memory:stest2
echo 5000000 > /sys/fs/cgroup/memory/stest2/memory.limit_in_bytes //10 MB
#set swapping to same value to not allow swapping
#klappt nicht, nicht da, keine berechtigung
echo 5000000 > /sys/fs/cgroup/memory/stest2/memory.memsw.limit_in_bytes
*/
/**
* true: outputs debug information to stdout
......@@ -48,6 +60,11 @@ public class Main {
*/
static Character expectedOutputFromProgramChar = '>';
/**
* the char that matches every user programs output (this is used when the output does not matter)
*/
static Character wildcardOutputMatchChar = '*';
/**
* the stored conversation / protocol of the test run
*/
......@@ -69,6 +86,15 @@ public class Main {
//moved to the test server
static String NewLineString = "\n";
/**
* the profile name for firejail
*/
static String FirejailProfilePath = "~/syndrom/syndrom.profile";
/**
* the cgroup name to use (currently not used)
*/
static String DefaultCGroupName = "syndrom";
public static void main(String[] myArgs) throws IOException {
......@@ -238,7 +264,7 @@ public class Main {
if (useDebugTestContent) {
testContents.add("$ 1 arg2 \"arg and 3\" arg4 10");
testContents.add("");
testContents.add("> Test2");
testContents.add("> 1");
//testContents.add("< Test Echo");
//testContents.add("> Test Echo");
}
......@@ -425,20 +451,34 @@ public class Main {
return 100;
}
if (Objects.equals(output.trim(), testLine.substring(1).trim())) {
String tr = testLine.substring(1);
String trimmedTestLine = tr.trim();
Conversation.add(testLine);
if (isDebug) {
System.out.println(testLine);
}
if (trimmedTestLine.length() > 0 && trimmedTestLine.charAt(0) == wildcardOutputMatchChar) {
//program output does not matter but we need output
Conversation.add(expectedOutputFromProgramChar + " " + toSafeLine(output.trim()));
} else {
Conversation.add(expectedOutputFromProgramChar + " " + toSafeLine(output.trim()) + " (expected: " + testLine.substring(1).trim() + ")");
if (isDebug) {
System.out.println(expectedOutputFromProgramChar + " " + toSafeLine(output.trim()) + " (expected: " + testLine.substring(1).trim() + ")");
//check if output matched
if (Objects.equals(output.trim(), trimmedTestLine)) {
Conversation.add(testLine);
if (isDebug) {
System.out.println(testLine);
}
} else {
Conversation.add(expectedOutputFromProgramChar + " " + toSafeLine(output.trim()));
Conversation.add("expected: " + trimmedTestLine);
if (isDebug) {
System.out.println(expectedOutputFromProgramChar + " " + toSafeLine(output.trim()));
System.out.print("expected: " + trimmedTestLine);
}
outputMismatch[0] = true;
}
outputMismatch[0] = true;
}
} else {
//ignore line
}
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment