Default Test Runner (for Syndrom)

Requirements

The current version reqires

• Java 1.8

Return Codes

The return codes and workflow is described in the test server.
See https://gitlab.informatik.uni-halle.de/Syndrom/TestServer

All output written to stdout is normally captured by the test-server and forwarded to the client

Idea

We go through the test content and apply it line by line e.g. read line, then write line, ...

This has the disadvantage that some test will pass that should not ... the input/output order might be wrong This is because we use bufferedReader so if the user program writes something before we write to it, the user program output is stored in the buffer

So resolve this we would have to check the buffer if it has content before we write to the user program. Unfortunately we currently don't know any approach to check the buffer without blocking the thread till we get input e.g. BufferedReader.ready() will block if we have no content or can give invalid results...

see Known Issues > Test protocol input/output order

Different Idea

Use something like test runner create process: /bin/bash -c ([user prog] 2>&1) | tee protocol.txt see https://www.linuxquestions.org/questions/linux-software-2/bash-how-to-redirect-output-to-file-and-still-have-it-on-screen-412611/

• use a sub shell to ensure stdout and stderr are synchronized
• pipe stdout and stderr
• use tee to redirect stdout and stderr to a file

The idea is that the test runner then writes input for the user program to protocol.txt and then to the user program The output from the user program is automatically written to protocol.txt as well as the stderr (also synchronized via sub shell)

The problem is that we don't know which line in protocol.txt is input/output/stderr so we would need to create a custom program (e.g. java) as tee

Another problem (at least i think it's a problem) is synchronization between runner writing to protocol.txt and output from the user program to protocol.txt

e.g. if the user program writes in an infinite loop (e.g. "output") and we write to the user program (e.g. "input") we could get something like <output>input and all permutations of this because both processes writing the to file at the same time

Another problem is that we would write all input at the start to the program (so we could also use pipes to feed the user program with input) this is not what we want in some tests e.g.

<1
>1
<2
>2

When we write all input at once to the user program the user program buffer might get filled up (theoretically, actually normal tests should not be that big)

After some testing with Queue<String> conversionProtocol = new ConcurrentLinkedQueue<String>(); and we fill the queue async

we get for the TestMediendatenbank (see below) the output

---- queue ---- 
<Musik
<Best of OOP-Hits 2017
<OOP-Team
 --> Best of OOP-Hits 2017 von OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation
 --> Best of OOP-Hits 2017 (Dokumentation)
 --> false
---- end queue ---- 

for the first second test which is good!

BUT we need to wait after each write to the user program ~100ms else (if we don't wait) we get

---- queue ---- 
<Musik
<Best of OOP-Hits 2017
<OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation

 --> Best of OOP-Hits 2017 von OOP-TeamBest of OOP-Hits 2017 (Dokumentation)
false
---- end queue ---- 

probably because the writes are too fast compared to the time the user program needs to write back it gets worse if the user program does some other stuff before writing (then me might need to wait ~200ms)

this slows down the tests drastically and we don't know how much we need to wait... so async is probably not the way to go

That being said we could get some value from this when we async read user program output too in cases where the protocol starts with a read the runner blocks but the user program might have written something

in this case we could capture the output from the user program and show that? instead of no output because the runner is blocked

After some more testing with globbers and normal reads it seems that globber and normal reading from the same stream (user program output) will somehow corrupt the stream (sometimes) so both (or only one) globber/normal bufferedReader can lead to weird results e.g.

(we added a TTTTTTTEEEEESSSST output to the class)

---Queue
<Musik
<Best of OOP-Hits 2017
<OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation
>TTTTTTTEEEEESSSSTBest of OOP-Hits 2017 von OOP-Team
>false
---END
exit:0
<Musik
<Best of OOP-Hits 2017
<OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation
>
expected:Best of OOP-Hits 2017 von OOP-Team
>Best of OOP-Hits 2017 (Dokumentation)
expected:false
EOT

where queue has two lines in one (TTTTTTTEEEEESSSSTBest of OOP-Hits 2017 von OOP-Team) and the normal output is missing a line (just empty >)

if we only use the normal reads then we get

exit:0
<Musik
<Best of OOP-Hits 2017
<OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation
>TTTTTTTEEEEESSSST
expected:Best of OOP-Hits 2017 von OOP-Team
>Best of OOP-Hits 2017 von OOP-Team
expected:Best of OOP-Hits 2017 (Dokumentation)
>Best of OOP-Hits 2017 (Dokumentation)
expected:false
EOT
>false
expected:

where nothing is missing

Known Issues

Test protocol input/output order

e.g.

import java.io.*;
public class TestMediendatenbank {
    static BufferedReader in;
    public static void main(String[] args) throws IOException {
        in = new BufferedReader(new InputStreamReader(System.in));
        testeMedien();
    }
    public static void testeMedien() throws IOException {
        Medium m1 = erzeugeMedium();
        System.out.println(m1);
        
        Medium m2 = erzeugeMedium();
        System.out.println(m2);
        
        System.out.println(m1.equals(m2));
    }

    public static Medium erzeugeMedium() throws IOException {
        String wahl = in.readLine();
        String titel; int spielzeit;
        titel = in.readLine();
        switch(wahl) {
            case "Musik":
                return new MusikCD(titel, in.readLine());
            case "Film":
                return new FilmDVD(titel, in.readLine());
            default:
                return null;
        }
    }
}

with the test (1. version)

<Musik
<Best of OOP-Hits 2017
<OOP-Team
>Best of OOP-Hits 2017 von OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation
>Best of OOP-Hits 2017 (Dokumentation)
>false

which obviously works..

But the following test (2. version) will succeed but should fail!

<Musik
<Best of OOP-Hits 2017
<OOP-Team
<Film
<Best of OOP-Hits 2017
<Dokumentation
>Best of OOP-Hits 2017 von OOP-Team
>Best of OOP-Hits 2017 (Dokumentation)
>false

because the user program outputs a line after the 3rd test line.

The test runner cannot handle this properly because it uses buffered readers/writers. Actually before every write line (to the user program) there is no check if the user program outputs anything... theoretically this would help to solve this problem but e.g. in the example above the line System.out.println(m1); (where we write Film to the program)... So before writing Film we would check if the user program wrote something (m1). But the check is done before the creation of m1 finished and thus before m1 is outputted. This can be solve if we wait e.g. 1s before checking but this is not clean and does not solves common cases

So the only way to solve this is to rewrite the test like the first version.

--> all outputs from the user program are buffered until we really expect the read from the user program

Timeout online but working locally

You test your program locally and input text and everything is working but in the online version there is a timeout and no output.

This is probably because of the re creation of some reader e.g. BufferedReader

import java.io.*;
public class TestMediendatenbank {
    public static void main(String[] args) throws IOException {
        testeMedien();
    }
    public static void testeMedien() throws IOException {
        Medium m1 = erzeugeMedium();
//        System.out.println(m1);//Variante 2 funktioniert
        Medium m2 = erzeugeMedium();
        System.out.println(m2);
        System.out.println(m1);//Variante 1 funktioniert nicht
        System.out.println(m1.equals(m2));
    }

    public static Medium erzeugeMedium() throws IOException {
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        String wahl = in.readLine();
        String titel; int spielzeit;
        titel = in.readLine();
        switch(wahl) {
            case "Musik":
                return new MusikCD(titel, in.readLine());
            case "Film":
                return new FilmDVD(titel, in.readLine());
            default:
                return null;
        }
    }
}

then somehow the input is broken so the user program no longer receives our input. We could also re create our readers/writers in the loop where we write/read to/from the user program. This sometimes work but only if the timing is right. There are some cases (found through testing) where the creation of BufferedReader gets out of sync and the user program still don' get input...

The only known fix is to create the BufferedReader only one time in the user program e.g.

import java.io.*;
public class TestMediendatenbank {
    
    static BufferedReader in = null;
    public static void main(String[] args) throws IOException {
        in = new BufferedReader(new InputStreamReader(System.in));
        testeMedien();
    }
    public static void testeMedien() throws IOException {
        Medium m1 = erzeugeMedium();
        //System.out.println(m1);//Variante 2 funktioniert
        Medium m2 = erzeugeMedium();
        //System.out.println("test");
        System.out.println(m2);
        System.out.println(m1);//Variante 1 funktioniert nicht
        System.out.println(m1.equals(m2));
    }

    public static Medium erzeugeMedium() throws IOException {
        String wahl = in.readLine();
        String titel; int spielzeit;
        titel = in.readLine();
        switch(wahl) {
            case "Musik":
                return new MusikCD(titel, in.readLine());
            case "Film":
                return new FilmDVD(titel, in.readLine());
            default:
                return null;
        }
    }
}

It works locally because the terminal somehow fixes this...