An answer contains the students files/solution for a single Task.


An assignment groups multiple Tasks. It has a title, start-date, deadline and some other meta-data. In real life it would be some form of worksheet you give to students.

(Docker) Containers

Containers are isolated processes that run on Linux machines. They are always based on an Image. Containers allow secure execution of submitted code.


An Evaluation is the result of a executed Evaluation Definition on a students Answer. It consists of multiple Evaluation Steps that have been executed by Evaluation Runners.

Evaluation Definition

The Evaluation Definition refers to the contents of the codefreak.yml file. It defines which Evaluation Runners should be run on an Answer and the configuration for them.

Evaluation Runner

An Evaluation Runner executes the automated evaluation of source code. There are different types of runners that accept different forms of configuration. Runners are always executed in Containers.

Evaluation Step

An Evaluation Step groups all Feedback that has been generated by an Evaluation Runner by testing the submitted source code. The result of an Evaluation Step can be "successful" or "failed". The step is only successful if all Feedback is "successful".


The smallest unit in the evaluation context is Feedback. It consists at least of a single "summary" that gives some helpful feedback on an Answer. A Feedback optionally refers to a file or parts of it. Feedback is generated automatically by all evaluation runners or can be created manually by creating comments.

(Docker) Image

An Image is the boilerplate for Containers. It contains all files that form a basic filesystem and some default instructions to run in containers.


A submission groups all Answers of a single User for one Assignment.


A task contains the boilerplate code the students start working on and the Evaluation Definition.

Task Pool

As a teacher you might use the same Tasks over and over again in different Assignments and/or semesters. The Task Pool allows you to define Tasks once and use them in different Assignments.