Types of Evaluation

Code FREAK allows running automated evaluation on answers. There are different types of evaluation that each serves a different purpose. The evaluation types are also called runners. Each runner is configurable to fit most needs when examining source code.

The following is just an overview about the runners. For details on configuration please check out the configuration definition reference.

Unit/Functional Testing (junit)

This type of runner is useful for performing predefined unit tests on students source code. The junit-Runner performs tests by running a command and analysing a resulting jUnit XML file. This format can be generated by various unit-testing frameworks for different languages like:

These are just examples. Your favourite testing-tool might be able to generate jUnit compliant XML, too!

The junit runner will execute one or multiple commands in the container and afterwards look for jUnit XML files. The XML files then will be parsed and imported as Feedback.

Example Configuration:

The following example contains the default configuration for the junit runner. It uses the official gradle Docker image and runs the gradle test command on the source code in /home/gradle/project. After execution the runner will look for TEST-*.xml files in the results-path directory.

# ...
  - step: junit
      image: gradle
      project-path: /home/gradle/project
      results-path: build/test-results/test
        - gradle test

Environment Variables

During evaluation of the answer the same dynamic environment variables as for the commandline runner ar available. See Commandline Environment Variables.

Static Code Analysis (codeclimate)

The static code analysis runner is currently based on Code Climate. Code Climate is a very powerful and extensible tool that gives helpful insight on issues in source code.

By default Code Climate runs some so called Default Checks. In addition there are a lot of plugins for different languages and purposes. You can find a list of Plugins here in the official documentation.

Code Climate is configured by using an external .codeclimate.yml or .codeclimate.json file. This file has to be included in the task boilerplate code but can be hidden by the codefreak.yml configuration. You can find the full configuration reference here.

Example Configuration

The following configuration simply runs the codeclimate step with default settings. In addition there is a .codeclimate.yml file alongside the source code that is hidden for students. The configuration enables the sonar-java plugin – an industry standard code analysis tool.

# ...
  - .codeclimate.yml
  - step: codeclimate
    enabled: true

CLI Commands (commandline)

The commandline runner allows to perform basic evaluations and examines the exit-code of shell-commands. The evaluation step is only successful if the exit-code of all commands is 0. With this runner you could check e.g. for existing directories or files.

Currently, the CLI commands are not executed in a shell-session. This means you cannot rely on variables or results from previous commands.

Example Configuration

The following example configuration checks with the test utility if the directory /home exists. This will either return the exit-code 0 if it exists or 1 if it does not exist (or is no directory).

# ...
  - step: commandline
      image: ubuntu
        - /usr/bin/test -d /home

Environment Variables

During evaluation of the answer the following environment variables are available

Variable Name Example Value Explanation



The value is always "true". Indicates running in a CI environment.



UUID of the answer currently processing



UUID of the assignment currently processing. The variable might be empty if running in task-pool testing mode.



UUID of the submission currently processing



UUID of the task currently processing



UUID of the user the answer belongs to



First name of the user the answer belongs to



Last name of the user this answer belongs to



Username/mail address of the user this answer belongs to

Comments (comments)

The comments runner is a placeholder currently. Comments are always possible and don’t have to be enabled in configuration.

Comments are not added by runners but can be added by teachers via the UI. Open the Answer page of a submission and click the + button next to the line number of a file.