Types of Evaluation

Code FREAK allows running automated evaluation on answers. There are different types of evaluation that each serves a different purpose. The evaluation types are also called runners. Each runner is configurable to fit most needs when examining source code.

The following is just an overview about the runners. For details on configuration please check out the configuration definition reference.

Unit/Functional Testing (junit)

This type of runner is useful for performing predefined unit tests on students source code. The junit-Runner performs tests by running a command and analysing a resulting jUnit XML file. This format can be generated by various unit-testing frameworks for different languages like:

These are just examples. Your favourite testing-tool might be able to generate jUnit compliant XML, too!

The junit runner will execute one or multiple commands in the container and afterwards look for jUnit XML files. The XML files then will be parsed and imported as Feedback.

Example Configuration:

The following example contains the default configuration for the junit runner. It uses the official gradle Docker image and runs the gradle test command on the source code in /home/gradle/project. After execution the runner will look for TEST-*.xml files in the results-path directory.

codefreak.yml
# ...
evaluation:
  - step: junit
    options:
      image: gradle
      project-path: /home/gradle/project
      results-path: build/test-results/test
      commands:
        - gradle test

Environment Variables

During evaluation of the answer the same dynamic environment variables as for the commandline runner ar available. See Commandline Environment Variables.

Static Code Analysis (codeclimate)

The static code analysis runner is currently based on Code Climate. Code Climate is a very powerful and extensible tool that gives helpful insight on issues in source code.

By default Code Climate runs some so called Default Checks. In addition there are a lot of plugins for different languages and purposes. You can find a list of Plugins here in the official documentation.

Code Climate is configured by using an external .codeclimate.yml or .codeclimate.json file. This file has to be included in the task boilerplate code but can be hidden by the codefreak.yml configuration. You can find the full configuration reference here.

Example Configuration

The following configuration simply runs the codeclimate step with default settings. In addition there is a .codeclimate.yml file alongside the source code that is hidden for students. The configuration enables the sonar-java plugin – an industry standard code analysis tool.

codefreak.yml
# ...
hidden:
  - .codeclimate.yml
evaluation:
  - step: codeclimate
.codeclimate.yml
plugins:
  sonar-java:
    enabled: true

CLI Commands (commandline)

The commandline runner allows to perform basic evaluations and examines the exit-code of shell-commands. The evaluation step is only successful if the exit-code of all commands is 0. With this runner you could check e.g. for existing directories or files.

Currently, the CLI commands are not executed in a shell-session. This means you cannot rely on variables or results from previous commands.

Example Configuration

The following example configuration checks with the test utility if the directory /home exists. This will either return the exit-code 0 if it exists or 1 if it does not exist (or is no directory).

codefreak.yml
# ...
evaluation:
  - step: commandline
    options:
      image: ubuntu
      commands:
        - /usr/bin/test -d /home

Environment Variables

During evaluation of the answer the following environment variables are available

Variable Name Example Value Explanation

CI

true

The value is always "true". Indicates running in a CI environment.

CODEFREAK_ANSWER_ID

3a7b9bb9-4efe-4435-9df3-89b1bf7c01ab

UUID of the answer currently processing

CODEFREAK_ASSIGNMENT_ID

d420f578-fb0c-4974-afbe-42ec5856ab3b

UUID of the assignment currently processing. The variable might be empty if running in task-pool testing mode.

CODEFREAK_SUBMISSION_ID

3a24a325-bb5a-4de8-bbe7-cd99ca02fa19

UUID of the submission currently processing

CODEFREAK_TASK_ID

6d476ae7-9d45-4153-9157-a046192c40dd

UUID of the task currently processing

CODEFREAK_USER_ID

aefa1307-f247-4440-b1b1-ccc37f6563b9

UUID of the user the answer belongs to

CODEFREAK_USER_FIRST_NAME

Jane

First name of the user the answer belongs to

CODEFREAK_USER_LAST_NAME

Doe

Last name of the user this answer belongs to

CODEFREAK_USER_USERNAME

jane.doe@student.example.org

Username/mail address of the user this answer belongs to

Comments (comments)

The comments runner is a placeholder currently. Comments are always possible and don’t have to be enabled in configuration.

Comments are not added by runners but can be added by teachers via the UI. Open the Answer page of a submission and click the + button next to the line number of a file.