Types of Evaluation
Code FREAK allows running automated evaluation on answers. There are different types of evaluation that each serves a different purpose. The evaluation types are also called runners. Each runner is configurable to fit most needs when examining source code.
The following is just an overview about the runners. For details on configuration please check out the configuration definition reference.
Unit/Functional Testing (junit
)
This type of runner is useful for performing predefined unit tests on students source code. The junit-Runner performs tests by running a command and analysing a resulting jUnit XML file. This format can be generated by various unit-testing frameworks for different languages like:
-
jUnit (Java)
-
Jest (JavaScript)
-
pytest (Python)
-
Google Test (C++)
-
PHPUnit (PHP)
These are just examples. Your favourite testing-tool might be able to generate jUnit compliant XML, too!
The junit
runner will execute one or multiple commands in the container and afterwards look for jUnit XML files. The XML files then will be parsed and imported as Feedback.
Example Configuration:
The following example contains the default configuration for the junit
runner. It uses the official gradle
Docker image and runs the gradle test
command on the source code in /home/gradle/project
. After execution the runner will look for TEST-*.xml
files in the results-path
directory.
# ...
evaluation:
- step: junit
options:
image: gradle
project-path: /home/gradle/project
results-path: build/test-results/test
commands:
- gradle test
Environment Variables
During evaluation of the answer the same dynamic environment variables as for the commandline
runner ar available. See Commandline Environment Variables.
Static Code Analysis (codeclimate
)
The static code analysis runner is currently based on Code Climate. Code Climate is a very powerful and extensible tool that gives helpful insight on issues in source code.
By default Code Climate runs some so called Default Checks. In addition there are a lot of plugins for different languages and purposes. You can find a list of Plugins here in the official documentation.
Code Climate is configured by using an external .codeclimate.yml
or .codeclimate.json
file. This file has to be included in the task boilerplate code but can be hidden by the codefreak.yml
configuration. You can find the full configuration reference here.
Example Configuration
The following configuration simply runs the codeclimate
step with default settings. In addition there is a .codeclimate.yml
file alongside the source code that is hidden for students. The configuration enables the sonar-java
plugin – an industry standard code analysis tool.
# ...
hidden:
- .codeclimate.yml
evaluation:
- step: codeclimate
plugins:
sonar-java:
enabled: true
CLI Commands (commandline
)
The commandline runner allows to perform basic evaluations and examines the exit-code of shell-commands. The evaluation step is only successful if the exit-code of all commands is 0
. With this runner you could check e.g. for existing directories or files.
Currently, the CLI commands are not executed in a shell-session. This means you cannot rely on variables or results from previous commands. |
Example Configuration
The following example configuration checks with the test utility if the directory /home
exists. This will either return the exit-code 0
if it exists or 1
if it does not exist (or is no directory).
# ...
evaluation:
- step: commandline
options:
image: ubuntu
commands:
- /usr/bin/test -d /home
Environment Variables
During evaluation of the answer the following environment variables are available
Variable Name | Example Value | Explanation |
---|---|---|
|
|
The value is always "true". Indicates running in a CI environment. |
|
|
UUID of the answer currently processing |
|
|
UUID of the assignment currently processing. The variable might be empty if running in task-pool testing mode. |
|
|
UUID of the submission currently processing |
|
|
UUID of the task currently processing |
|
|
UUID of the user the answer belongs to |
|
Jane |
First name of the user the answer belongs to |
|
Doe |
Last name of the user this answer belongs to |
|
Username/mail address of the user this answer belongs to |
Comments (comments
)
The comments runner is a placeholder currently. Comments are always possible and don’t have to be enabled in configuration.
Comments are not added by runners but can be added by teachers via the UI. Open the Answer page of a submission and click the +
button next to the line number of a file.