Definition Files

Table of Contents

1. Assignment
2. Task
3. Evaluation

In Code FREAK, assignments and tasks can be defined YAML files. This includes the configuration of automated evaluation.

1. Assignment

An assignment is defined by a directory (or packaged archive) containing a codefreak.yml with the following schema.

Table 1. Assignment Object
Field Name	Type	Description
title	`string`	REQUIRED. The name of the assignment.
tasks	`[string]`	List of paths to task definitions (relative to the assignment).

Example

title: My Assignment
tasks:
  - my-task-1
  - my-task-2

Corresponding file structure

├─ codefreak.yml
├─ my-task-1
│  ├─ codefreak.yml
│  ├─ main.c
├─ my-task-2
│  ├─ codefreak.yml
│  ├─ ...
├─ ...

Complete example

2. Task

A task is defined by a directory (or packaged archive) containing source code files and a codefreak.yml with the following schema. The codefreak.yml will be excluded from the source code files.

Table 2. Assignment Object
Field Name	Type	Description
title	`string`	REQUIRED. The name of the task.
description	`string`	Detailed instructions for students. Basic markdown is supported.
hidden	[`string`]	Patterns of files that should be hidden from students. Matching files are only included for evaluation. If matching files are created by students, they are ignored for evaluation.
protected	[`string`]	Patterns of files that should be read-only. Students will be able to see matching files but modifications are ignored for evaluation. Non-existent files can be protected to prevent their creation.
evaluation	[Evaluation Step Object]	List of automatic evaluation steps.

File patterns use the Ant pattern syntax. For more information refer to the official documentation.

Example

title: My Task
description: Write a function `add(int a, int b)` that returns the sum of `a` and `b`
hidden:
  - src/test/**
  - .codeclimate.yml
protected:
  - build.gradle
  - settings.gradle
evaluation:
  - step: codeclimate
  - step: junit

3. Evaluation

Automatic evaluation consists one or more steps. Each step is implemented by a so-called runner. Runners take the student submissions and execute e.g. static code analysis. They then collect the results to give feedback to the student and support the teacher’s grading. The same runner can be used in multiple steps (usually with different configuration). An evaluation step is defined as follows.

Table 3. Evaluation Step Object
Field Name	Type	Description
step	`string`	REQUIRED. The name of the runner. See below for supported ones.
title	`string`	An optional human readable name for the evaluation step. Summarize briefly for your students what this evaluation step checks (e.g. "Test functions.py", "Class XYZ Tests", "Code Style", …). The default title depends on the runner.
timeout	`integer`	An optional custom timeout in seconds after which the evaluation step will be canceled and marked as failed. There is a system-wide default timeout of 5min.
options	Map[`string`, ?]	Runner-specific configuration. See below for details.

3.1. Command Line Runner

Name: commandline

This is the most basic runner. It executes arbitrary commands and uses the output as feedback. The evaluation result is based on exit codes of the commands.

Table 4. Command Line Runner Options
Key	Type	Description
image	`string`	REQUIRED. The name of the Docker image in which the commands are run.
project-path	`string`	REQUIRED. Absolute path of where the student submission is placed.
commands	[`string`]	List of commands that executed. Working directory is the `project-path`.
stop-on-fail	`boolean`	Whether to stop when the first command fails or always execute all commands. Default is `true`.

Example

evaluation:
  - step: commandline
    options:
      image: gradle
      project-path: /home/gradle/project
      commands:
        - pwd
        - gradle test
        - ls build/test-results/test

3.2. Code Climate Runner

Name: codeclimate

This runner performs static code analysis via Code Climate. It is configured by including a .codeclimate.yml in the task files. See official documentation for details.

Example

evaluation:
  - step: codeclimate

Example .codeclimate.yml

plugins:
  sonar-java:
    enabled: true

3.3. JUnit Runner

Name: junit

This runner executes unit tests in Java projects via JUnit. It is pre-configured to work with standard Gradle projects.

Table 5. JUnit Runner Options
Key	Type	Description
image	`string`	Inherited from Command Line Runner. Default is `gradle`.
project-path	`string`	Inherited from Command Line Runner. Default is `/home/gradle/project`.
commands	[`string`]	Inherited from Command Line Runner. Default is `["gradle testClasses", "gradle test"]`.
stop-on-fail	`boolean`	Inherited from Command Line Runner. Default is `true`.

Example

evaluation:
  - step: junit