Definition Files

Table of Contents

In Code FREAK, assignments and tasks can be defined YAML files. This includes the configuration of automated evaluation.

1. Assignment

An assignment is defined by a directory (or packaged archive) containing a codefreak.yml with the following schema.

Table 1. Assignment Object
Field Name Type Description

title

string

REQUIRED. The name of the assignment.

tasks

[string]

List of paths to task definitions (relative to the assignment).

Example
title: My Assignment
tasks:
  - my-task-1
  - my-task-2
Corresponding file structure
├─ codefreak.yml
├─ my-task-1
│  ├─ codefreak.yml
│  ├─ main.c
├─ my-task-2
│  ├─ codefreak.yml
│  ├─ ...
├─ ...

2. Task

A task is defined by a directory (or packaged archive) containing source code files and a codefreak.yml with the following schema. The codefreak.yml will be excluded from the source code files.

Table 2. Assignment Object
Field Name Type Description

title

string

REQUIRED. The name of the task.

description

string

Detailed instructions for students. Basic markdown is supported.

hidden

[string]

Patterns of files that should be hidden from students. Matching files are only included for evaluation. If matching files are created by students, they are ignored for evaluation.

protected

[string]

Patterns of files that should be read-only. Students will be able to see matching files but modifications are ignored for evaluation. Non-existent files can be protected to prevent their creation.

evaluation

Map[string, Evaluation Step Object]

Map of key to an evaluation step object. They key acts as unique identifier for the evaluation step within this task.

File patterns use the Ant pattern syntax. For more information refer to the official documentation.
Example
title: My Task
description: Write a function `add(int a, int b)` that returns the sum of `a` and `b`
hidden:
  - src/test/**
  - .codeclimate.yml
protected:
  - build.gradle
  - settings.gradle
evaluation:
  my-step:
    title: My first check
    script: |
      echo "Hello World"
    report:
      format: default

3. Evaluation

Automatic evaluation consists one or more steps Each step consists of a script that evaluates the code and a report parser, that will parse the result into a list of feedback. An evaluation step is defined as follows.

Table 3. Evaluation Step Object
Field Name Type Description

script

string

REQUIRED. A (bash) script that will be executed for evaluation.

title

string

An optional human readable name for the evaluation step. Summarize briefly for your students what this evaluation step checks (e.g. "Test functions.py", "Class XYZ Tests", "Code Style", …​).

timeout

integer

An optional custom timeout in seconds after which the evaluation step will be canceled and marked as failed. There is a system-wide default timeout of 5min.

report.format

string

REQUIRED. The id/key of one evaluation report parser that will try to parse the result into a list of feedback. See Report Parsers for a list of currently supported parsers.

report.path

string

A path or matching-pattern for a file that will be read by the report parser. If a blank path is specified the console output will be passed to the report parser.