Creating assignments

In this chapter you will learn how to create an assignment. We will create a simple Python application with a single function add(a: int, b: int).

This chapter covers the following:

  • How to create an assignment and tasks

  • How to provide a basic file boilerplate to your students

  • How to define evaluations for your tasks

  • Running your tests locally

1. Prepare your tasks

Preparation is key! Don’t start working with Code FREAK if you haven’t thought about what to test and how to test. Before you start adding Code FREAK configuration to your tasks, make sure they are working/compiling locally on your machine! You can also test execution like Code FREAK does locally by installing Docker. More on this in section Testing locally with Docker.

Code FREAK allows students to write and execute their code without leaving the browser. Currently, we are using a browser-port of Microsoft’s VSCode by Coder. We recommend you also use VSCode for creating your tasks locally.

2. Creating the student boilerplate code

For this tutorial we use the following Python file with a single function add(a: int, b:int) → int. You can either let your student create these file or at least provide some initial template (boilerplate) to give them a fair starting point.
def add(a: int, b: int) -> int:
    # TODO: write the algorithm to add a and b
    return 0 (1)

if __name__ == "__main__":
    print("2 + 3 = %s" % add(2, 3))
1 This is the line the student has to correct to finish this task.

3. Writing tests

Which form of testing you choose for your tasks heavily depends on the tasks requirements. If students have to work on algorithms you should use some form of unit or functional testing. If they create a simple I/O application you can use a library that tests input and output.

In this example we will use pytest, a testing framework for Python. It is important that your testing framework supports JUnit XML output so Code FREAK is able to parse the results.

Alongside the file we will create a file that will contain some assertions to verify the add function.
from main import add

def test_function():
    assert add(2, 3) == 5
    assert add(-1, 2) == 1

You should verify that this works as expected on you local computer before continuing. We run the tests by executing the following command in the terminal.


This should give some output about a failing test. It is important that your tests are failing. Otherwise, you could not expect your students to fix the faulty source code.

Failing pytest output
==================== test session starts =====================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/user/python-example
collected 1 item F                                         [100%]

========================== FAILURES ==========================
_______________________ test_function ________________________

    def test_function():
>       assert add(2, 3) == 5
E       assert 0 == 5
E        +  where 0 = add(2, 3) AssertionError
================== short test summary info ===================
FAILED - assert 0 == 5
===================== 1 failed in 0.04s ======================

4. Checking Code Quality

There are many tools available for doing Static Code Analysis. Static Code Analysers can check code for possible bugs, proper formatting and many more cool things. They are also often called "linters". In contrast to Unit Testing you do not have to write individual tests as most static analysers ship with many built-in tests. A popular linting tool for Python is "Pylint" which will test if the code is formatted according to PEP8, and it will also find possible errors.

Pylint can be installed using pip install pylint and invoked using pylint [], e.g. pylint If Pylint produces no output it meant your code is fine. Otherwise, it will show a list of issues:

Example Pylint output
************* Module main C0114: Missing module docstring (missing-module-docstring) C0103: Argument name "a" doesn't conform to snake_case naming style (invalid-name) C0103: Argument name "b" doesn't conform to snake_case naming style (invalid-name) C0116: Missing function or method docstring (missing-function-docstring) W0613: Unused argument 'a' (unused-argument) W0613: Unused argument 'b' (unused-argument)

Your code has been rated at -5.00/10 (previous run: -5.00/10, +0.00)

If you call pylint with --output-format=json it will print the issues in a format that can be read by Code FREAK. This will be important when we prepare the configuration for Code FREAK.

5. Creating the basic directory layout

You should already have a file and a file. Transform your directory layout to match the following.

/ (1)
├─ codefreak.yml (2)
├─ task-1/ (3)
│  ├─ codefreak.yml (4)
│  ├─ (5)
│  ├─ (6)
1 The / directory is the root of your assignment.
2 The codefreak.yml directory in your project contains the definition of the assignment. See Assignment’s codefreak.yml for further explanation on its contents.
3 The task-1 directory is the root directory of your Task. You can create as many tasks per project as you like. For this example we only have one task in our assignment.
4 The task-level codefreak.yml contains the configuration of your task, how it is evaluated, etc. See Task’s codefreak.yml for more information on its contents.
5 The is an example of boilerplate code you give to your students. You can also create tasks without any boilerplate code at all if you want your students to create everything on their on.
6 The file contains the tests we will run on our students code. Even if it is visible now we will hide it from students by using the task’s codefreak.yml.

6. Add Code FREAK configuration

We now have to add two configuration files to the directory-layout, so Code FREAK knows how to evaluate the task. As explained in the previous section there are two different codefreak.yml files: One for your assignment and one for each task.

Please see the Definition Files Chapter for a full reference of the codefreak.yml files.

6.1. Assignment’s codefreak.yml

Don’t worry you have never worked with YAML (.yml) files. The syntax is pretty straight forward and easy to read.
title: "Python assignment"
  - task-1

The codefreak.yml in your assignment root-directory is very slim. It only contains the title of your assignment and a list of directories that contain the individual tasks.

6.2. Task’s codefreak.yml

The codefreak.yml file in your task directory is where the magic happens. We will define two evaluation steps for our task: A static code analysis with Code Climate (because the config is super easy) and a unit test with pytest.

title: Create an addition function in Python (1)
description: |
   Please create an `add(a: int, b: int): int` function in the
  `` file, that returns the sum of parameter `a` and `b`.

   Find some useful help in the [official reference](
  - "**/*" (2)
evaluation: (3)
  unit-tests: (4)
    title: Unit Tests
    script: |-
      pip3 install pytest
      pytest --junitxml=test-results.xml
      format: junit-xml
      path: test-results.xml
  code-quality: (5)
    title: Code Quality
    script: |-
      pip3 install pylint
      pylint --ignore-patterns='.*?' --output-format=json -ry *.py > pylint-report.json
      format: pylint-json
      path: pylint-report.json
1 The first two lines will add a title and some description to our task. The weird description syntax is a multi line string in YAML. The description allows basic Markdown syntax.
2 In the hidden property we define a list of directories and/or files that will be hidden from our students. Each instruction is a glob pattern. The codefreak.yml file is ALWAYS hidden for students.
3 Under evaluation: you can list one or multiple evaluation-steps. Each evaluation step has a unique key which will identify the job internally. In this case we define the steps unit-tests and code-quality.
4 The unit-tests step will invoke pytest and write a JUnit-compatible XML file to test-results.xml. This file will be picked up by the report parser junit-xml.
5 Our second evaluation step code-quality will simply invoke pylint and create machine-readable JSON output. Because pylint does not have a special parameter to write the report to a file we have to use bash output redirection to write the output to a dedicated file. The asterisk means we will check all .py files, except those suffixed with

7. Testing locally with Docker

Before you upload your assignment to Code FREAK you should try to run the tests locally with Docker. After installing Docker you can use the following command-template to test execution locally:

docker run -it --rm \
       -v ${PWD}:/home/coder/project \
       --entrypoint "" \
       cfreak/ide:1 \
       sh -c 'pip3 install pytest && python -m pytest --junitxml=test-results.xml'
If you are on Windows replace the $PWD variable with the absolute path to your task-1 directory. On Mac OS and Linux the $PWD variable will be replaced accordingly.

8. Creating and uploading an assignment

If your assignment is ready to be uploaded to Code FREAK you have to create a zip or tar archive. The archive name itself does not have to follow any naming scheme.

The root directory of the archive has to contain the assignment codefreak.yml file! So do not create a archive that contains another assignment-123 directory. Be warned that many archive programs do this by default. The safest way is selecting the codefreak.yml file and all task-* directory and use your context menu to create an archive of these file.


├─ codefreak.yml
├─ task-1/
│  ├─ codefreak.yml
│  ├─
│  ├─


├─ my-assignment/
│  ├─ codefreak.yml
│  ├─ task-1/
│  │  ├─ codefreak.yml
│  │  ├─
│  │  ├─
Code FREAK executes all commands on Linux. There might be cases where the file permissions are important. Either use a tar archive to retain file permissions or set the correct permissions in one of your commands.

After you created your archive go to your Code FREAK installation and create a new assignment by uploading the archive.