Programmatically place comments

codePost makes it easy to manually annotate student code. If you also run autograding scripts to test student code for correctness, you might want to programmatically annotate student code to show which code passed correctness tests, and which code failed. That would save all the effort of manually reading through autograder output and placing comments in student code by hand.

Let's learn how to programmatically place comments on student code by parsing autograder output.

You can find all of the code used in this tutorial here.

1. The goal

Let's say our students submitted one homework file called homework.py. Don't worry, we'll talk about how to extend this tutorial to other languages and multiple files at the end.

homework.py contains a bunch of methods. Our autograder script tests each of these methods by calling them with various inputs, and comparing the student code's output with expected output. Our autograder logs its output in a .txt called tests.txt.

Autograder Summary

-- Test 1 --
Function: find_max
Input: [-1, -2]
Expected output: -1
Actual output: 0
Result: Fail

-- Test 2 --
Function: reverse
Input: [1,2,3,4]
Expected output: [4,3,2,1]
Actual output: [4,2,3,1]
Result: Fail
...

Alongside the definition of each function written by the student, we want to place a codePost comment that shows the output of the autograder tests corresponding to that function. We'll end up with a submission that looks like this.

3354 — A homework file with autograder output summarized next to each function definition.

We're going to assume that we've already uploaded student submissions to codePost (if you need help uploading, check out this tutorial).

We need to do two things:
(1) parse the output of tests.txt
(2) place comments in homework.py.

We'll start with #2, which will show us exactly what we need to accomplish in #1.

2. Placing comments using the codePost API

Since we're going to place comments on each student's submission, we need to loop through all of the submissions corresponding to this assignment. Here's how we can do that with the codePost API.

#!/usr/local/bin/python3

import codepost

codepost.configure_api_key(api_key="YOUR API KEY")

# retrieve your course
this_course = codepost.course.list_available(name="NAME", period="PERIOD")[0]

# retrieve the assignment
this_assignment = this_course.assignments.by_name(name="ASSIGNMENT_NAME")

# retrieve a list of submissions
submissions = this_assignment.list_submissions()

# loop over submissions
for submission in submissions:
  ...

Now, we can fill in the body of the loop. Inside the loop, we need to do the following.
(a) Get the contents of tests.txt and homework.py
(b) Use the contents of tests.txt to figure out which tests correspond to which functions
(c) Figure out where each function is defined in homework.py, so we can place comments precisely
(d) Place a comment on each function definition that has at least one test

Let's start with (a). This is easy. We can access the submission's files and then search by name using the codePost API's handy by_name function.

# get the files we're interested in
test_file = submission.files.by_name(name="tests.txt")
student_code = submission.files.by_name(name="homework.py")

Next, (b). To parse the contents of tests.txt, we'll write a function. We'll do this in the next section. For now, assume the existence of a function called parse_test_output that returns a dictionary where each key is function name and each value is a string corresponding to the test output.

Finally, (c). We'll write a function called find_function_definition in the next section to do this. We'll pass this function a function name and the student's code, and we'll get back an object that specifies where the function is defined in the student's code.

Armed with these two functions, we can move on to (d). To place a codePost comment, we need to know the following properties:

file: homework.py
text: we'll concatenate the results of any tests corresponding to the function
pointDelta: we'll use 0, but you could choose to take points off for failed tests
rubricComment: we'll set this field to None, but you could use rubric comments to better track which tests students are failing
startChar, endChar, startLine, endLine: provided to us by find_function_definition

We're ready to go!

# get the files we're interested in
test_file = submission.files.by_name(name="tests.txt")
student_code = submission.files.by_name(name="homework.py")

# parse test output
tests_by_function = parse_test_output(test_file._data['code'])

# loop over functions which have at least one test
for function_name in tests_by_function.keys():
	
  # parse student code to figure out where to place comment
  where_to_place = fund_function_definition(function_name, student_code.code)
  
  # define comment using codePost comment schema
  comment = {
  	'file': student_code._get_id()
    'text': ("\n").join(tests_by_function[function_name]),
    'pointDelta': 0,
    'rubricComment': None,
    'startChar': where_to_place['startChar'],
    'endChar': where_to_place['endChar'],    
		'startLine': where_to_place['startLine'],
		'endLine': where_to_place['endLine'],    
  }
  
  codepost.comment.create(**comment)

There's one problem here. codePost renders comment text using Markdown. That means we have to escape any brackets if we want them to show up properly in codePost comments.

# join all tests corresponding to the funciton called function_name
test_text = ("  \n").join(tests_by_function[function_name])

# escape brackets for Markdown
test_text_escaped = test_text.replace('[', '`[').replace(']', ']`')

# define comment using codePost comment schem 
comment = {
  'text': test_text_escaped,
  'file': student_code.id,
  'pointDelta': 0,
  'rubricComment': None,
  **where_to_place, # save space by deconstructing where_to_place
}

We should add some error handling here, but other than that, all we have left to do is define parse_test_output and find_function_definition!

Let's do that now.

3. Write helper functions

First, let's write parse_test_output. To do this, we need to loop over the contents of tests.txt, and do the following:

If we find the start of a test, capture all 6 lines of it. We can assume all tests (and tests only) contain a string that looks like Test \d.
Figure out the function the test corresponds to (indicated in the second line of test output)
Add the test to the function's test list

The following code achieves that.

import re

def parse_test_output(text):
  # indicates start of test output
  regexp = re.compile(r'Test \d')
  LINES_IN_TEST = 6

  # gather together all test output objects
  tests = {}
  linesSearched = 0
  lines_of_output = text.split('\n')

  while linesSearched < len(lines_of_output):
    line = lines_of_output[linesSearched]

    # does this line correspond to the start of a test?
    if regexp.search(line):

      # Capture the contents of this test
      test = "  \n".join(lines_of_output[linesSearched:linesSearched+LINES_IN_TEST])

      # what function does this test correspond to?
      # function specified on line 1 of the test output
      functionName = re.sub(r'Function: ', '', lines_of_output[linesSearched+1])

      # store in value array with function name as key
      if functionName in tests:
        tests[functionName].append(test)
      else:
        tests[functionName] = [test]

      # we just captured the whole test
      linesSearched += LINES_IN_TEST
    else:
      # continue scanning
      linesSearched += 1

  return tests

Writing find_function_definition is even easier. All we need to do there is:

search the student's code for the input function's definition
if we find it, report the line and characters where the definition exists in the code string

def find_function_definition(function_name, code):
  lines = code.split('\n')
 
	# this corresponds to a Python function definition. See below for other langs
  stringToFind = 'def %s' % (function_name)

  for index, line in enumerate(lines):
    startChar = line.find(stringToFind)

    # if we found the string corresponding to the right function definition
    if startChar > -1:
      return {
        'startChar': startChar,
        'endChar' : startChar + len(stringToFind),
        'startLine': index,
        'endLine': index,
      }

  # couldn't find the function definition
  return None

4. Run it!

That's all there is to it! We can now run our script, and watch as the codePost API places comments for us. Here's the full code (excluding the helper functions, which are tucked away in a file called helpers.py)

#!/usr/local/bin/python3

# Imports
import codepost
from helpers import parse_test_output, find_function_definition

# Set some required variables
course_name = '<COURSE NAME>'
course_period = '<COURSE PERIOD>'
assignment_name = '<ASSIGNMENT NAME>'

test_output_file = 'tests.txt'
student_code_file = 'homework.py'

codepost.configure_api_key(api_key='<YOUR API KEY>')

###################################################################################################

# try to find course
course_list = codepost.course.list_available(name=course_name, period=course_period)
this_course = course_list[0]
this_assignment = this_course.assignments.by_name(name=assignment_name)

# retrieve list of assignment's submissions
submissions = this_assignment.list_submissions()

# loop through submissions
for submission in submissions:

    # get files corresponding to this submission
    test_file = submission.files.by_name(name=test_output_file)
    student_code = submission.files.by_name(name=student_code_file)

    if (test_file is not None) and (student_code is not None):

        # the 'code' of the test output file is the test output
        tests_by_function = parse_test_output(test_file.code)

        # loop over the functions which were tested at least once
        for function_name in tests_by_function.keys():

            # use helper function to figure out where the function corresponding to function_name
            # was defined
            where_to_place = find_function_definition(function_name, student_code.code)

            if where_to_place is not None:

                test_text = ("  \n").join(tests_by_function[function_name])
                test_text_escaped = test_text.replace('[', '`[').replace(']', ']`')

                # construct codePost comment
                comment = {
                  'text': test_text_escaped,
                  'file': student_code.id,
                  'pointDelta': 0,
                  'rubricComment': None,
                  **where_to_place,
                }

                # post the comment to codePost
                codepost.comment.create(**comment)

There are plenty of cool ways to extend this script. Here are some ideas:

Automatically deduct points based on the tests a student fails. You can do this by dynamically setting the pointDelta of a comment
Only show the full output of tests which a student fails
Use rubric comments to track which tests students are failing. To do this, you can create a rubric comment corresponding to each test. Then, you can place one comment per failed test (instead of concatenating tests into one comment per function as we did here). In those comments, set the rubricComment field so your comment applies the appropriate rubric comment. Then you'll get all the benefits of using a rubric, including tracking and the ability to batch alter point values after comments have been applied.

A. Extensions

Different language

If your student code is written in a language other than Python, all you need to do is tell parse_test_output what to look for in a function definition. In the version we wrote, this would require replacing stringToFind = 'def %s' % (function_name) with a string specifying how functions are defined in the language of your assignment. You could also use a regular expression to capture more complicated function definitions (e.g. those in statically typed languages, like Java).

Multiple files

If your students' submission involve more than one code file, you'll need to locate the right file, as well as the right location within that file, to place a test summary comment. The following loop body would work

# loop over submissions
for submission in submissions:

  # get test output
	test_file = submission.files.by_name(name=test_output_file)

  # parse test output
  tests_by_function = parse_test_output(test_file.code)

  # loop over functions which have at least one test
  for function_name in tests_by_function.keys():

    # parse student code files
    for file in submission.files:
    	where_to_place = find_function_definition(function_name, file.code)
      
      if where_to_place is not None:        
          # define comment using codePost comment schema
          comment = {
            'file': file.id
            'text': ("\n\n").join(tests_by_function[function_name]),
            'pointDelta': 0,
            'rubricComment': None,
             **where_to_place
          }
          
          codepost.comment.create(**comment)
					continue