Step 48: Reflect on the Autograder

(Note: This is Step 48, which means I have hit the end of a 12 post cycle and will not be writing next week.)

As we near the end of the semester – and the beginning of the summer – I’ve started thinking about the changes that I need to make to the autograder. The autograder is the first complete piece of software that has been battle-tested; although I have written substantial sections of code before for research, that has only been used by ten or so people. In contrast, the autograder has been in heavy use for most of the semester. My 35 students have generated over 94,000 test results (!) from over 3,600 submissions. I’m somewhat surprised that the system has held up at all, given that I wrote most of it over three weeks.

With no classes over the summer, it’s time to think about updating the code. There are a number of technical challenges that I won’t describe here; I will instead talk about some uses of the autograder that I didn’t expect.

First, one thing I didn’t expect is that I would use the autograder less as the semester progress. It makes sense in retrospect, since I focused more on program design and less on implementation. When students started designing their own classes, it became impossible to autograde their work, even with the introspect offered by Python. This coincides with the shift towards assignments that allow more creativity, such as creating images. These assignments were difficult to grade in general, but that’s a topic for another time.

Aside from the creativity of the students, however, I did find uses for the autograder that are not just about testcases. The students’ most recent assignment is writing an AI for the dice game Pig, when the students only “passed” if they beat a random player more than half the time over 10,000 games. I suppose this is similar to assignments where students must write efficient data structures. A different use for the autograder is to include checks for coding style; I haven’t fully implemented this yet, but I do ensure that student code can be imported cleanly as a module. I intend to use pylint in the backend to provide style checking in the future.

Entirely separately, although the autograder was designed to allow student-written test cases – which does work – I only used that feature a small handful of times. This is related to the creative assignments above: I feel that there is only a short period of time when students were comfortable enough with code to write their own test cases, and when I want students to start thinking about code structure such that simple testing is not possible. This is of course a larger issue than the autograder – I will likely need to tweak my assignments to require both an autograded component as well as a creative component.

Mostly though, except for a couple of odds and ends with additional information that I need to capture – due dates for assignments, for example – I thought the autograder worked well. The general architecture is solid enough that I’ve been thinking about “hiring” students to redesign the frontend to make it more usable… which would be a cool way for students to continue their computer science education while bringing things full circle.

Step 48: Reflect on the Autograder

One thought on “Step 48: Reflect on the Autograder

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s