Step 71: Manage Crowdsourced Test Cases

One of the main reasons I wrote my own autograder is I wanted students to be able to test their code on other people’s test cases. This is my second semester teaching CS1, and although in both semesters I have used this feature, I have not found it as flexible as I would like. There is only a small window when students are sufficiently comfortable with the concepts – especially functions, but also loops – where they can write good test cases, but not may still struggle with writing the code. I suppose crowdsourced test cases are useful beyond that point, but that is also where CS1 starts becoming more creative, meaning I start reducing the use of the autograder. Together with the schedule of the course (weekly assignments), I really only get to use the crowdsourcing feature once.

This semester, the crowdsourcing was used on a Connect 4 lab. Students had to write a function that determined where a token would go in a column (lowest_empty_row), a function that asked for and verified user input (get_input), and a function that checked for the winning condition (has_connect_four). I motivated test-driven development at the beginning, then asked students to write and submit test cases for has_connect_four, using provided helper functions to easily create a board. The autograder is set so that during the lab, their own code would only run against their own test case. (Of course, they could also do this directly on their computer.) After the lab is over, I would then toggle the autograder so everyone’s test cases are used, then re-run all submissions on all the test cases. These results – both the correctness of the code, and the discrimination of the test cases – then become the students’ grades.

I thought the lab was mostly successful. The success is from students having “better” test cases than I did – there were a number of off-by-one errors that I didn’t catch, and some near-hits that I didn’t look too closely into. The failure is that, although students are rewarded/punished for the quality of the test case, I do not have any deliberate process for students to reflect on and improve their own test cases. The main constraint is time; although it’s a three-hour lab, students can only afford to spend an hour (more realistically, 30-45 minutes) on the test cases, so they still have time to finish the functions. Even if students had more time, however, I am not sure how I would guide this reflection. One idea would be for them to see the test cases that others (including themselves) failed, and see if they can identify the common programming error that caused it. I am not convinced that this is feasible, nor that it would transfer to awareness of likely mistakes in the future.

I have mentioned this test case crowdsourcing idea to several people, including the folks at zyBooks (which I use for my course), and interestingly I have had minor pushback on its utility. In addition to how students may not write good enough test cases – which this lab in particular offers a counter example – the argument was almost that testing (aka. quality assurance) is a whole different skill set, and therefore it does not play as big a role in CS1. To me, the fact that quality assurance exists as a separate department is evidence for the opposite – that writing good test cases is so hard we need experts to do it, which is all the more reason students should start learning early. That said, my inability to fully take advantage of the autograder, even in the second iteration of the course, suggests a mismatch between what I want to do and how I organize the course.

I want to end on some slightly technical thoughts on how test cases should be graded. For the Connect 4 lab, I graded not by test case but by test suite – that is, how many students failed any of the test cases that a student submitted. In particular, the student who fails the most people gets full marks, then everyone else gets a proportionally lower score based on how many people they did not fail. This means that everyone’s code was perfect, students would not be penalized for not finding bugs with their test cases.

While this gets the rough correlation correct, it doesn’t contribute to the goal of making students reflect on how to write good tests. I have a vague idea of grading individual test cases based on how many other students failed that test. The problem I don’t have a good way of combining this information; simply summing these numbers would favor quantity over quality of tests, while discounting students who fail multiple would revert the grading to what I do now.

I still believe in the idea of crowdsourced testcases. All the issues I brought up here are logistical and not technological, and I would love to hear success stories from others of how they teach test-writing skills.

Step 71: Manage Crowdsourced Test Cases

Step 70: Apply for a Tenure Line

I apologize for the very late post; some search-related duties kept me busy.

Now that the job posting is public (please advertise it widely!), I want to describe the administrative process we went through to get the approval to post the ad. This is a part of faculty life that I’ve never thought about before, nor have I read about it anywhere, so I thought it would be useful to describe. I’m not sure how much of this applies outside of Oxy, and other institutions may have completely different terminology and procedures.

To start, the college often refers to funding a faculty as a “tenure line” – that is, from funding someone from assistant professor level, through their promotions over the years, until they retire. I like to think of tenure lines as belonging to a department: for example, if someone is retiring, the department may try to hire a new faculty to replace that person. This is not entirely correct, since departments can grow or shrink, and sometimes entirely new departments are created, but I imagine that the total number of tenure lines across the college only changes slowly.

The application process starts with a position request completed by the department. The request is half about the need for a new faculty member, and half about how such a hire would impact the rest of the college. I found the latter particularly interesting, since these considerations presumably only exist at small liberal arts colleges. Beyond the need to fill a gap or to remain competitive with peer institutions, the other departments and/or campus entities should also be benefiting from the new faculty. Reading back through the request for this post, I get the feeling that this is the heart of how the college grows, by the careful consideration of whether the new faculty would contribute to the college’s strategic plan, and how it generally enhances the quality of the education here.

The position requests usually takes several weeks to write, although for us it took less time due to great collaboration within the department. The completed request is then sent to our Academic Planning Committee (APC), together with signatures and at least one support letter external to the department. APC, a committee composed of the Dean and faculty from a cross section of disciplines, then decide whether to approve the request. Since it’s mostly tenured faculty who serve on APC – while it’s possible for untenured faculty to be appointed by the Dean if it was felt that they would offer a missing perspective, such appointments are unlikely – I am not privy to the discussion that occurs. All we get, via a response to the department chair, are comments on the position request, and whether an acceptance is possible after revisions.

Assuming the revisions are completed, the final obstacle before posting the ad is the detailed search plan. As the name suggests, this document covers the details of how the search will be run – from the exact ad to be posted, to how applications will be processed, to the timeline for the various interviews. The most interesting part of writing this document, at least for me, was the discussion on how to ensure we are reaching out to underrepresented applicants; in particular, borrowing an idea from a search in Philosophy, we are reaching out to diversity-focused organizations for help. This and other such strategies seemed to satisfy our Affirmative Action committee, who worked with the Dean for the final approval of our search.

Which is where we are now: job ads posted and waiting for applicants. I will likely write about the search process again when I am allowed to, since it’s fascinating to be on the other side of the job market. As I said at the beginning, please do share the ad with anyone you think would be interested in the position.

Step 70: Apply for a Tenure Line

Step 72.0: Volunteer at GHC

I’ve written before about missing the registration for Grace Hopper, which is why I am going alone instead of with some students. Seizing the opportunity, however, I registered to be a volunteer note-taker and blogger, which will make me go to interesting sessions and meet new people.

This post mostly exists to populate the currently-empty #GHC16 tag, but I will post more under there as GHC gets closer – and of course when I’m actually in Houston!

Step 72.0: Volunteer at GHC

Step 69: Define My Role in Addressing Diversity

As a STEM professor, what do you see your role is in addressing the lack of diversity in STEM?

My colleague tweeted this question from her student this past week. I realized I didn’t have a good answer. Mostly when I think about diversity, I think about specific things I have done, but I don’t think of them as my role, since I only came up with those actions ad-hoc, while the bigger question is about my overall approach to and vision of diversity. The question also asks specifically about my role as a professor in a STEM field, which again is not something I have considered.

Part of my hesitation in giving a clear answer to the question is that I do things that help diversity computer science, but diversity was not the main goal. For example, I’ve written about the lab where students must confront their implicit assumptions about how names, ages, genders, locations, phone numbers, etc. should be represented in a program. To use the obvious example, if they used a Boolean is_male to represent gender, then their program necessarily will not be able to accommodate alternate gender identities. While this is an exercise that brings issues of diversity and bias to the foreground – and I wrote the previous blog post that way – for me it was more important that students understand that they are making implicit decisions as they write code. My goal was to make those decisions explicit, even given the possibility of students deliberately choose to only represent binary genders.

There are other examples as well: choosing to work with a community partner that serves urban LA was a semi-deliberate choice, and both expands my and my students’ horizons and serves as a high-impact practice to engage underrepresented students, but my thoughts were more focused on how designing for a non-tech-savvy user base would be a good challenge. Hosting Hours of Code may make computer science more accessible to women and students of color, but again most of my thoughts are on getting the general population excited about technology and growing the computer science program.

Reflecting on these examples, what I would argue is that diversity is not a terminal goal for me – that is, I don’t think I am pursuing diversity for the sake of diversity. What I think I am pursuing is helping my students be better computer scientists, better intellectuals, and better… people. And I think just from that goal there are many places where diversity matters. Some of my students are students of color and first-generation students, and the support and mentorship they might need is different from my other students whose parents have advanced degrees. My students who are studying computer science should know that the Silicon Valley tech startup bubble is just a very small, very rich, and very homogeneous part of the world, and writing programs that only target that population is neglecting the “unexotic underclass”. And for all potential future students, I want to make sure I am not restricted by the few students who have are lucky to have previous experience, and that others can see how the discipline can contribute to whatever other field they are exploring, including social justice and inclusion if they so choose.

As a STEM professor, I have the privilege of influencing how my students develop. My role in addressing the lack of diversity is to ensure that students have the resources to succeed. Whether this means support, mentoring, awareness, exposure, or opportunity, I hope to see these students become thoughtful individuals who themselves are concerned with inclusion.

Step 69: Define My Role in Addressing Diversity

Step 68: Place Advanced CS Students

I want to talk about student placement a little bit.

The most obvious placement is if students have taken AP Computer Science. APCS has changed since I took it in the 2004-2005 school year. I only recently learned that they discontinued APCS AB in 2009 (according to Wikipedia, due to lack of student interest). What has continued is APCS A, which includes variables, branches, and classes, but not data structures or recursion. Without deliberately designing for APCS, our two-course core sequence breaks cleanly at OOP, with inheritance straddling the line. Our current policy is that doing well on the APCS exam (a 4 or a 5) allows students to take the second course.

Increasingly, however, I’m finding students who has computer science experience but who did not take APCS. For example, I currently have a student whose parents work in the tech industry, and so have spent the last year learning Java and Python by following books and online tutorials. Two different students are enrolled in Edx’s version of Harvard’s CS50, and one of them is asking if we will give credit for it. (It turns out Oxy does not have a “no online courses” policy, which made my life difficult.) CS50, in particular, uses C, which means students get a decent grounding in loops and arrays, but with no experience with classes and OOP.

This is a problem, because now the students can’t go directly to the second course, but half of the first course also fails to provide a challenge. The discussions I’ve read over from other schools advocate for erring on the side of letting these students catch up in more advanced courses. We have yet to offer the second course, at least after we restructured the curriculum last year, so it may be possible to accommodate these students. Even if the syllabus doesn’t change, I don’t feel classes are too difficult a concept to grasp, so the extra work may not be overwhelming.

Since we haven’t formalized a policy, the de facto result is that these students are in my CS1 course. I’m lucky that so far they have not directly affected the learning atmosphere in any serious way – for example, by asking questions that show they clearly know more, thereby intimidating other students. In fact, I deliberately pair them up with weaker students in lab, which provides an opportunity for peer learning. Groups with hetergeneous skill levels is still possible without the advanced students, of course, but the mutual recognition of their status between the student and me has led to valuable conversations about pedagogy.

The last thing I will say is that I have been watching how these students react to the material. While they are already familiar with loops, their computational thinking/problem solving speed is surprisingly not much faster than the best students who do not have any experience. Most of these students learned a language other than Python, but I don’t think that completely explains the lack of a gap. I don’t have a good alternate hypothesis, however, outside of the general notion that they have yet to internalize the structural and procedural abstractions of a seasoned programmer, which only comes with experience. For now, the short in class brain-teasers still make them think, and I take it as a sign that it’s at least not a complete waste of their time.

Looking ahead, I expect the number of self-taught computer science students will increase, as MOOCs and other online resources gain visibility, and I foresee formally deciding on the placement criteria soon. My current ideal is to get administrative permission for some students to take the course for half the usual credits, which would allow more granular placement. One can dream.

Step 68: Place Advanced CS Students

Step 67: Learn from Summer Research Students

In addition to mentoring a research student, this summer I also volunteered to be an “area coordinator”. The idea is that I serve as a second mentor for a group of about ten students roughly in the same area of research (psychology or cognitive science), to be available for students who may not be part of a lab. In practice, however, most students on campus over the summer know others who are also doing research, so they have social support outside of the area group.

This left me with about an hour a week with these students to fill. The role I decided to take is to introduce students to being a researcher – while the advisor of their specific project focuses on whatever subject they are exploring, I can instead talk about obstacles in doing research. Early on, as students got used to summer research, I talked about finding the right relationship with their advisor. I drew partially from the grad school experiences of my own and of other students, but also from Handelsman et al.’s Entering Mentoring: A Seminar to Train a New Generation of Scientists. The pamphlet was a great resource, as it contained reflection questions for both mentors and mentees. One such question, “Would you like to be able to spend more time with your mentor?”, was one for which different students at Michigan had different preferences. Discussing this question with the students likely led to me asking for semi-weekly check-ins, breaking free of the mold from grad school.

Around week three of I spent one hour talking about the administrative details of research – how to filter through the deluge of papers, how to stay organized with all the findings, as well as how to remain (relatively) stress-free. I had several insights here. First, it wasn’t until that point when I realized students might never have faced this amount of reading before. While a social science course may assign a hundred pages a night, the skills for skimming one long source is different from selecting from hundreds of sources. Strangely, I am not sure where I learned to filter through papers either, and I’ve never asked students to do it as part of a course. In fact, this realization came in useful recently, as I am mentoring a student for their comprehensive project and I had to talk them through this process.

It wasn’t clear at first how useful it would be for me to talk about work-life balance. Summer research has lower expectations and is not as all-consuming as grad school. I think it was ultimately useful to bring it up, though, as I’ve heard from multiple students that they feel under pressure simply from not absorbing the material as quickly as they would like – even if their expectations may not be realistic to begin with. Depending on the student, I have found that they sometimes will push themselves to spend more time on the problem, when taking a break may be a more productive path forward. Again, I am learning how to adjust the reins for each student, and when to reach out and offer assistance.

One topic that I didn’t think would be interesting, but which occupied one meeting, was the career path of an academic. More than half of the students in my area group are considering grad school, so we naturally spent time on the application process and grad student life. What caught their attention instead was my personal path to my current position, and what the next couple years would look like for me. In this case, students had no real need to understand the tenure and promotion system, but simultaneously I am also surprised that I have never explained this topic before.

The final thing I will say about this experience is that, although I tried to spend time on technical writing, I did not have any material that would let me do so. I resorted to advising that students read through The Elements of Style, but that seems inadequate and weak compared to critiquing real writing. From my summer student, I do now have the full history of a piece of writing as it goes through the editing process, but I still lack the pedagogical content knowledge to teach writing to more than one student at a time. Since I eventually intend to teach first-year writing, this is a shortcoming I would like to rectify.

Although I enjoyed being an area coordinator this summer, I am not sure I would do it again. If I were to reprise the role, I would like to define more concrete goals for the students, as opposed to the ad-hoc curriculum I created this summer. The rewards of leading the sessions is not quite worth the time I put in, and given my overall inability to find balance in my research, I should probably spend more effort there instead.

Step 67: Learn from Summer Research Students

Step 66: Embrace the Liberal Arts

I noticed a curious trend in the classes I’m teaching:

  • For Intro to Cog Sci, we are moving to a model of short response papers for every lecture, mostly graded for completion but with randomized occasional feedback, plus three papers, a midterm, and a final. This is compared to last year’s model where we had required forum posts, then longer homeworks, midterms, and a final.
  • For Intro to Computer Science, I started the semester presenting why students should study computer science, but also presented a list of news articles about societal implications of improperly-thought-out technology (for example, not training your AI on people of color). I am also offering extra credit for an edited paper on increasing diversity in Oxy’s computer science program.
  • The Practicum course that I am teaching had their first visit to the community partner this week. The plan was for them to learn about what their project really entails, but as part of the presentation, they also got a lecture on the media bias towards violence and away from poverty.

The trend I’m seeing is the inclusion and emphasis on liberal arts topics – not just “soft” people skills, but drawing in and including more liberal arts topics in science classes. The strange thing is that I have not thought about these changes as deliberately broadening the scope of these classes. Rather, the intention has generally been to get students to engage more deeply. It’s interesting to me that these two goals have the same outcome at all – I can imagine making computer science more engaging by diving ever deeper into technical topics, and in fact I can think of students who probably will enjoy the journey from loops to recursion to higher-order functions to monads.

I do see the appeal of higher and higher levels of abstraction, but its inappropriateness for introductory courses aside, it just feels like the wrong kind of complexity. This seems related to my against the model CS curriculum, and the concreteness makes it easier to analyse. I don’t think it’s just that these connections are application-oriented, as opposed to theory-oriented; it’s explicitly the reaching across disciplines that excite me. I’m not sure I can justify this bias outside of saying that I like generalists.

This post is neither here nor there, and I thought I would have more to say about where my classes are going. I’ll just stop.

Step 66: Embrace the Liberal Arts

Step 65: Advise New Students

As a second year faculty – or rather, no longer as a first-year faculty – one of my new duties is to advise incoming students on their courses. I was initially anxious about this – my vacation travel made me miss a new advisor workshop, and although I later met with one of the workshop leaders, most of what I learned were procedural details of how to deal with edge cases, and very little about how to approach my advisees’ interests.

It turns out that the first advising session doesn’t require you to talk much about the student at all. I spent over six hours on Friday on advising – one hour with my entire new advising group, then 30 minutes of individual advising for each of my 10 advisees. 30 minutes is really not enough time to do anything beyond finding courses that interest them and backups in case those courses are full, and convincing students not to take certain combinations of classes.

Some observations:

  • My most valuable asset as an advisor was knowing how college works. The majority of my advice are things that older students take for granted: don’t take multiple heavy courses, don’t fill your semester with courses from the same area, check for co-requisites, etc. I suspect many students would have done as good a job here.
  • I also found myself being able to catch time conflicts without plotting everything out. I found this ability pedagogically interesting, since my advisees could have figured this out by themselves, except it would have taken them longer. It reminded me of some introductory computer science concepts, which students could slowly apply if given time, but which I could understand at a glance.
  • By the end of this process, I found myself shepherding students into a few classes that I knew had a lot of seats remaining and fulfilled a core requirement. This is after figuring out what classes they do want to take, of course, but after that, it was too exhausting to continue looking for new courses that might fulfill a distribution requirement.
  • Also by the end of this, I had completely lost track of what courses each of my advisees were taking. I tried to write down each student’s plan, but skip a student… and could not remember that I talked about even 30 minutes later.
  • First-years registered by timeslots, and some of my advisees were in the latest groups. Despite my best efforts, I could not find enough backups for students. I suspect I under-estimated how quickly classes would fill – the 10 seats in Intro to Cog Sci, for example, were filled by noon, before half the students had registered. There is also a hidden variable of reputation I’m not taking into account, and which I have no direct way of observing.

Now that the initial wave of advising is over, it’s time to plan out how I want future advising sessions to go. I would like my advisees to think through what they want to get out of college and to define their intermediate-term goal – or more generally, to be deliberate about their path. I heard from a friend who studied at Oberlin Conservatory of Music that she had to say where she saw herself in 5, 10, and 30 years. That sounds a little intense, especially if students don’t know even know what they want to major in, but I understand the intent.

Any advice for a new advisor in that direction would be appreciated.

Step 65: Advise New Students

Step 64: Speculate on the Future Role of Professors

In what is otherwise a mediocre book, Tyler Cowen’s Average is Over contained this passage about the role of professors in a future where online educational material is abundant and AI can precisely measure the progress of students:

In other words, in the longer run, professors will need to become more like motivational coaches and missionaries. The best professors have understood this for years and have been serving that function from the beginning. What’s less well understood is that improvements in AI will make these the remaining roles of what we now call “professors.” The professor, to survive, will have to become a motivator and coach in essence and not just accidentally or in his or her spare time.


It will become increasingly apparent how much of current education is driven by human weakness, namely the inability of most students to simply sit down and try to learn something on their own. It’s a common claim that you can’t replace professors with Nobel-quality YouTube lectures because the professor, and perhaps also the classroom setting, is required to motivate most of the students. Fair enough, but let’s take this seriously. The professor is then a motivator first and foremost. Let’s hire good motivators. Let’s teach our professors how to motivate. Let’s judge them on that basis. Let’s treat professors more like athletics coaches, personal therapists, and preachers, because that is what they will evolve to be.

The book makes some assumptions about the progress of technology, but since that’s not the focus here, I will accept that assumption as is. On the more plausible side, I will accept that we can monitor the performance of students and model how well a student understands each topic. (The latter is actually a hard problem; I have read papers on modeling arithmetic learners, and there is a surprising amount of data-gathering involved. For this post, however, we can assume that this has been done for every existing college course and subject.) For the sake of argument, let’s say we also include cameras (for emotional recognition, eg. frustration) and virtual reality (and haptics!), even if these technologies will take a decade to mature.

What is the role of the professor in this world?

For one, I agree that professors will be motivational, at least in part. I start my introductory computer science course by telling students that they will struggle, but that I will work with them through it. Especially with computer science, which pushes students in new ways, students often need the reassurance that they can figure it out. And the motivational component is not just coursework related – throughout the summer I have been exchanging emails with students about potential majors and courses of study, and placating their worries about not knowing exactly how their college career will work out. With new student advising happening on Friday, this is a role that I am sure will take up more of my time in the near future.

The question for me is whether professors have any other beyond motivation. Elsewhere, I have written about how technologists are often biased against the human element, something I am also guilty of. Cowen suggests that the “classroom setting” may be essential, but I think even more crucial is the physical presence of the professor. As all-encompassing as Facebook is, it can’t subsume physical interaction as a medium, and hugging an old friend has an intangible je ne sais quoi that cannot be reproduced online. Similarly, being in the same room as professor and other students may bring something that merely video chatting (or VR) cannot bring. The nebulousness of this concept makes it hard to talk about, however, and regardless, I can’t dismiss the possibility that physical presence is ultimately only useful as motivation.

Is there a function that professors currently fulfill, then, which goes beyond motivation and which even the advanced technology we’ve assumed can’t replicate? There are two that come to mind.

First, among the technology we’ve assumed, I was tempted to list that the “best method” of teaching (for all subjects) have been discovered. What I realized, however, is that it’s possible this “best method” may require a guide to be there. Even in computer science, I can envision Heisenbugs that require more context than the computer has data for, in which case there must be a human in the loop. For other disciplines, even if there are algorithms to classify “good” writing from the bad, I suspect that the coherence of the essay would still require a human grader (but I would be interested to know that I’m wrong).

The other role that professors play – and one that I have stepped into – is counseling. I have already mentioned talking to students about majors, and while making sure they get through the process may be motivational, helping them find the right major is a different job entirely. Understanding the psychology of a student, then adapting to it and suggesting a way forward, is not within the skill set of a computer (at least, not without assuming much bigger advances in AI).

If there is a thread between these two roles, it’s that the professor must take a wider context into account which, I suppose, is a standard weakness of AI systems. So it’s not surprising that professors would be left with the messy human elements.

…Although, I suspect that this says more about how we envision computers than the job of a professor.

Step 64: Speculate on the Future Role of Professors

Step 63: Deal with New Year Pains

I am traveling to San Francisco for the week for my first (and only) real, go-away vacation of the summer. So I want to just briefly talk about some new academic year pains that I’m dealing with.

  • I will be teaching the intro CS course again, but neglected to reserve seats for incoming first-years. What I should have done when I realized this is to require instructor permission for new registration, but I didn’t do that either until last week. As a result, the course only has 3 out of 32 seats for new students. Given that this course has a relatively high workload and can be psychologically exhausting, I do want to be careful about overloading students new to college.
  • Completely separate from incoming students, there are also more existing students who want to take the course than there are seats. As I mentioned before the previous semester, it’s a good problem to have, but dealing with “waitlists” (which is not a thing at Oxy) and having to turn students down is getting annoying. Let’s just say that, with a little bit of luck, this will hopefully not be a problem I have to deal with next year.
  • The other course I’m (co-)teaching is Intro to Cognitive Science, which I co-taught both semesters last year. I do like the team teaching, especially since the faculty are from different disciplines. However, since we rotate faculty every year and faculty have the freedom to include their specialty interests in the syllabus, we have to rethink the schedule of the course at the end of every summer. This is my first time re-teaching the course, so I enjoyed discussing what worked and what should change, but whether we continue with this model or whether we impose more of a canon is an open question for the department.

I am also teaching a software engineering/practicum “independent study”, but although there is paperwork to get everything set up, the preparation so far has been smooth. I suppose I am also anticipating my first required committee service.

Other than that, full steam ahead to advising next week, then classes the week after!

Step 63: Deal with New Year Pains