Step 8: Prepare for a Topics in AI Course (Part 1)

I have known that I will be teaching a Topics in AI course since March. It’s standard for new faculty, both at liberal arts colleges and at larger universities, to teach a course on their research their first semester. For me, however, that would have been entirely insensitive to the new department and the lack of computer science students, especially given the niched area of cognitive architectures. Although Oxy currently has an AI course, it is offered by the cognitive science department and has no direct computer science requirements (although a Computational Approaches to Cognition course is one of several possible prerequisites). The course therefore struggles with teaching material that students both with and without computer science experience could understand. So a slightly more advanced AI class, taught from a computer science perspective, is a good fit for both my first semester and for the college.

Since I’ve spent some time thinking about this course and what should be in it, and since I’m keeping a blog about my life as a computer science professor, I thought I would write about my plans on the course thus far. It’s likely that this would change drastically once the semester starts, but comparing the end result with these initial plans would also be informative. This post covers the general outline of the course, and coming posts will talk more about each topic in detail.

What are the goals of the course?

As with all AI courses, an obvious goal of this course is for students to gain familiarity with the algorithms used in AI. One thing I’ve always felt that AI courses fail at, however, is teaching students how to think about AI problems. Students may come out knowing how an algorithm works, but they lack the ability to represent real problems in a form upon which the algorithms could be applied. This process is non-trivial, and often requires knowledge of the problem domain that lie outside computer science. For example, when I taught the AI course at Michigan, I created a Bayesian network that calculated the probability that a student would receive an A for a course. After some questions examining how different known facts affected the prediction, I then asked students for a factor that they thought were missing, and to suggest the right causal relations and probabilities such that the new factor can be incorporated in to the model. Despite the simplicity of the question – there were basically no wrong answers – I was surprised to learn that students disliked the question. I was not as surprised to find that although most of the homework was reused this semester, this particular question was taken out.

The moral of the story for me is that while students may understand the technicalities of calculating probabilities and likelihoods, they do not understand the work that goes into creating the representation, which often determines whether the chosen method would succeed. This is something that I value as much as the ability to code up a complicated algorithm, and is something I want students to start thinking about from the first AI technique they study. Along similar lines, since this is an upper-level computer science course, I want students to also know where the algorithm fails and having a sense of the open research questions that have yet to be answered.

(Of course, knowing when to apply an algorithm and its failure cases is not a skill unique to AI, and I plan on making this a part of the main computer science sequence as well.)

Finally, since a number of students will be coming into the course with a cognitive science background, I will likely how the techniques relate to cognitive science, either as a tool for modeling, or as differences from how people approach the same problem.

Who are the students?

The course is listed in the catalog as requiring both Introduction to Cognitive Science and Fundamentals of Computer Science (the CS1 course). Notably, the existing AI course is not a prerequisite, meaning I cannot rely on students knowing what search means in an AI context. These students are likely to be comfortable with Java, although only optionally having a basic understand of data structures and algorithms. I’m not as much worried about their ability to implement complicated algorithms as I am with their ability to write non-trivial programs. Students also may or may not have some math background for computer science (eg. logic, set theory, probability, etc.), but this is less of a concern. I am hoping that making students work in pairs or groups will help reduce some of these disparities, but whether it works will depend on the exact experience of the students I get.

What topics are going to be covered?

I plan on covering at least four topics, three of which I’ve decided on:

• Reinforcement learning
• Bayesian networks
• Cognitive architectures

The first two topics are often in standard AI courses, but do not make an appearance in Oxy’s current AI course. When/If I reorganize Oxy’s AI course, I may or may not include these topics. I have yet to determine the last topic for the course, but it will likely be driven by student interest.

These topics were chosen partially because I have experience teaching them – there were all part of the AI course I taught – and partially because they do not rely too heavily on the ideas of search (that is, missing the basic AI course should not hinder students too much). These topics were also chosen because of their relation to cognitive science, especially for reinforcement learning and cognitive architectures. Whether I actually emphasize these connections depends on whether students are interested in them.

How is the course organized?

With four topics, each topic will receiving roughly a month of class time. Given the discrete topics, my plan is to have the roughly the same approach for each one. The first two weeks will be scaffolded exercises, for students to build up to the point of understanding the basics of the field and being able to implement the relevant algorithms. The last two/three weeks will focused on a paired/grouped project, with the goal of either applying the technique to something that interests the students or to explore how the results of the technique vary on more difficult domains. For each group and each topic, students will have to write a short report on what they learned, as well as a short in-class presentation (probably require each group to present at least once over the semester). At the end, there will also be a chat with each student individually, a conversation during which their understanding will be much more apparent than through an exam. My main concern currently is the amount of work this would require of students, particularly if the students are not yet comfortable structuring larger programs. I believe this work load is doable, however, provided I keep an eye on student performance.

In the coming weeks, I will expand on this general plan and sketch out in more detail what I’m planning for each topic, including my ideas for the undecided one.

Step 7: Collect the Right Data

How do you measure the success of a computer science program?

I am this because I have been thinking about data collection, about what longitudinal statistics I would need to measure the health and growth of a department. Is surprised to learn several months ago that Michigan does not have comprehensive records of their students. They may have the number of students in computer science but year, but that’s already the lowest granularity. There has been talk of starting a program to track what courses students take and why they continue or not continue, which I think I’d necessary to any more sophisticated diagnosis and improvement.

In the introductory class I lectured for this past semester, for example, we not only kept track of students’ anticipated next CS course, but also their attitudes about computer science. For example, here are some questions that we asked them at both the beginning and the end of the semester:

• I believe computer science can make a positive impact on the world.
• I believe computer science will help people.
• After graduation, there are equal opportunities for a career in computer science for males and females alike.
• I believe that knowledge about computer science will be more important in the future than it is now.
• I believe that having a career in computer science is as compatible as any other career with having a quality family life.
• My opinion of computer science is representative of those of my gender.
• I find computer science intimidating.
• I can see myself in a computing-related career in the future.
• I can see myself as a computer scientist in the future.
• Someone who takes future CS classes (e.g. 280) will be a coder for the rest of his or her life.
• I believe that other students in computer science will be welcoming to me.

I think these are good questions, especially if they are asked every semester. I wonder, though, if there are questions which would allow comparison between colleges – I don’t know of any effort to have something standardized questionnaire that would allow that kind of analysis. Short of that, perhaps the questions from national surveys could be adopted, which would allow comparison with national trends.

Other obvious data to keep include course evaluations, the jobs that graduates get (perhaps even over multiple years after graduation), their course trajectory and grades, etc. Fit the last one, I wonder if it’s possible to build some kind of model that uses future course grades to induce the quality of a previous course.

But there are also other statistics to keep, which do not have to do with the student population. One metric if diversity that occurred to me, for example, is the race and gender of invited speakers. While not as direct as measuring the diversity of the students, it is seems like a plausible symptom of a department that is not thinking sufficiently about diversity, much like looking at the demographics of professors.

Along similar lines, a non-student statistic to track is the type of assignment given in class. Is the assignment of interest only to computer scientists, or does it involve some real world use case? Is it text based or is it a media computing project, or is it purely algorithmic with no output? It would be interesting to then correlate these projects to student performance or opinion, to get insight on what projects excite students.

Step 6: Trace the Computer Science Culture

Why does computer science have a culture? Do other disciplines have a culture?

It’s hard to explain what I mean by “culture”. Part of it is a stereotype, how others see programmers as people who code alone all day in a dark basement. By that definition, I might say that theater majors are stereotyped as being overly sentimental and melodramatic. But it’s also more than that. Take hackathons, which to me has always had a vibe of narcissism, of wanting to impress others either by your own technical ability or by being able to stay awake the longest, and of forced bonding. I’m sure many – maybe even most – hackathons are not like that, but among the computer scientist/programmer population there does exist traces of these attitudes.

What’s surprising to me is that computer science seems to be unique in that regard. Mathathons or chemathons don’t seem to exist outside of charity events, and I can’t see physicists trying to out-do each other as a community (although I’m sure many individual mathematicians/chemists/physicists are insufferable know-it-alls, as are many computer scientists). I don’t know if programmers are more worshipful of technical ability, or if they are more prone to letting their ability to go their head. I can’t tell if the stated desire to “make the world a better place through technology” is genuine, if people who say it are merely parroting people in Silicon Valley (although, why do those people say it?) or if they truly, egomaniacally believe it’s their destiny.

Although I’m starting a department, I don’t think I can get away with designing a culture from scratch. Oxy is not a closed system – the students I interact with are already aware of the culture of computer science, for better or for worse. One of perceptions is that computer science is about technical cleverness, otherwise known as the ability to “hack”. I have nothing against technical cleverness, and I value it as much as anyone else, but I think computer science has a culture where cleverness is valued to the exclusion of other factors, like applicability or whether the cleverness was a good idea in the first place.

This focus on esoteric knowledge for its own sake is hurting the diversity of computer science. I can get into a discussion about vi and emacs, but the differences are irrelevant for the newcoming programmer – but the newcomer doesn’t know that it’s irrelevant. And the result is they incorrectly think that they are unprepared or don’t have the ability, or that computer science is only about these kind of minutiae. Nowhere reflected in this mindset are the things I value more in computer science: learning tidbits of other disciplines under the guise of automating their work, and having a framework to make conceptual distinctions that I previously couldn’t.

I’m okay with computer science having a culture, and I’m okay with that culture being about cleverness, but I want that culture to also be about computer science as a modeling tool, about the philosophical questions that computer science poses including, occasionally, whether computers are the right tools for the job at all. And if I cannot reduce the worship of cleverness, I’ll just have to work harder to emphasize the other parts.

Step 5: Ensure Technical Excellence

Last time I talked about how computer science relates to the liberal arts; this post is about the other side, about my thoughts on computer science as technical, almost vocational, field.

The first question is one of balance. The more I think about an ideal computer science curriculum, the more I see a tension between the demands of the liberal arts, of computer science as field, and of the billion-dollar industry that is technology. A liberal arts bent demands a focus on new modes of thinking (eg. map versus territory); a computer science bent demands a focus on algorithmic efficiency (eg. data structures and big-O); and a vocation bent demands a focus on the skills required to work in a software company (eg. version control and agile development). The precise amount of time I should be spending on each is not clear to me yet, although I suspect going with my natural inclination will do a decent job.

More than the software engineering/software developer skills, I’ve been thinking about the difference in technical ability between the graduates of liberal arts colleges (“college” graduates) and the gradates of state universities (“university” graduates). I have talked to a number of my friends about the liberal arts college experiences, and one trend I noticed is that many of them feel like their undergraduate courses were not as rigorous or technical as the courses they took (or saw others take) in grad school.

I see two possibilities. It’s possible that the students’ feelings reflect reality, that college graduates are in fact technically weaker (by whatever metric). The question would then be why this difference exists. Some ideas:

• University students take more computer classes. While it’s possible this is a curricular difference, I suspect that the curricula require the same number of credit hours. Instead, it’s more likely that university students do not feel the need to diversify beyond their computer science classes, while college students do due to cultural influences.
• University computer science classes are more challenging, and use more “technical” languages. There is some truth in both of these components. Since college computer classes are offered less frequently and often include non-majors, the difficulty may be reduced to accommodate all students. I’ve also noticed a trend for college classes to prefer Java, while university classes often use C or C++. The latter two languages require the programmer to do more bookkeeping, so students may come out with a better grasp of the underlying principles.

Alternately, students’ feelings are exactly that, and there is in fact no difference in technical ability between liberal arts graduates and state university graduates.

• An obvious possibility is that these feelings of inadequacy is merely the imposter syndrome, and that in fact all graduate students feel insufficiently prepared. A simple way to test this would be to talk to university graduates.
• There is a sample size effect. Since there are many more university graduates than college graduates, there are also more students of extreme ability, against whom students compare themselves. I know for a fact that there are also university computer science students who have a surprisingly terrible understanding of the subject.

Even if it’s true that graduates of liberal arts colleges are weaker computer scientists than their university counterparts, I refuse to believe that this is a necessary outcome. The first step to addressing this is to understand why this difference (or this illusion of difference) exists, and I intend to keep question in mind as I transition from a public university to a liberal arts college.