In the game industry, it's traditional to do a post-mortem at the end of a project: everyone on the development team sits down and tries to identify what went right and what went wrong, as a way of avoiding past mistakes on future projects. I wanted to bring that same spirit to my classes, particularly the one that I'll be teaching again immediately after winter break.
Now, in both cases, brutal honesty is hard to come by. If you're working on a development team, your desire to be honest for the good of the company is tempered by your desire to not tell your boss why he sucks. Likewise, it's hard for students to tell a professor why his class sucks if he hasn't yet submitted final grades. So that much, I'm used to.
That said, there's a self-selection bias that exists in a classroom but not in industry. That is, at a game company everyone on the team shows up for the post-mortem, because it's part of their job and they're getting paid for it, so the people who got burned on the project will be there as advocates for change. After a class is over, the students who hated the experience are going to leave as soon as possible; they don't want to stick around for a post-mortem. The only students who remain will be the ones who were already loving the class and don't want it to end, so any comments will be skewed very positive. Meanwhile, the students who struggled and had real issues won't be there to let me know, so I'm likely to repeat my mistakes by catering to a certain breed of student while possibly alienating others. (Don't get me wrong. The students who participated made some great suggestions that I'm going to implement next time around; I just want more information from a wider range of perspectives, because that tends to lead to more surprises.)
I suppose that's what the student course evaluations are for, but I don't really trust those either. Many students don't take them seriously and will comment more on the professor's hair style than teaching ability; also, it doesn't give me the ability to ask targeted questions ("what do you think about Homework #4?") so the comments may be too general. Oh, and I had my evaluations done about two-thirds of the way through the course, so anything that happened in the last third won't be represented on the evals.
I was thinking of perhaps using class time to conduct a post-mortem, as a way of forcing the issue, but then I'd have to remove something else -- so I'd be cheating my current students out of valuable class time, in order to benefit the students next quarter. Not really fair.
Any other ideas of how I might get honest feedback from all my students -- both the ones that love the course, AND the ones that hate it?