Ineffectiveness of Teaching: Exam Results & Solutions

[Moonbear]

A guy from U. Minnesota told us that rather than randomly grouping students for small-group work, they intentionally put together teams consisting of a good student, a mediocre student, and a poor student. My comment was that groups then can't be assigned until well into the term.

In the med classes, we have tried various combinations of assigning groups. We've tried it based on their previous semester scores (since we teach second semester), and we've tried it completely randomly, and we've done it alphabetically, and we've done it so that they have the same group all semester, and we've done it so they switch groups two or three times a semester.

Others who use team-based learning for a variety of subjects in a variety of schools and levels have also looked at the impact of group assignments.

Basically, there are a few things to avoid, but otherwise, it does not matter. You want to try to balance for prior experience in a subject. So, if I'm teaching anatomy and some students in my class have taken an anatomy course before, I would want to break them up among the groups. Doing well in another subject is not necessarily a predictor of doing well in the subject I teach, though the students who do very poorly in their first semester may indeed be weak students who will do poorly second semester too, so we split them up if they all end up assigned to one group. Otherwise, random assignments work fine.

The risk of trying too hard to balance groups by putting a good, middle and poor student into each group is that you set up a situation where the really good student ends up carrying the whole group and the group let's them do all the work because they get things right.

I would not use the top students as examples. First, I don't like putting any student on the spot in the classroom. It makes the environment very uncomfortable. Second, I'm not sure that the class watching a good student breeze through a problem is necessarily helpful for them to learn how to work through the spots where they got stuck. Having a good student demonstrate how to do a problem is not really different from me demonstrating for them. Instead, having a mediocre student do the problems probably works better. They aren't so lost as to not even be able to start or to look like a fool in front of the class, but will have places that they can't just figure out the next step on intuition alone, which presents a teachable moment to explain to the class HOW to figure out what that next step should be. I also would not want to make one student appear to be the "teacher's pet."

Yes, there are students who only associate with other students not doing well in the course, and who think nobody passes and it's all impossible. To let them know that it is not impossible, and because they do need the feedback of how they are progressing compared to the rest of the class, for each exam, I post the exam average. Because some students were getting into that rut this last exam, thinking nobody could have done well, I posted what the high score was too. (This time around, someone DID get a 98%.)

After final grades are announced, I post a grade distribution curve so they can see that a lot of students do get As and Bs, the majority get Cs, and some do get Ds and Fs in spite of everything.

I'm actually going to try something a little different on the student evaluations of the coures this year. I want to see how those evaluations vary by where students sit in the classroom...just roughly divided by front, middle and back of the room. To do this, I'm putting on a question about clarity of visual aids (i.e., text on slides), and then asking them to tell us where they sit in the classroom so if improvement is needed, we know more about where the visibility drops off. But, I think we all suspect, but nobody quantifies, that the students who sit up front do better and pay more attention and generally like classes better than those sitting all the way in the back of the room. I think this is important to identify, because the classroom I teach in has double the seating capacity of my class size (to leave us room to spread out during exams), and that means nobody HAS to sit in the back if they can't clearly see the lecture slides from there. So, if the students in the middle can see, and those in back can't, then that tells us something about their motivation to really pay attention to the lectures. (I'm also doing this because I KNOW the other lecturer in the course uses slides with text that is too small to see all the way in the back of the classroom; I've pointed out to her to be careful about her font selection because of the size of the room, but she hasn't done anything about it, so I want her to get that feedback from the students too.)

Anyway, I really am enjoying this particular discussion, because the general statement that Andy Resnick made about too many instructors wanting to just blame the students is certainly consistent with my observations. I'm glad to see DenverDoc starting up a discussion that recognizes that when students aren't doing well in a course, it means the instructors need to do something different to engage them, get them caught up, explain things better, etc. I cringe when I hear other faculty who have been teaching a long time who get their evaluations and say things like, "What do the students know?" or "Oh, they always complain," or "Don't worry about what the evaluations say, they don't mean anything anyway." Now, granted, I have seen bad evaluation questions that really aren't going to be helpful other than as a means for students to vent their frustrations (one they ask our med students is "Was the material presented in lecture of an appropriate level for first year med students?" Huh? In that case, yeah, how would they know if it wasn't? That's something faculty peers should be addressing, not students). But, when evaluations are consistently bad, it means you've really missed the boat on how to approach that class.

Some of the other faculty in my department have shared with me the critiques on their teaching evaluations, because they are sitting there laughing about it, not taking them seriously and just venting their frustration at having to be evaluated by students. Since we team-teach in that course, I've sat through their lectures and been in the labs with them. You know what? I agree with the students in most cases.

By the way, last year, I knew I was going to be taking over as coordinator for the course I'm running this year. Since the previous coordinator couldn't care less what was happening by then, just happy to no longer have to coordinate it for the future, I got free reign to write up the questions for the course evaluation. I KNEW there were problems with the course, knew students didn't like it, knew there was VAST room for improvement. So, I didn't write the standard questions. I didn't need those to know they were going to rate things as horrible. Instead, I wrote questions that really got to the points that were important...how can we improve? So, instead of them telling me they didn't like lab (attendance already demonstrated that), I asked questions like, "Given the choice, would you prefer an unstructured lab or assignments due in every class?" "Did you usually work with the same group of students for every group activity, or did you change groups each time?" "Would you prefer to choose your own groups, or have them assigned by the instructor?" "If a new textbook were chosen for the course, should it have more or less detail?" "If a new textbook were chosen for the course, should it have more or less text?" "If a new textbook were chosen for the course, should it have more or less pictures?"

I actually got a lot more constructive criticism and good insight from the students from that approach than when we use questions like, "Did you like the book assigned?" The responses were not all what you'd expect either. The students WANTED in-class assignments. They WANTED attendance taken. Overall, this year students are more engaged in the class, they show up, even those who aren't doing well are trying (except maybe 2 or 3 who are still just drifting). Exam scores are up. The students all seem to be having fun and enjoying learning, even as they are struggling.

I also have put up a "suggestion box" on my course website, where students can anonymously give feedback for improvements all during the term. I do tell them that the two things I can't change are to teach less material or to make the exams easier. But, I tell them that if they have suggestions of ways we can help make it easier to learn all that material or do well on the exams, that will certainly be considered.

There is one aspect of the course that I let them decide how it will be done, and those are the team activities they do. On the first day of class, we did an exercise that was not graded for them to get an idea of what the activities would be like. Then, I let the class vote on how many points future activities were worth (I gave them a range to choose, of 1 to 5 points per question...this wasn't a "sky's the limit" question). When I started out the course, I was doing it as all-or-nothing. They weren't thrilled with that, because they lost a lot of points if the got close but not quite. So, based on their feedback, I started giving partial credit for close-but-not-quite answers. This has actually meant the in class assignments got harder, because I can't just give them answer choices that are right or wrong, but have to have a lot more shades of really close but not exactly right for some reason to have a basis for awarding partial credit. They really seem to enjoy that. However, I then got feedback from some groups that they weren't always in agreement on the answers, and half the team would overwhelm the other half and give the wrong answer, which didn't make the students who knew the right answer happy. So, I've added a new component to this "game" which is that they can have a mutiny in the group, but they better be absolutely sure they want to disagree with the group before risking mutiny, because if they choose to disagree with the group and are wrong, they will lose all the points. (I won't give them partial credit on those because I don't want it to be an individual assignment where they all just give their own answers and stop working as a team, so there needs to be a big risk to disagreeing with the group that someone must be very confident the group is wrong before dissenting.) Overall, these activities really work out to be a very small percentage of their grade. But, giving them ownership of that aspect of their grade really has given them a pretty positive outlook on the course.

NEVER before in all my teaching have I had students come up to me and say, "I understand WHY we are doing these activities, but..." when they had something to critique or suggest about it. I'm going to add more of these activities each year, because they really do learn from them. It just wasn't feasible to do them every lecture the first year because it takes time to write really good questions for that sort of activity.

I make them all case-based and clinically relevant...things like, "You are the first to arrive at the scene of an accident" and then I go into an explanation of some injury. I.e., "You see a large chunk of metal protruding from the lateral side of the victim's arm..." and have them answer questions about the anatomy...are there major blood vessels at risk and which ones, what nerves could be damaged, what muscles might be damaged, what functions might be compromised and what future disabilities might this patient face if the damage can't be repaired, etc.

You can do the same thing with some basic math too if that's what you need to teach them...I might ask something like, "A physician has ordered a patient be given an i.v. drip of a medication at a rate of 10 mg/h. You hang a 250 ml i.v. bag of saline and set the flow rate as 20 ml/h. The medication is dispensed from the pharmacy in a vial containing a solution of 250 mg/ml. You add 1 ml of the medication to the i.v. bag. A few hours later, the patient dies as a result of an overdose of this medication. Who needs to call a lawyer, you or the prescribing physician? Explain."

I have NO qualms about mentioning things like patients dying or malpractice lawsuits when posing questions to my students...nursing, med or dental. They all need to be aware that they are not learning this stuff as busy work, but because people's lives depend on them getting it right EVERY time, not just 70% of the time.

 

[Klockan3]

I guess it depends on whether or not you believe the statement is true or not. The current education fad presents data to show it is true (specifically, student FCI scores do not change), but that begs the question as to what the FCI measures.

I looked up FCI, and it seems that it just checks some fundamental concepts about Newtonian physics. Things that every good student should have no problem with. So to me it seems like they are placing focus on the "lesser" students.

But of course when you start calling 80% of the class "lesser" then something is wrong. Maybe that is the problem people have with physics, that lectures do not give them enough "proof" for it to feel intuitive.

Then the question rises, how many taking these courses are actually capable of learning this stuff well without a huge amount of work? The ratio of high school students taking physics have doubled the last 20 years, that ought to have some effect. No matter what people say some people do struggle much more with abstract concepts than others.

 

[berkeman]

Great post, Moonbear.

 

[Andy Resnick]

I agree! Lots of great ideas in there.

 

[Sankaku]

Great post, Moonbear.

Yes, very interesting. I like the approach to student feedback you have taken. So many instructors seem to want to avoid getting honest feedback on their teaching. Making the feedback specific, as you have done, helps make it constructive.

 

[denverdoc]

I am so pleased for all the wonderful responses and stimulating discussion. Some of the issues Moonbear addressed were exactly what I needed to hear, as I had been fumbling in the dark re random vs "overly guided" efforts to have a savior and believers. I also wondered as has every teacher re seating. At one point I ended up with a forced rotation where each row of students rotates forward each week with the front row going back. My ratings went up--class performance, maybe? Some became squatters because of various hearing/visual issues I couldn't ignore. Generally folks liked the move, and those who incessanty chat in the backrow were shut down.

I was watching my class taught CPR today--out of necessity to make room for the dummies, a circular arrangement of chairs. What a difference. The rank and file system may not be so good. I am lucky insofar as the class sizes are 30. But engaging those 6 sitting 40-50 feet away, just as difficult as reaching the 30 or so in the ack row of an auditorium of 200. I think what I might try is a horseshoe shaped arrangement no more than two rows deep and wander about the core as I lecture. Las Vegas model.

A bit back i heard a radio show wherein a non-traditional educator pointed out that in so amny ways our answer to teaching is still based on a model 300 years old, and that the R/D budget for education was a drop in the bucket compared to the NIH. A small drop compared to anything really. But this is one obstacle I face daily is a corporate structure with idiots in charge--I can't get a $1K clicker system that forces the audience to answer questions and can be used for grading instead of daily quizzes that eats up time, yet all to happy to buy into a 50K dollar sim man that does nothing right.

In this regard my problems are twofold. There is always the horse that can be led to water and can't drink because of some obstruction, or sniffs the water and decries foul, and those who just ain't thirsty. That is the greatest challenge. Gimme medical students any day for ease of teaching: toss out any half assed sylllabus, and a couple of slide banks in some moderately funny fashion, and a couple understand the material as well as you do, and if you're real unlucky a few will complain. Instead, I shudder to think re the potentia for misadventure if upper classes and/or certification/lisencure soesnt get in the way.

 

[Moonbear]

Part of the problem of our education being "antiquated" is that there's no requirement for university faculty to have ever taken a course about education. Apparently, conventional wisdom assumes that just because you figured out how to get a Ph.D. in the field you teach, you must have figured out how to teach by osmosis or some such.

Though, yes, the other part of the problem is that the funding and studies on educational methods are pretty abysmal. I think of the "ideal" studies I'd like to do, but then realize they are impossible to conduct at the levels of funding one can get for educational research.

But, the sum total of that is that university faculty keep reinventing the wheel with educational methods. As you talk about things like seating in a horseshoe configuration, it occurs to me that, indeed, my high school teachers were using that approach effectively 20 or so years ago. I actually think older classrooms were designed better than newer ones, especially when you have desks and chairs that are moveable, rather than bolted to the floors. Being able to configure your classroom seating as you want makes so much more sense to me, at least for smaller class sizes.

 

[Klockan3]

The most important thing is that the teacher should like to teach. No amount of tricks matters if you don't, and just about any trick works if you do. What happens is that when you try new tricks they notice that you actually care about them learning things causing them to shape up a bit. That doesn't mean that mass reproducing these all over would have any significant effect.

 

[Vanadium 50]

Part of the problem of our education being "antiquated" is that there's no requirement for university faculty to have ever taken a course about education. Apparently, conventional wisdom assumes that just because you figured out how to get a Ph.D. in the field you teach, you must have figured out how to teach by osmosis or some such.

And in secondary schools it's exactly the opposite. "A good teacher can teach any subject" is a commonly heard mantra, often used to explain why physical science teachers don't really need a background in physical sciences.

 

[Moonbear]

The most important thing is that the teacher should like to teach. No amount of tricks matters if you don't, and just about any trick works if you do. What happens is that when you try new tricks they notice that you actually care about them learning things causing them to shape up a bit. That doesn't mean that mass reproducing these all over would have any significant effect.

I do agree that if you don't like it, nothing in the world is going to make it appear you do. On the other hand, it is not enough to like it. Yes, if you like it, you will strive to find ways to improve, but it's unfortunate that so much time is wasted spinning one's wheels to find ways to improve when there are time-tested theories that really can help right from the start.

And in secondary schools it's exactly the opposite. "A good teacher can teach any subject" is a commonly heard mantra, often used to explain why physical science teachers don't really need a background in physical sciences.

I know! That mantra only holds true if by "good teacher" you mean someone with an extremely highly educated background in a wide variety of subjects.

This is why I like having this forum, and folks like Chi Meson around who have formal training in education. We need to do more to bring the two worlds together. It's actually on my plans for the next adventure in funding applications, to get a state-wide outreach program going to provide some added training for secondary school teachers in the sciences (particularly anatomy...we have three universities in the state that teach medical or osteopathic students, and all have faculty interested to one degree or another in outreach, so as soon as I get a few other projects off the desk, that's the next one to delve into). I really do hope that in return, those of us providing advanced anatomy training will reap some benefit of the educational theory background offered by the teachers.

To me, if I got to have the ear of Obama or Congress for just 10 minutes, that would be the thing I'd recommend as most important for improving our country's educational system...funding for more cross-communication between secondary and higher educators.

 

[Astronuc]

To me, if I got to have the ear of Obama or Congress for just 10 minutes, that would be the thing I'd recommend as most important for improving our country's educational system...funding for more cross-communication between secondary and higher educators.

I made such a point 30 years ago. It occurred to me that when I was in high school, there was no feedback from universities to our high school, and then when I was going through university, I noticed there was no effort for faculty to interface with school districts - except for the odd program to bring high school (usually science) teachers in for one week of training on some introductory level of a particular discipline.

I haven't seen any effort to change what I see as a gross deficiency in the last 30 years. Local school districts seem to toddle along on their own. The results locally are a pervasive mediocrity with the possible exception of the top 5% of students who get put in the best programs with the best teachers while the rest of the student body seems to founder.

I have seen some effort on the part of certain education groups like the AAPT and ACS to do something, but it doesn't seem to be sufficiently broad to have a huge impact on the majority of education (or majority of students). I was fortunate to have teachers who actually participated in national math and chemistry organziations. The two teachers were also department heads, taught the honors courses, and battled the bureaucracy as best they could.

 

[denverdoc]

Part of the problem of our education being "antiquated" is that there's no requirement for university faculty to have ever taken a course about education. Apparently, conventional wisdom assumes that just because you figured out how to get a Ph.D. in the field you teach, you must have figured out how to teach by osmosis or some such.

Though, yes, the other part of the problem is that the funding and studies on educational methods are pretty abysmal. I think of the "ideal" studies I'd like to do, but then realize they are impossible to conduct at the levels of funding one can get for educational research.

But, the sum total of that is that university faculty keep reinventing the wheel with educational methods. As you talk about things like seating in a horseshoe configuration, it occurs to me that, indeed, my high school teachers were using that approach effectively 20 or so years ago. I actually think older classrooms were designed better than newer ones, especially when you have desks and chairs that are moveable, rather than bolted to the floors. Being able to configure your classroom seating as you want makes so much more sense to me, at least for smaller class sizes.

In total agreement here, MB. I have been blessed with the good fortune of having a handful of outstanding teachers in my past, but obviously know only enough to try as best as I can to emulate their styles which clicked with me. But this sort of self-referential thinking aint doing any good for some kid raised in a house without a single book, who has never had curiosity reinforced, and who seems most interested not in mastery of the material, but moving on to the next course where he will be even more confused and angry, until he quits or gets kicked out. I will likely try to rectify this with some local courses the campus will pay for--but who knows whether the teachers there will be any good either?

Can anyone recommend some on-line courses, books, or other resources? I have some down time over the next few months as my primary duties will be in developing a http://www.laerdal.com/document.asp?docid=1022609" as the programming llanguage used by the mannekin/robot is LSL.

My god when I first ventured into second life, I thought to myself, when did all this happen? I really do need to get out more.

 

[mikerocks4250]

Maybe the problem is at the foundation and not in the class room?

 

[Moonbear]

Maybe the problem is at the foundation and not in the class room?

Can you clarify what you mean by foundation? Do you mean the school and their policies, or do you mean the background of the students, or perhaps some of both?

The biggest challenge in teaching really is teaching the lower level student. Those top students are easy...they love learning for the sake of learning and you almost don't need to be there for them to learn. But, you have to teach the students you get, not the ones you necessarily want. So, trying to come down to the level of the poorer student is really challenging.

I'm getting in my teaching evaluations now (submitted anonymously online, so I can read them as they are submitted). So far, as much as I'm working on improving it, I'm still getting comments from some that my explanations of material are going over the student's heads. Though, that may be a bit because I stepped back and let the TAs do more work this semester. The TAs are getting great evaluations, and I'm laughing about that, because as I walked around and listened into them teaching the students, I heard them repeating almost word-for-word the way I taught them the material 2 years ago and the way I would still explain it if I was teaching it instead of the TAs. Usually I only needed to step in when the concepts got a bit more difficult and the TAs didn't have enough background to teach those small bits of the material, so it's not surprising they thought I went over their heads more often, because that was the tough stuff the TAs weren't even touching. But, I'm still working on that. I keep looking at the exam results and figuring out the trouble spots and each year improve my lectures on those topics until I finally get it right.

I'm also laughing a bit because one of the complaints is that there were too many students crowded around one table for them all to see what was being explained or pointed out. Ha ha! That's because they all flocked to the tables the cute TAs were at (and yes, I know they were drooling over the TAs...several admitted it openly). But, in the end, that sort of critique helps me out, even if it's a ding on the evaluations. It allows me to go to my dept. chair and show him that the student:faculty ratio is totally out of whack and I need to have more faculty or TAs in the lab.

By the way, I do think something finally clicked for my students. Their final exam scores were absolutely outstanding, and I was so happy to finally see the show me their full potential. I know they COULD have done that all semester if they applied themselves...they are fairly bright students (yeah, there are always some exceptions, but we won't dwell on those), so all they really needed was to learn how to study right.

The interesting thing is as I talk to students from last year or two years ago, once they've had more classes and started getting into the clinics, they are much more appreciative of what they learned than during the class itself. Also, they realize after they've taken more classes just how much effort our department puts into teaching them (not just me, but all the faculty in our courses). It makes me wish we could send out teaching evaluations a year after the course ends to really find out what they think of us in retrospect.

 

[Sankaku]

I'm getting in my teaching evaluations now...

Very interesting, Moonbear. I enjoy reading your discussions of your teaching methods and their impacts.

 

[denverdoc]

MB, I think we all struggle with this dilemma of how to reach and teach the lower level student. Let me explain a bit about my situation and ask for any insights/additional comment you may have on this issue.

Where I work is first of all is a for profit private enterprise. It is both a corporation and "family owned" biz with generous stock options for FTE employees. Some who have been there for the ten years have reaped huge financial rewards, and include everyone from the receptionist to security personnel.

In addition we have built in incentives that can amount to a couple of additional paychecks per year which depend in essence on student retention and promotion.

That's the background; the palatable side of the equation is that in New Mexico we are starved for well trained health care workers of all stripes, and the public educational system here might be described as mediocre in a good year.

The bottom line is that the admission standards are non-existant. And so I start each "boot camp" with the hope that in the right environment all seeds will prosper. I arrange at my own expense additional hours of lecture: 1:1 tutorials are available at no charge from upper classmen, and still some people seem not to get it, no matter how much effort is made.

Now if it were some matter besides taking care of lives, I could probably find some uneasy moral equilibrium that allows me to fail a few, and not incur the wrath of fellow empoyees by failing too many, yet still not spend too much time on those who probably shouldn't be there in the first place, but because of either greed or a pay for a shot lotto ticket at a better life...

But as a physician who has seen too many disastrous results from basic incompetence...I just can't seem to get a handle on this slippery pile of conflicting issues. Much of this I have already said and don't want to bore folks with excess detail, but I hate these additional incentives which should be assumed at the start or you're in the wrong biz, and put peer pressure in the equation.