US Education and the Testing Industry: Past time to rethink multiple guess

By PHK

I suppose it’s heartening to read that Charles Murray, the author of the 1994 controversial book “The Bell Curve” in which he – and coauthor Richard Herrnstein – argued that “those who get ahead in America (mostly whites) are genetically endowed with more intelligence than those who do not (disproportionately African-Americans)," is having second thoughts about the SAT as a reliable measure of college aptitude, e.g. from their perspective more or less "genetically endowed intelligence."

Frankly, I thought it was almost conventional wisdom the SATs are one of the least reliable measurements of a student’s future success in college and that high school grades, activities and recommendations are the best. Murray and others including the wonderful people who have amassed fortunes inflicting the SATs (and other multiple-guess “IQ” exams) on college bound American teenagers, however, have still not entirely given up on these kinds of supposedly “objective” tests as predictors of an individual’s future success.

They now say, according to The New York Times on September 19, that the SAT’s Subject Tests are the single most important indicator.

Oh, come on. Who’s kidding whom?

Or if these still mostly multiple choice question based tests are, in fact, accurate predictors, then something’s wrong with U.S. undergraduate education today because even these kinds of subject matter tests mostly demonstrate that some people are better able to play the College Board’s test-taking-timed-guessing game than others. This, after all, is also what the lucrative SAT cram course industry and books teach.

The problem at the university level – particularly for first and second year students in large classes – is that the same form of guesswork exams are used in too many classes by too many professors and graduate teaching assistants because correcting even short essays takes thought, time and the ability to explain to the student why he, or she, didn’t get the inflated grade he, or she thought he or she deserved. Besides that, the textbook publishers make this even easier by providing text banks from which test questions can be drawn. Multiple choice questions are the fastest and least controversial to correct.

So in a perverted sense, the American testing industry is right: if a student has learned how to ace multiple choice exams without really trying in high school or even earlier – thanks to “no child left behind”- then SAT type exams should be excellent predictors of a student’s ability to succeed through the first couple of years of large college classes as well as to go on to ace the LSATS, the MCATs and the GREs – all also brought to us as screening devices by our wonderful private sector testing industry.

Where things break down, however, is when students are required to write in-class, closed book, essays. Or to give in-class oral presentations based on out of class research. Both skills are required for success in the professional world. This includes entry into and advancement in the U.S. Foreign Service. Written and oral skills are also required for successful PhD candidacy. Multiple guess exams don’t test or train for either.

I think American academia is, therefore, doing the students and this country a major disservice by not requiring written essays and oral presentations – and at all levels. But that means smaller classes, comprehensive exams at the end of each class and better trained, treated and paid instructors.

To do this, as the recently released 2007 OECD comparative study of education in 40 member and partner countries suggests, means substantially more money going into American classrooms and substantially less into administrative overhead. It does not mean more money overall for U.S. education. That same study shows that the US spends more per student than almost every other country that participated in the OECD survey. Most countries surveyed were in Europe or Asia but they also include Canada, Mexico, Australia, China, India and Russia. The 2007 survey demonstrates that the US educational system is slipping in comparison with others – not the other way around.

This should be a wake up call.

It should mean, therefore, taking an especially hard look at and realigning how US taxpayers’ money on education is spent.

To do so, it might help to examine how other countries achieve far better results with less money and adopt or adapt some of their more successful practices. Lavish football stadiums, multiple layers of administrative oversight, teachers that lack even basic subject matter competency and 4,000 student high schools requiring pistol toting guards to keep order do not result in quality education for anyone.

I realize one size does not fit all and that there are some excellent schools and teachers in this country. I’m also not suggesting everything that works for Finnish students for example – and the Finns rank among the highest on the OECD survey once again – will work in the US. But I think that the private testing industry in the US needs major overhaul before it becomes part of the solution, not the problem. And yes, there are a number of excellent examples that could be adapted or adopted from elsewhere.

Comments

[CKR]

Ah yes: the in-class, closed-book essay. I walked into my first semester exam in World History with my blue book and sat down. The professor handed out a sheet with very little writing on it. Three questions. Short ones. Discuss.

I had studied hard for the exam, but my first impression was that we had never covered this material in class. I had never thought any of that out. I might as well walk out now, leaving a blank blue book. Utter terror.

I don't recall the exact questions, but I thought, well, I might as well dredge up whatever I know about the times and places mentioned. French Revolution. Uh-huh. There were many facts and dates in my head, and I started pulling out the ones that were relevant to the questions. Or close.

Patterns started to emerge. I could write a sentence, then another one. Things fitted together, and I eventually came up with three coherent essays.

It took the whole three hours. I got a B in the course, which was good enough by my lights. I may have learned more from the exam than I did from the course.

[Eric]

Sorry PHK, I have to disagree with you on some of what you say.

As someone who's currently winding my way through the educational system, I definitely agree that the SAT is kind of dumb in a lot of ways. I did well on it in large part because I 'test' well. But the overall idea is sound- there has to be a way to compare an 'A' at one high school to an 'A' in another. Comparing a grade at one high school to one at my academically rigorous high school is nearly impossible without some consistent standard. That's why the SAT Subject Tests are reasonable- they test knowledge of a subject in a fairly reasonable way. There's more random facts than I would like to see, but it isn't terrible.

I don't agree at all with your solution. I've never heard of any company in the private sector that requires a sit-down and closed-book essay as part of the hiring process (As we all know, civil service is very different from the private sector). The things that are necessary are teamwork, presentations, and the ability to look things up. At a job I worked at, I needed to calculate the percent change of some stock prices. I forgot the formula for percent change, so I looked it up online. Problem solved. No one is going to point at me randomly and ask me for the equation, so I don't find it very useful to memorize it.

We should be evaluating students based on their participation in (necessarily small) classes, presentations, and their ability to use what they know to conduct further analysis and work in a team setting. Closed-book essays that require simple regurgitation of facts test none of these and are only a measure of how well a student can cram. A test like the one CKR mentioned above is closer to reality, but still unrealistic.

[PHK]

Eric: Then why are high school grades the single best predictor of college success? Maybe it's because they are a measure of a student's motivation to learn and succeed? That's what I think is important - and there are a variety of ways to deal with deficiencies.

What drives me nuts about the SAT is that it's basically a vocabulary test and an upper middle class urban vocabulary at that.

I grew up in a rural area and if the words for different kinds of trees and types of lumber had been on the test my class would have gotten 800s. They weren't - and some very smart kids did poorly on the SAT. However, I had schoolmates (and later students) as well as friends who were not great multiple choice test takers but did very well in college, grad school and later in life because they worked hard, systematically, knew how to analyse, how to study for written finals and comprehensives and were motivated to succeed.

When I said closed book exams, I had a reason. Namely, the Internet has grossly increased the possibilities of cheating and buying papers in particular. Personally I prefer open book exams and research papers - so that someone can do precisely what you said you did.

I don't think, however, that closed book exams are necessarily recitations of the facts - at least not the exams I gave or took in either college or grad school. When I taught, I made sure the questions required thoughtful answers as well as a display of what the students should have learned during the semester to marshall their arguments. There were no "right" or "wrong" answers.

I also used a variety of evaluation tools. They included class presentations, short personal reflection papers on assigned reading, video and guest lectures at the 300 level as well as a short final research paper upon which their oral presentation was based.

But as you point out, small classes are essential. It's just not possible to do the kind of evaluation required otherwise. Regular feedback is part of the learning process. Small classes are also staff intensive -but the US needs to get off the multiple choice/large class fix and reduce funding to administration while increasing it to faculty - if it wants to continue to compete.

And I don't think the kind of test CKR mentioned is unrealistic - but it can be pretty scary if you've never experienced that kind of exam before.