notes.2, cr fri 18 apr 1997 by rha 2. Banesh Hoffman in foreword, june 30, 1962, jacques barzun writes for the past 30 months... this means, sinca ca january 1960 running debate on testing trig by bh articles in harper's magazine, 1961 and the american scholar, 1959. barzun himself had questioned the tests in the 1940s in his book teacher in america see also william h. whyte, jr., the organization man simon and schuster, 1956, jacques barzun, the house of intellect, doubleday anchor, 1957. ----------- Contents [verbatim] 1 A Little Learning Is a Dangereous Thing 2 The Business of Testing 3 The Flight from Subjectivity 4 Objectivity and Ambiguity 5 The "Best" Answer 6 Better Minds 7 "What's in a name?" 8 National Merit 9 Statistics 10 Challenge to the Testers 11 Critical Thinking 12 Editorial Woes 13 "The wind bloweth where it listeth" 14 Return to the Colonies 15 Interlude on the Adantages of Science 16 Einstein Slighted 17 Light on the Atom 18 David and Goliath 19 Don't Be Pro-Test -- Protest Index ----------- 1 A Little Learning Is a Dangereous Thing This chapter condemns multiple-choice test, in which questions are to be answered by a choice, without giving a reason. Two examples are given of multiple-choice questions having wrong answers which can be defended as right: one from Mr. Batty, 1959, the other, the "colonies" question, from the SAT booklet of 1956. 2 The Business of Testing The story of Oscar Buros, Mental Measurements Yearbook, from 1938. The multiple-choice testing industry. 3 The Flight from Subjectivity Difficulties with grading essay questions: subjectivity, handwriting, etc. 4 Objectivity and Ambiguity Weaknesses in the case of the professional testers' case for the multiple-choice test, repleat with many examples. 5 The "Best" Answer Another problem of multiple-choice tests. The best answer requires a subjective decision because there is more than one answer which is correct, or none is. 6 Better Minds Multiple-choice tests descriminate against creative and clever people. For example, CEEB tests do not correlate well with subsequent college performance. 7 "What's in a name?" * IQ tests do not measure intelligence, unless it is defined as performance on IQ tests. And in that case, the ambiguity of the word harms thte people scorred, whether high or low. * the SAT (verbal and arithmetical) scores are akin to IQ scores, correlate poorly with intelligence, and with scholastic aptitude. * The English Composition test does not measure ability at English composition. In fact, the test has had a negative effect on the teaching of English composition in schools. 8 National Merit Here, in service of his conviction that MC tests are; * biased against the critical and creative intelligence in favor of the superficially brilliant, * poor indicators of actual scholastic performance the author goes after the National Merit Scholarship Corporation. In 1959 for example, the NMSC, * examined almost half a million students, * rejected 98% of them on the basis of an MC test alone, * rejected 98% of the survivors on the basis their academic records, * named 920 as National Merit Scholars. Hoffman notes that, according to the NMSC itself, 82% of these ended up in the top 25% of their class, so the predictive value is poor. Also, he pointed out that the NMSC agreed with him that the tests are not predictive of creativity. 9 Statistics "The purpose of this chapter is mainly to undermine excessive faith in statistics" and it specifically criticizes this one: The correlation of a high school student's SAT score with her subsequent freshman grades in college is about 0.50. This correlation is less than that of the heights and weights of a group of people. 10 Challenge to the Testers This a chapter of strategy. Hoffman chronicles the litany of sharp criticism aimed at the MC testing establishment, and its agility in dodging. He reasons that the best strategy for bringing out the weaknesses and forcing a response from the test industry is to expose the weaknesses of individual exemplary questions from actual tests. Also, here we find, on page 150, an excellent short list of the problems with MC tests. the strrategy was applied in two articles by hoffman: spring 1959 , the american scholar, and march 1961, harper's magazine. cf p. 157. 11 Critical Thinking This short chapter is a critique of a single MC test question, showing once again that a poor question penalizes the most clever applicants. 12 Editorial Woes In his Harper's Magazine article, Hoffman had attacked five specific questions. One, the "critical thinking" question, was revisited in this book in chapter 11 above. Here is another. The ETS had responded to the Harper's article with a booklet defending all five questions, in 1961. In this chapter Hoffman demolishes the ETS response to this one question, the "editor question". 13 "The wind bloweth where it listeth" the question dissected in this chapter, the "wind blowing" question, is from barzun, the house of intellect, 1957, and was taken from the SAT booklet of 1956. its style shows it to be a model of hoffman's style. as before, hoffman devastates the feelbe defense of the ETS to barzun's critique. 14 Return to the Colonies the hoffman critique on the colony question first appeared in the american scholer, spr 1959. the ETS response, in a later issue of the same periodical, and extended in the booklet of 1961, is taken to task in this chapter. 15 Interlude on the Adantages of Science introduces the advantages of science questions in the challenge strategy, two such are treated in the next two chapters. 16 Einstein Slighted Well, no point argueing with Hoffman about a physics question. The first of the two science questions busted by Hoffman, the subject of this chapter, has to do with the chemistry of combustion, and is described by ETS as of average difficulty. While the dull student goes directly to the wanted answer, a more knowledgeable student (Hoffman, for example) sees no acceptable answer. To this complaint, published first in the Harper's article, the ETS put forth its beset defense, which Hoffman very convincingly shows to be inadequate. Most readers of this chapter would agree, I believe, that the subject question is a poor one, and that it is biased (unintentionally perhaps) toward the duller mind. 17 Light on the Atom Here we find the second science question, from physics, and regarded as difficult. The devasting analysis of this SAT question also appeared in the Harper's article, and subsequently defended by the ETS in its response pamphlet. In this counter response, Hoffman makes global points against the integrity of ETS and its overall defense of MC tests. 18 David and Goliath Here Hoffman gives us most of the detailed correspondence between a student (DAvid) and the ETS (Goliath) in which the student cojplained (correctly) about a poor question on the English Composition test of March 18,1961, given at ethesda, MD. The defense of Goliath is roundly trounced by Hoffman, and we are convinced that the ETS will go to any lengths to defend a question, no matter how fualty it may be. 19 Don't Be Pro-Test -- Protest Hoffman looked at an entire SAT test and concluded that a large proportion of the questions were defective. In conclusion, he uterly condemns MC testing, and seems to approve of the essay type tests used in other countries. end: notes.2