TESTING AND EVALUATION IN FOREIGN/SECOND LANGUAGE PROGRAMS
Fall 2005
Meeting Time : Thursdays, 3:30 – 6:00 p.m.
Meeting Place : Modern Languages Bldg. 306
Instructor: Renate A. Schulz
Office: LSB 312
Phone: 621-7388 or 621-7385
e-mail: schulzr@u.arizona.edu
Listserv: SLAT587@listserv.arizona.edu
Office Hours: Tuesdays, 1:00 – 3:00 p.m. and Wednesdays, 10:00 – noon
or by appointment
Intended Audience for 587 : Ph.D. students in SLAT, M.A. candidates in foreign languages or ESL, in-service and prospective foreign/second language teachers, and FL/L2 program directors/supervisors
Course Objectives
–Understand fundamental concepts and principles of psychometric measurement as well as strategies for alternative/performance assessment
–Learn how to construct language tests for various skills and various measurement purposes for delivery via paper and pencil and via computer-assisted testing (CAT)
–learn how to analyze tests, interpret test results, and utilize test results to improve quality of tests
–evaluate various assessment approaches appropriate for language learning
–examine current trends and problems in language testing and evaluate the role of testing and evaluation in the overall FL/L2 curriculum
–be able to critically read research in language testing
Course Materials
Required:
Hughes A. (2003) Testing for Language Teachers , sec. ed. Cambridge University Press.
*Selected additional readings available at E-Reserves: http://eres.library.arizona.edu/courseindex.asp Student password is: testing
ILTA Language Testing Bibliography 1991-1999 . (Print from ILTA website: www.iltaonline.com )
Recommended:
Norris, J. M., Brown, J. D., Hudson, T. & Yoshioka, J. (1998). Designing Second Language Performance Assessments . Honolulu: University of Hawaii Press.
Tentative Course Outline: Topics and Assignments
Please note that instructor reserves the right to change assigned readings during semester.
Aug. 25 Introductions ; Review of Syllabus
Professional resources for language testing (ILTA Language Testing Bibliography available on ILTA website)
History of FL testing
Types of tests and their use: Quiz
Basic concepts of testing and evaluation
Sept.1 Readings: Hughes chap. 1 – 5; *Barnwell (1996).
Explore Testing/Assessment on the Internet. Share short description of at least one site (with URL) on 587 listserv.
Some start-up URLs:
International Language Testing Assn. (ILTA):
Hughes textbook site: http://uk.cambridge.org/elt/tflt/weblinks/
TOEFL: www.toefl.org
Center of Applied Linguistics: www.cal.org (Be sure to look at FL Test Database); and websites of national foreign language centers.
Summary/Discussion/Quiz – TF
Basic stats for assessment purposes
Standardized tests
Discrete-point vs. communicative (integrative, pragmatic) testing
Sept. 8 Readings: Hughes, chap 6, 7
* Standards for Foreign Language Learning: Preparing for the 21st Century (1996). *Swender & Duncan (1998).
Recommended to deepen understanding of basic statistics for assessment: *Davidson, F. (2000) or * Brown, J.D. (1991 and 1992).
Summary/Discussion/Quiz – MC
Instructional Objectives and Standards
Pros and cons of multiple choice testing and its place in FL/L2 assessment
Assessing language knowledge vs. language use
Sept. 15 Readings: *Bailey (1996); *Wolf (1993); Hamp-Lyons (1997).
Summary/Discussion/Quiz – matching:
Washback
Communicating in the interpretive mode: Testing the receptive skills
Sept. 22 Readings: Hughes, chap. 11, 12, 13
Summary/Discussion/Quiz – short answer:
Testing the receptive skills, cont.
Sept. 29 Readings: Hughes, chaps. 8, 9, 10 *Kuo. & Jiang (1997).
Summary/Discussion/Quiz – CLOZE:
Communicating in the interactive/presentational modes
Testing oral proficiency: The FSI and ACTFL Oral Proficiency Interview (OPI), SOPI, SPEAK Test
Oct. 6 Project Proposal Due
Readings: Hughes, chaps. 14, 15, 16; *Hamp-Lyons & Connor-Linton (1995).
Summary/Discussion/Quiz – MC:
Interactive/Presentational Mode, cont.
Oct. 13 Readings: Norris, et al. (1998), chap. 1 – 4. *Shohami (1998);
Summary/Discussion/Quiz – Essay
Performance assessment (aka “alternative,” or “authentic” assessment, incl. portfolios)
Oct. 20 Readings: Norris et al., chaps. 5 – 7; Lynch & Shaw (2005); “Portfolio assessment in the foreign language classroom” in www.nclrc.org
Performance assessment cont.
Examination
Oct. 27 Readings: Hughes, Appendix 1; *Alderson, Clapham & Wall (1995) , chap. 4; *Code of Ethics for ILTA; *Shohamy (1997). McNamara (1998)
Recommended reading: *Davis (1997); *Lynch (1997);
Summary/Discussion/Quiz – short answer
Testing the test
Interpreting test scores
Ethics in testing
Nov. 3 Due: Assessment development tasks (oral presentation only)
Readings: *Dörnyei (2001);. *Busch & Turner. (1993); Horwitz (2001).
Summary/Discussion/Quiz – CLOZE (any type)
Demo/critique of assessement tasks
Assessing personal variables (attitude, motivation, learning styles, etc.)
Nov. 10 Readings: Hughes, Appendix 2; *Jamieson (2005), *Brown (1997): *Dunkel, P. (1999).
Recommended: *McNamara, T. (1996).
Additional Assignment: Take one of the U of A CAPE (computer-adaptive placement exam) examinations.
Summary/Discussion/Quiz – Matching
Demo/critique of assessment tasks
Introduction to Computer-Assisted Testing and Computer Adaptive Testing
Nov. 17 Readings: *DiFino & Lombardino (2004); Ganschow, Sparks & Shaw (2001); Sparks (2005); Sparks, Javorsky & Ganschow (2005).
Prognostic testing
Learning Disabilities and the FL requirement
Administration of MLAT (Modern Language Aptitude Test)
Group work: Development of assessment tasks
Nov. 24 Thanksgiving, no class
Nov. 30 Post abstract of your final course project/paper on listserv
Dec. 1 Due: Final written copies of assessment development tasks
Readings: * Pennington & Young (1989); *Nunan, D. (1992);
*National Board for Professional Teaching Standards, Standards for FL Teachers (2001); *INTASC Model Standards for Licensing …; (2002).
Discussion of prognostic testing and disability accommodation cont.
Multiple Modes for Evaluating Teaching
Program/curriculum evaluation
Presentation of course project abstracts
Course evaluationDec. 7 Final copy of paper and abstract due
Course Requirements
1. Course exam (30%)
2. Course project (can be done individually or–for extensive research projects–in groups of up to four students): Research project or review paper, examining in depth an area relevant to L2/FL assessment. Submit one-page proposal for instructor approval by Oct. 6 . If project is a review paper, the proposal should include title, purpose, and focus of project, and start-up bibliography. If project is an empirical research study, include your research question(s), a description of your data collection method(s) and a start-up bibliography. The final paper (10 pages minimum, not counting bibliography, tables, charts or appendices) together with an abstract should be typed, double-spaced, and following APA or MLA guidelines. Please submit paper by December 7, in folder, without cover page, binding or staples. Please post your abstract on the 587 listserv for your fellow students as soon as possible, but no later than November 30. (20 %)
3. Leading discussions . Groups of 2-3 students will serve as discussion leaders on assigned readings and prepare, administer, and evaluate a 10-minute quiz (following the format specified) on the assigned readings. The graded quizzes will be collected by the instructor and count as pass/fail (completed/not completed) under course participation (see #5 below). Please note that discussion team should focus on most salient points of reading assignment and find ways to engage class members in productive review and discussion. Discussion and quiz may not exceed 40 minutes. (20%) EACH STUDENT NEEDS TO SIGN UP TO SERVE AS DISCUSSION LEADER FOR A MINIMUM OF TWO SESSIONS.
4 . Development of assessment tasks
Working in teams of two or three, students will develop language assessment tasks (language and level to be decided on by team), assessing achievement/proficiency in interpersonal, interpretative and presentational language use, language structure and culture. Please use the “Assessment Development Planner” sheets provided for specification of individual tasks/items. Assignment should include minimally three evaluation task types for each category. Please note that an evaluation task or item can assess more than one factor. Assessment tasks should approach target language use (communicative authenticity) characteristics as closely as possible. BE CERTAIN THAT ALL TEAM MEMBERS REVIEW EACH ITEM TO ASSURE THAT ITEM IS VALID FOR THE PURPOSE AND LEVEL INDICATED, HAS CLEAR INSTRUCTIONS, AND COMES WITH APPROPRIATE EVALUATION CRITERIA/PROCEDURES.
Submit final copies of “Assessment Development Planner” sheets for all tasks/items by December 1 . Be ready to present one of your “best” or most problematic assessment tasks for critique or help in class by Nov. 3. (20%)
5. Since this course meets only once a week, regular attendance and active course participation (including quizzes under #3 above) are essential. Please note that any unexcused absence or missing quiz will reduce the final sum of your course points by 10 points and may, therefore, lower your final grade for the course. (10%)
Bibliography
Alderson, J. C., Clapham, C., and Wall, D. (1995). Language test construction and evaluation . Cambridge: Cambridge University Press. Chap. 4 “Pretesting and analysis,” pp. 73 – 104.
Bailey, K. M. (1996). Working for washback: A review of the washback concept in language testing. Language Testing, 13, 257 –279. (Please note that vol. 13. #3 (1996) is special issue on washback.
Barnwell, D. P. (1996). “Integrative Language Testing” and “Direct Integrative Testing,” pp. 107-135 in A history of FL testing in the US from its beginnings to the present. Tempe, AZ: Bilingual Press.
Brown, J.D. (1991 and 1992). Statistics as a foreign language–Part 1: What to look for in reading statistical language studies (pp. 569-603), and Part 2: More things to consider in reading statistical language studies (pp. 629-665). TESOL Quarterly 25, 4 and 26, 4.
Brown, J. D. (1997).Computers in language testing: Present research and some future directions. In electronic journal Language Learning and Technology 1, No. 1, 44-59. ( http://llt.msu.edu/vol1num1/brown/default.html
Busch, M. & Turner, J. (1993). Using Likert scales in second language research. TESOL Quarterly 27, 733-39.
Code of Ethics for ILTA (2000). Language Testing Update 27 (Summer), 14-22. (Available on ILTA website www.iltaonline.com )
Davidson, F. (2000). The language tester’s statistical toolbox. System , 28, 605-617.
Davies, A. (1997). Demands of being professional in language testing. Language
Testing 14, 3, 328-339.
_____. (1997). The limits of ethics in language testing. Language Testing 14, 3,
235-241.
DiFino, S. M. & Lombardino, L. J. (2004). Language learning disabilities: The ultimate foreign language challenge. Foreign Language Annals 37, 390 – 392.
Dörnyei, Z. (2001). New themes and approaches in second language motivation research. Annual Review of Applied Linguistics 21, 43-59.
Dunkel, P. (1999). Considerations in developing or using L2/FL proficiency computer aided tests. In electronic journal Language Learning & Technology 2,2:
Ganshow, L., Sparks, R. L. & Shaw R. (2001). The issue of accommodations, waivers, and course substitutions for students who have difficulties learning a foreing language: An interview. ADFL Bulletin 32, 60 – 69.
Hamp-Lyons, L. (1997). Washback, impact and validity: Ethical concerns. Language Testing 14, 3, 295-303.
Hamp-Lyons, L. & Connor-Linton, J. (1995). “Rating nonnative writing: The trouble
with holistic scoring” and “Looking behind the curtain: What do L2 composition ratings
really mean?” TESOL Quarterly 29, 759-765.
Horwitz, E. K. (2001). Language anxiety and achievement. Annual Review of Applied Linguistics 21, 112 – 126.
Hughes, A. (2003). Testing for language teachers , sec. ed. Cambridge: Cambridge University Press. Required textbook.
INTASC FL Standards Committee (2002). Model Standards for Licensing Beginning Foreign Language Teachers: A Resource for State Dialogue.
Jamieson, J. (2005). Trends in computer-based second language assessment. Annual Review of Applied Linguistics 25, 228-242.
Kuo, J. & Jiang X. (1997). Assessing the assessments: The OPI and the SOPI. Foreign Language Annals 30, 503-512.
Lynch, B. K. (1997). In search of the ethical test. Language Testing 14, 3, 315-327.
Lynch, B. & Shaw, P. (2005). Portfolios, Power, and Ethics. TESOL Quarterly 39, 263 – 297.
McNamara, T. (1996). Concepts and procedures in Rasch measurement. Measuring second language performance . New York: Longman.
McNamara, T. (1998). Policy and social considerations in language assessment. Annual Review of Applied Linguistics , 18, 304-319.
National Board for Professional Teaching Standards (2001). World Languages other than English: Standards for students ages 3 – 18.
Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Honolulu: University of Hawaii Press. Recommended textbook.
Nunan, D. (1992). Program evaluation. Chapter 9 in Research methods in language
learning . Cambridge Univ. Press, 184-210.
Pennington, M. C. & Young, A. L. (1989). Approaches to faculty evaluation for ESL. TESOL Quarterly , 23, 619-646.
Shohamy, E. (1997). Testing methods, testing consequences: Are they ethical? Are they fair? Language Testing 14, 3, 340-349.
____ (1998). Evaluation of learning outcomes in second language acquisition:
A multiple perspective. In Heidi Byrnes (ed.) Learning foreign and second languages: Perspectives in research and scholarship. New York: MLA, pp. 238-261.
Sparks, R. L. (2005). Intelligence, learning disabilities, attention deficit hyperactivity disorder, and foreign language learning problems: A research update. ADFL Bulletin 36, 43 – 49.
Sparks R., Javorsky, J. & Ganschow, L. (2005). Should the Modern Language Aptitude Test be used to determine course substitutions for and waivers of the foreign language requirement? Foreign Language Annals 38, 201-210.
Standards for foreign language learning: Preparing for the 21st century (1996). Yonkers, NY: ACTFL: National Standards in Foreign Language Education Project.
Swender E. & Duncan G. (1998). ACTFL performance guidelines for K-12 learners. Foreign Language Annals 31, 479-491.
Wolf, D. F. (1993). A comparison of assessment tasks used to measure foreign language reading comprehension. Modern Language Journal 77, 473-489.