PROCEEDINGS

Testing /
Assessment

 
     TESTING/ASSESSMENT
Back Proceedings
An Experience with Performance-based Assessment
Janette C. de Oliveira
Leni Puppin
Solange Jaccoud da S. Bonn

INTRODUCTION

Owing to the complexity involved in conceptualizing performance assessments we have considered the following defining characteristics opportune as a brief introduction to the idea with particular reference to second and foreign language contexts:

“It seems appropriate to think of distinctions between tests which require, for instance, "fill-in items in a cloze test" and performance tests in terms of a continuum from least real world or authentic to most real world or authentic.” (Norris, Brown et. al., 1998)

“A defining characteristic is that actual performances of relevant tasks are required of candidates, rather than more abstract demonstration of knowledge, often by means of paper-and-pencil tests.” (MacNamara, 1996)

“What distinguishes performance assessments from other types of tests, then, appears to be that (a) examinees must perform tasks, (b) the tasks should be as authentic as possible, and (c) success or failure in the outcome of the tasks, because they are performances, must usually be rated by qualified judges.” (Norris, Brown et. al., 1998)

BACKGROUND INFORMATION

Centro de Línguas para a Comunidade (CLC) is a Foreign Languages Center run by Universidade Federal do Espirito Santo (UFES). It has a twofold objective: (1) to provide in-service training for students from the Language Department, and ultimately (2) to offer language courses to the community at a low cost.

CLC’s English staff has currently 4 academic coordinators, 8 permanent teachers, 50 undergraduate school trainees and a student population of about 4.700 people. Mixed social backgrounds and literate qualifications make up the profile of the students. Students age range from 11 to 75. English, Spanish, German, French and Portuguese as a Foreign Language are taught at all levels, from basic to advanced. Highest in demand are English and Spanish, in this order. Class size ranges from 15 to 20 students per group.

The Performance-based Assessment Strategy proposal was initially agreed upon between CLC’s administrators and an external consultant specialized in assessment, measurement and evaluation. Later the proposal was refined by the participant team and extended to the other language courses.

CLC’s needs for implementing changes in the current assessment procedures grew out of teachers and students’ dissatisfaction with the following issues:

  • Tests results were not reflecting students’ capability (Allan, 1995);
  • Low interrater reliability (Gamaroff, 2000), e.g. judgements on student’s pieces of writing varied significantly from one teacher to another;
  • Teachers were encouraged to utilize “communicative” approaches but kept assessing their students with traditional instruments, e.g. ‘provide questions to given answers’, ‘fill in the blanks with the correct word’ and similar;
  • Teachers were either not provided with or never developed verbal descriptors instruments, for instance, to assess their students during oral presentations; therefore assessment and grading were highly subjective.
  • Many test items showed poor content validity (Popham, 1981, Davies 1990, Heaton, 1990), e.g. they bore little resemblance to the activities that teachers usually conducted in the classroom.
  • Many tests had low construct validity and were considered valid only within program achievement. These instruments were merely addressing themselves to the content of the course instruction via the program and not to the real world construct (Popham, 1981, McNamara, 2000)

IMPLEMENTATION STAGES

The activities carried out during the implementation stages can be summarized as follows:

  • Stage I, Activities:

    1. selection of a representative group of English teachers;
    2. development of a performance objectives continuum describing what is expected from students on each of the four skill areas – listening, speaking, writing and reading – at the end of each ‘cycle’ - two hundred hours of instruction;
    3. discussion, revision and finalization of continuum;
    4. design, refining and finalization of performance objectives;
    5. establishment of rating grids.

  • Stage II: Activities:

    6. identification and selection of appropriate instruments to measure desired performance;
    7. design of an instrument ‘bank’ based on pre-established criteria of validity and reliability (according to Popham, 1981, validity and reliability issues concerning measurement instruments should answer the following questions, respectively: ‘is the instrument an appropriate one for what needs to be measured?’ and ‘Does the instrument yield consistent results?’)

  • Stage III: Activities

    8. preparation of support materials, e.g. an instruction manual for the application of instruments, a letter to students and parents containing information about the new system;
    9. training sessions for teacher-trainees to implement and use the new system;
    10. piloting of instruments on pre-selected groups of beginners, intermediate and advanced;
    11. analysis of data, identification of problems and search for solutions;
    12. publication of results and implementation of changes that will result in improvement to quality of the teaching/learning process.

| 1 | 2 |
BRAZ-TESOL  Rua Coronel Oscar Porto, 800 - Paraíso - São Paulo/SP - CEP 04003-004  PHONE/FAX  55 11 3559 8782  - braztesol@braztesol.org.br
.:. Copyright BRAZ-TESOL - 1986 / 2007 .:. All rights reserved .:. designed by Quorum .:.
LOGIN Chat! E-mail BRAZ-TESOL Member Area