Translation assessors’ perceptions of assessment scales criteria and its effect on the process and outcome of their assessment
Keywords:
Translation Assessor’ perception, Translation assessment, Assessment scale, Analytic scale, Holistic scaleAbstract
The present study investigated the Translation Assessor’ perceptions of the assessment scales and their subsequent assessment behaviors for two analytic and holistic assessment scales. Hence, nine highly experienced Translation Assessor were asked to verbalize their thoughts while assessment student translations using Translation holistic scale and the analytic scale of ESL Translation Profile. Upon analyzing the think-aloud protocols, four themes emerged. The findings showed that when assessment holistically, the Translation Assessor either referred to the holistic scale components to validate their assessments (validation) or had a pre-evaluation reading to have more reliable assessment (dominancy). In analytic assessment, on the other hand, the Translation Assessor used a pre-evaluation scale reading in order to keep the components and their criteria to memory to evaluate the text more accurately (dominancy) or regularly moved between the text and the scale components to assign a score (oscillation). Furthermore, the results of a Wilcoxon signed-ranks test showed that when using the holistic and analytic assessment scales, the Translation Assessor assigned significantly different scores to the texts. On the whole, the results showed that the way the Translation Assessor perceived the scale components will affect their judgements of the texts. The study provides several implications for rater training programs and EFL translation assessment.
References
Bacha, N. (2001). Translation evaluation: what can analytic versus holistic translation scoring tell us? System, 29, 371–383.
Barkaoui, K. (2011). Effects of marking method and rater experience on ESL translation scores and rater performance. Assessment in Education Principles Policy and Practice, 18(3):279-293.
Barkaoui, K. (2010a). Think-aloud protocols in research on translation assessment: An empirical study of their veridicality and reactivity. Language Testing, 28(1), 51–75.
Barkaoui, K. (2010b). Variability in ESL translation assessment processes: The role of the assessment scale and rater experience. Language Assessment Quarterly, 7(1), 54–74.
Barkaoui, K. (2007). Assessment scale impact on EFL translation marking: a mix-method study. Assessing translation, 12(2), 86-107.
Brown, A. V. (2009). Students' and teachers' perceptions of effective teaching in the foreign language classroom: A comparison of ideals and assessments. The Modern Language Journal, 93(1),46-60. https://doi.org/10.1111/j.1540-4781.2009.00827.x
Choi, Y. (2002). FACETS analysis of effects of rater training on secondary school English teachers’ scoring of English translation. Journal of the Applied Linguistics Association of Korea, 18(1), 257-292.
Charters, E. (2003), The use of think-aloud methods in qualitative research an introduction to Think-aloud Methods. Brock Education, 12(2),68-82.
Coombe, C. (2010). Assessing foreign/second language translation ability. Education, Business and Society: Contemporary Middle Eastern Issues ,3(3),178-187.
Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry & research design: Choosing among five approaches. Thousand Oaks, California: SAGE.
Cumming, A., Kantor, R., & Powers, E. D. (2001). Scoring TOEFL translations and TOEFL 2000 protocol tasks: An investigation into Translation Assessor’ decision making and development of a preliminary analytic framework. (TOEFL Monograph Series, Report No. 22.)
Cumming, A., Kantor, R. & Powers, D. E. (2002). Decision making while assessment ESL/EFL translation tasks: A descriptive framework. Modern Language Journal, 86(21), 67–96.
Davidson, M., Howell, W. & Hoekerna , P.(2000). Effects of ethnicity and violent content on
rubric scores in translation samples. Journal of Educational Research, 93(6).18-31.
Deygers, B., Van Gorp, K., & Demeester, T. (2018). The B2 level and the dream of a common standard. Language Assessment Quarterly, 15(1), 44–58.
Eckes, T. (2006). Translation Assessor’ perceptions of scoring criteria in translation and speaking performance assessments. Paper presented at the 28th Language Testing Research Colloquium (LTRC), Melbourne, Australia.
Eckes, T. (2008). Rater types in translation performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185.
Elbow, P., (1999). Ranking, evaluating, and liking: sorting out three forms of judgments. In: Straub, R. (Ed.), A Sourcebook for Responding to Student Translation. Hampton Press, Inc, New Jersey, 175–196.
Goodwin, S. (2016). A Many-Facet Rasch analysis comparing translation rater behavior on an academic English reading/translation test used for two purposes. Assessing Translation, 30, 21-31.
Hamp-Lyons, L. (1991a). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language translation in academic contexts. Norwood, NJ: Ablex. (pp. 241–276).
Hamp-Lyons, L. (2007).The Impact of Testing Practices on Teaching: Ideologies and alternatives. Cummins, J., Davison, C.(Ed). International Handbook of English Language Teaching - (pp.487-504). Springer.
Hijikata-Someya, Y.,Ono, M., Yamanishi,H.(2015).Evaluation by native and non-native English teacher Translation Assessor of Japanese students’ summaries. English Language Teaching, 8 (7), pp. 1-12.
Hirai, A., & Koizumi, R. (2013). Validation of empirically derived assessment scales for a story retelling speaking test. Language Assessment Quarterly, 10(4), 398–422.
Huang, J. (2012). Using generalizability theory to examine the accuracy and validity of largescale ESL translation assessment. Assessing Translation, 17, 123–139.
Huot, B. (1990b). Reliability, validity, and holistic scoring: what we know and what we need to know. College Translation and Communication, 41(2), 201–213.
Hunter, D., Jones, R. M., & Randhawa, B. S. (1996). The use of holistic versus analytic scoring for large-scale assessment of translation. The Canadian Journal of Program Evaluation, 11(2), 61–85.
Kim,S, & Lee,H. K.(2015). Exploring Rater Behaviors During a Translation Assessment Discussion English Teaching, 70(1),97-121.
Khodi, A. (2021). The affectability of translation assessment scores: A G-theory analysis of rater, task, and scoring method contribution. Testing in Asia ,11,1-27. https://doi.org/10.1186/s40468-021-00134-5.
Knoch, U. (2007). The development and validation of an empirically-developed assessment scale for academic translation. Unpublished PhD, University of Aucklan.
Jacobs, H.L., Zinkgraf, S.A., Wormuth, D.R., Hart®el, V.F. and Hughey, J.B. (1981). Testing ESL translation: a practical approach. Rowley, MA: Newbury House.
Joe, J., Harmes, J., & Hickerson, C. (2011). Using verbal report to explore rater perceptual processes in scoring: a mixed method application to oral communication assessment. Assessment in Education: Principles, Policy & Practice, 18, 239-259.
Jeong, H. (2019), Translation scale effects on Translation Assessor: an exploratory study, Jeong Language Testing in Asia, 9(20),1-19. https://doi.org/10.1186/s40468-019-0097-4
Li, H., & He, L. (2015) A Comparison of EFL Translation Assessor’ Translation-Assessment Processes Across Two Types of Assessment Scales, Language Assessment Quarterly, 12(2), 178-212, DOI:10.1080/15434303.2015.1011738
Li, J. & Lindsey, P. (2015).Understanding variations between student and teacher application of rubrics. Assessing translation.26, 67-79.
Liu, Y. & Huang, J.(2020).The quality assurance of a national English translation assessment: Policy implications for quality improvement. Studies in Educational Evaluation,67(2), 1- 8. 10.1016/j.stueduc.2020.100941
Lumley, T. (2002). Assessment criteria in a large-scale translation test: what do they really mean to the Translation Assessor? Language Testing,19 (3) 246–276.
Lumley, T. (2005). Assessing second language translation: The rater’s perspective. New York: Peter Lang.
McNamara, T. (1996). Measuring second language performance: Applied linguistics and language studies, ed. C. Candlin. London: Longman.
Meissel, K., Meyer, F., Yao, E. S., & Rubie-Davies, C. M. (2017). Subjectivity of teacher judgments: Exploring student characteristics that influence teacher judgments of student ability. Teaching and Teacher Education, 65, 48–60.
Mumford, S.& Attay, D. (2021). Teachers’ perspectives on the causes of rater discrepancy in an English for Academic Purposes context. Assessing Translation,48,1-11.
Ohta, R., Plakans, L., & Gebril, A. (2018). Integrated translation scores based on holistic and multi-trait scales: A generalizability analysis. Assessing Translation,38, 21–36.
Ono, M., Yamanishi, H. Hijikata, & (2019). Holistic and Analytic Assessments of the TOEFL iBT® Integrated Translation Task. JLTA Journal,22 65–88.
Panadero, E. & Jonsson, A., (2020). A critical review of the arguments against the use of rubrics. Education Reseasrch Review, 20. 1-19. https://doi.org/10.1016/j.edurev.2020.100329
Plakans L. &Gebril (2017). Exploring the relationship of organization and connection with scores in integrated translation assessment. Assessing Translation,31, 98-112.
Qi, D. S. (1998). An inquiry into language-switching in second language composing processes. Canadian Modern Language Review, 54(3), 413-435.
Rahayu, E.Y. (2020). The Anonymous Teachers’ Factors of Assessing Paragraph Translation. Journal of English for Academic and Specific Purposes.3(1),1-19.
Sakyi, A. A. (2003). A study of the holistic scoring behaviors of experienced and novice ESL instructors, University of Toronto, Toronto, Canada.
Sakyi, A. A. (2000). Validation of holistic scoring for ESL translation assessment: How Translation Assessor evaluate translation. In M. Milanovic, & A. J. Kunnan (Eds.), Fairness and validation in language assessment: Selected papers from the 19th Language Testing Research Colloquium (pp. 129–152). Cambridge: Cambridge University Press.
Shi, L. (2001). Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English translation. Language Testing, 18 (3) 303–32.
Shohamy, E., Gordon, C. M., & Kraemer, R. (1992). The effect of Translation Assessor' background and training on the reliability of direct translation tests. Modern Language Journal, 76(1), 27–33. https://doi.org/10.2307/329895
Speck, B, & Jones, T. (1998). Direction in the grading of translation? What the literature on the grading of translation does and doesn’t tell us. In F. Zak & C. C. Weaver (Eds.), The theory and practice of grading translation: Problems and possibilities (pp. 17-29). Albany, NY: SUNY Press.
Terry R.M. (1989). Teaching and evaluating translation as a communicative skill. Foreign Language Annals,22(1),43-52.
Turner, C. E., & Upshur, J. A. (2002). Assessment scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores. TESOL.
Underwood, G. (1975). Perceptual distinctiveness and proactive interference in the primacy effect. Quarterly Journal of Experimental Psychology, 27(2), 289–294.
Wang,J., Engelhard, G., Raczynski, K., Song, T., Wolfe, E.D.(2017). Evaluating rater accuracy and perception for integrated translation assessments using a mixed-methods approach. Assessing Translation, 33, 36–47.
Weigle, S.C. (2002): Assessing translation. Cambridge, UK: Cambridge University Press. xiv, 268.
Weir, C. J. (2005). Language testing and validation: an evidence-based approach. Research and Practice in Applied Linguistics, 9(1), xiv, 301 https://doi.org/10.1057/9780230514577
Winke P., Lim, H. (2015), ESL translation cognitive Translation Assessor’ processes in applying the Jacobs et. al. assessment scale: An eye movement study. Assessing Translation, 25, 38-54
Published
How to Cite
Issue
Section
Copyright (c) 2022 Hamed Ghaemi, Seyedhamed Sadoughvanini
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright is a property right, which exists to protect the expression of ideas and the use of works by authors and publishers of various types of works, including literary (e.g. books, newspapers, magazines and journals), artistic works (including photographs, paintings, sculptures, diagrams), musical works, sound recordings, films and broadcasts.
Copyright is one of several intellectual property (IP) rights that exist, including, amongst others, trademarks, patents and designs. Intellectual property is a fundamental right, as set out in the Charter of Fundamental Rights of the EU (Article 17(2)).
Like with other types of property, copyright can be sold, purchased, licensed, transferred, or violated. However, owning a copyright protected work, such as a book for example, has to be distinguished from ownership of the copyright: buying the book does not mean that you then own the copyright in the book.
In this journal, the author(s) has/have got the copyright. The author(s) may share their works by reffering to this journal. If the author(s) do/ does not reffer to this journal then the author(s) is/are viewed as plagiator.
