Development and initial argument-based validation of a scoring rubric used in the assessment of L2 writing electronic portfolios

Abstract: Although writing electronic portfolios (ePortfolios) help learners communicate digitally and provide a platform for works to be better collected and presented, challenges are present in their assessment. This paper reports the development and validation of a writing ePortfolio scoring rubric for an action research course for L2 learners. Using Bachman’s (2005) ‘Assessment Use Argument’ as a basis, two main claims to support the validity of the rubric were examined: (1) the rubric is a reliable tool; and, (2) the rubric is relevant to the construct being measured. A mixed-method approach was used in the development and validation of the eight-criteria analytic and holistic evaluation scoring rubric. Thirteen raters evaluated thirty-eight ePortfolios in the study. The analyses of raters’ ratings using a many-facet Rasch measurement approach and raters’ individual standardised open-ended interviews indicated that overall the rubric’s analytic categories functioned appropriately to assess the intended construct. However, the criterion ‘Ease of Navigation’ was found to be misfitting due to differences in raters’ evaluation of the same ePortfolios. Overall, the argument-based validation indicated that the scoring rubric is a reliable and valid instrument for the purpose of assessing L2 writing ePortfolios in the context for which it was developed.