uu.seUppsala University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Samstämmighet i skrivbedömning: Statistisk analys vid bedömning av två nationella skrivprov
Uppsala University, Disciplinary Domain of Humanities and Social Sciences, Faculty of Languages, Department of Scandinavian Languages. Nationella prov i svenska och svenska som andraspråk.ORCID iD: 0000-0002-0438-7463
2019 (Swedish)Report (Other academic)Alternative title
Rater agreement in the assessment of writing : Statistical analysis of the rating of two national tests (English)
Abstract [en]

Rater agreement in the assessment of writing. Statistical analysis of the rating of two national tests. This report presents a statistical analysis of assessment in national tests. It analyses rater agreement in the assessment of student texts written within the framework of two different course tests in Swedish at upper secondary school; the national test in Swedish 1 and the national test in Swedish 3. The purpose of the analysis is to discover the level of agreement between different trained raters, given the current design of the test. The paper aims to predict expected levels of reliability, given alternative assessment circumstances such as number of raters involved. Rater agreement is observed using a set of measures based on different assumptions about what might be expected regarding the ability of the raters to either be consistent or reach consensus. The statistical measures include exact agreement, Cohen’s and Fleiss’ κ statistics, intraclass correlation coefficient, many-facets Rasch measurement and generalizability coefficients. The statistical analyses based on the different approaches provide an overall picture of the reliability of ratings of student texts written within the framework of national tests in Swedish 1 and Swedish 3. In conclusion, the results suggest that the ratings meet conventionally acceptable levels of reliability reported in the literature. In addition, the results show that increasing the number of raters, either through calibrating the ratings of multiple individual raters or through social moderation in groups of raters, increases the reliability of the ratings. Although this second conclusion may seem self-evident, studies with a design that is sufficiently statistically sound to merit such conclusions are rare. The present report contributes to such a foundation.

Place, publisher, year, edition, pages
Uppsala, 2019. , p. 49
Series
Svenska i utveckling, ISSN 1400-7436 ; 36
Keywords [sv]
bedömning, nationella prov, reliabilitet
National Category
Humanities and the Arts
Research subject
Scandinavian Languages; Statistics; Psychology
Identifiers
URN: urn:nbn:se:uu:diva-396603OAI: oai:DiVA.org:uu-396603DiVA, id: diva2:1368421
Available from: 2019-11-07 Created: 2019-11-07 Last updated: 2019-11-07

Open Access in DiVA

No full text in DiVA

Search in DiVA

By author/editor
Dalberg, Tobias
By organisation
Department of Scandinavian Languages
Humanities and the Arts

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 21 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf