Vocabulary is a core component of language proficiency (Nation, 2020) and texts containing an appropriate and precise vocabulary are considered to be of high quality by assessors (e.g., Garner et al., 2019; Vögelin et al., 2019). Despite regularly being claimed as an explicit grading criterion in language tests, what constitutes sufficient/good vocabulary is often expressed in vague terms in guidelines to assessors (e.g., ETS, 2022). Every year in Sweden, upper secondary school students are required to take the national tests of English to ensure that their proficiency is on par with the level at which they study (Olsson, 2018). During the tests, students are required to write texts on a specific topic and these are then assessed by teachers with instructions of assessment created by a group of experts on behalf of the Swedish National Agency for Education. These instructions specify vocabulary as a grading criterion and indicate that there should be a progression in terms of lexical complexity between the lowest and highest grade. Furthermore, to ensure fair and equal assessment, the instructions provide graded example texts to assist teachers’ assessment. This paper aims to investigate the assessment of written productive vocabulary by analysing these graded example texts and texts graded by teachers during the exams. The material consists of a corpus of 142 graded example texts and 190 teacher graded texts from two courses written between 2011 and 2022. A range of measures of lexical sophistication (e.g., frequency, range, n-grams) were employed utilising different written subsections of the Corpus of Contemporary American English (COCA) as reference corpora. Also, one measure of lexical diversity was used (Moving Average Type-Token Ratio). The results indicate that there is no structured difference in terms of lexical complexity between texts awarded the lowest and highest grade, suggesting that written vocabulary proficiency was largely overlooked in the assessment. In addition, the findings call into question the construction and validity of the Swedish national tests of English since productive vocabulary, although an essential part of overall language competence and explicitly mentioned a grading criterion, does not seem to have been taken into consideration when the example texts were graded.