Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Google Research.
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Internet Systems Lab. Masakhane, Africa.ORCID iD: 0000-0002-5582-2031
IIIT Delhi, India.
IIIT Hyderabad, India.
Show others and affiliations
2021 (English)In: The 1st Workshop on Natural Language Generation, Evaluation, and Metrics: Proceedings of the Workshop, Association for Computational Linguistics, 2021, p. 96-120, article id 2021.gem-1.10Conference paper, Published paper (Refereed)
Abstract [en]

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.

Place, publisher, year, edition, pages
Association for Computational Linguistics, 2021. p. 96-120, article id 2021.gem-1.10
National Category
Natural Language Processing
Research subject
Machine Learning
Identifiers
URN: urn:nbn:se:ltu:diva-87438DOI: 10.18653/v1/2021.gem-1.10ISI: 000697564200010Scopus ID: 2-s2.0-85121350997OAI: oai:DiVA.org:ltu-87438DiVA, id: diva2:1601443
Conference
1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), Bangkok, Thailand (online), August 5-6, 2021
Note

ISBN för värdpublikation: 978-1-954085-67-1

Available from: 2021-10-08 Created: 2021-10-08 Last updated: 2025-02-07Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Adewumi, Tosin

Search in DiVA

By author/editor
Adewumi, Tosin
By organisation
Embedded Internet Systems Lab
Natural Language Processing

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 133 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf