ROUGE

The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metric used to evaluate the quality of generated text compared to a reference text. It does so by computing the overlap between two texts, for which a subsequent precision and recall value can be computed. The ROUGE score is often used in the context of generative tasks such as text summarization and machine translation.

A major difference with Perplexity comes from the fact that ROUGE evaluates actual text, whereas Perplexity evaluates logits.

Here’s a hypothetical Python example demonstrating the usage of unigram ROUGE F-score to evaluate a generative language model:

from torchmetrics.text import ROUGEScore
from transformers import AutoTokenizer, pipeline

pipe = pipeline("text-generation", model="openai-community/gpt2")
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")

/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(

Define the prompt and target texts

prompt = "The quick brown fox"
target_text = "The quick brown fox jumps over the lazy dog."

Generate a sample text using the GPT-2 model

sample_text = pipe(prompt, max_length=20, do_sample=True, temperature=0.1, pad_token_id=tokenizer.eos_token_id)[0][
    "generated_text"
]
print(sample_text)

The quick brown foxes are the most common species of foxes in the United States. They are

Calculate the ROUGE of the generated text

rouge = ROUGEScore()
rouge(preds=[sample_text], target=[target_text])

{'rouge1_fmeasure': tensor(0.3077), 'rouge1_precision': tensor(0.2353), 'rouge1_recall': tensor(0.4444), 'rouge2_fmeasure': tensor(0.1667), 'rouge2_precision': tensor(0.1250), 'rouge2_recall': tensor(0.2500), 'rougeL_fmeasure': tensor(0.3077), 'rougeL_precision': tensor(0.2353), 'rougeL_recall': tensor(0.4444), 'rougeLsum_fmeasure': tensor(0.3077), 'rougeLsum_precision': tensor(0.2353), 'rougeLsum_recall': tensor(0.4444)}

By default, the ROUGE score is calculated using a whitespace tokenizer. You can also calculate the ROUGE for the tokens directly:

token_rouge = ROUGEScore(tokenizer=lambda text: tokenizer.tokenize(text))
token_rouge(preds=[sample_text], target=[target_text])

{'rouge1_fmeasure': tensor(0.3448), 'rouge1_precision': tensor(0.2632), 'rouge1_recall': tensor(0.5000), 'rouge2_fmeasure': tensor(0.2222), 'rouge2_precision': tensor(0.1667), 'rouge2_recall': tensor(0.3333), 'rougeL_fmeasure': tensor(0.3448), 'rougeL_precision': tensor(0.2632), 'rougeL_recall': tensor(0.5000), 'rougeLsum_fmeasure': tensor(0.4000), 'rougeLsum_precision': tensor(0.3000), 'rougeLsum_recall': tensor(0.6000)}

Since ROUGE is a text-based metric, it can be used to benchmark decoding strategies. For example, you can compare temperature settings:

import matplotlib.pyplot as plt  # noqa: E402

temperatures = [x * 0.1 for x in range(1, 10)]  # Generate temperature values from 0 to 1 with a step of 0.1
n_samples = 100  # Note that a real benchmark typically requires more data

average_scores = []

for temperature in temperatures:
    sample_text = pipe(
        prompt, max_length=20, do_sample=True, temperature=temperature, pad_token_id=tokenizer.eos_token_id
    )[0]["generated_text"]
    scores = [rouge(preds=[sample_text], target=[target_text])["rouge1_fmeasure"] for _ in range(n_samples)]
    average_scores.append(sum(scores) / n_samples)

# Plot the average ROUGE score for each temperature
plt.plot(temperatures, average_scores)
plt.xlabel("Generation temperature")
plt.ylabel("Average unigram ROUGE F-Score")
plt.title("ROUGE for varying temperature settings")
plt.show()