.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "gallery/image/clip_score.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_gallery_image_clip_score.py: CLIPScore =============================== The CLIPScore is a model-based image captioning metric that correlates well with human judgments. The benefit of CLIPScore is that it does not require reference captions for evaluation. .. GENERATED FROM PYTHON SOURCE LINES 11-12 Here's a hypothetical Python example demonstrating the usage of the CLIPScore metric to evaluate image captions: .. GENERATED FROM PYTHON SOURCE LINES 12-20 .. code-block:: Python :lineno-start: 12 import matplotlib.animation as animation import matplotlib.pyplot as plt import numpy as np import torch from matplotlib.table import Table from skimage.data import astronaut, cat, coffee from torchmetrics.multimodal import CLIPScore .. GENERATED FROM PYTHON SOURCE LINES 21-22 Get sample images .. GENERATED FROM PYTHON SOURCE LINES 22-29 .. code-block:: Python :lineno-start: 23 images = { "astronaut": astronaut(), "cat": cat(), "coffee": coffee(), } .. GENERATED FROM PYTHON SOURCE LINES 30-31 Define a hypothetical captions for the images .. GENERATED FROM PYTHON SOURCE LINES 31-38 .. code-block:: Python :lineno-start: 32 captions = [ "A photo of an astronaut.", "A photo of a cat.", "A photo of a cup of coffee.", ] .. GENERATED FROM PYTHON SOURCE LINES 39-40 Define the models for CLIPScore .. GENERATED FROM PYTHON SOURCE LINES 40-48 .. code-block:: Python :lineno-start: 41 models = [ "openai/clip-vit-base-patch16", # "openai/clip-vit-base-patch32", # "openai/clip-vit-large-patch14-336", "openai/clip-vit-large-patch14", ] .. GENERATED FROM PYTHON SOURCE LINES 49-50 Collect scores for each image-caption pair .. GENERATED FROM PYTHON SOURCE LINES 50-59 .. code-block:: Python :lineno-start: 51 score_results = [] for model in models: clip_score = CLIPScore(model_name_or_path=model) for key, img in images.items(): img_tensor = torch.tensor(np.array(img)) caption_scores = {caption: clip_score(img_tensor, caption) for caption in captions} score_results.append({"scores": caption_scores, "image": key, "model": model}) .. GENERATED FROM PYTHON SOURCE LINES 60-61 Create an animation to display the scores .. GENERATED FROM PYTHON SOURCE LINES 61-91 .. code-block:: Python :lineno-start: 62 fig, (ax_img, ax_table) = plt.subplots(1, 2, figsize=(10, 5)) def update(num: int) -> tuple: """Update the image and table with the scores for the given model.""" results = score_results[num] scores, image, model = results["scores"], results["image"], results["model"] fig.suptitle(f"Model: {model.split('/')[-1]}", fontsize=16, fontweight="bold") # Update image ax_img.imshow(images[image]) ax_img.axis("off") # Update table table = Table(ax_table, bbox=[0, 0, 1, 1]) header1 = table.add_cell(0, 0, text="Caption", width=3, height=1) header2 = table.add_cell(0, 1, text="Score", width=1, height=1) header1.get_text().set_weight("bold") header2.get_text().set_weight("bold") for i, (caption, score) in enumerate(scores.items()): table.add_cell(i + 1, 0, text=caption, width=3, height=1) table.add_cell(i + 1, 1, text=f"{score:.2f}", width=1, height=1) ax_table.add_table(table) ax_table.axis("off") return ax_img, ax_table ani = animation.FuncAnimation(fig, update, frames=len(score_results), interval=3000) .. container:: sphx-glr-animation .. raw:: html
.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 42.536 seconds) .. _sphx_glr_download_gallery_image_clip_score.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: clip_score.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: clip_score.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: clip_score.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_