# !pip install anthropic --upgrade
Claude Vision
Claude 3 models can understand and analyze images, allowing for conversations that include both text and visuals.
Supported image formats include JPEG, PNG, GIF, and WebP. For optimal performance, resize images to no more than 1.15 megapixels and 1568 pixels in both dimensions.
You can include multiple images in a single request (up to 20 for API requests) for analysis
Each image counts towards your token usage. Approximate cost per image can be calculated using the formula: tokens = (width px * height px) / 750.
Claude has some limitations, such as identifying people, spatial reasoning, counting objects, and detecting AI-generated images. Always review and verify Claude’s interpretations.
Claude does not generate, produce, edit, manipulate or create images; it only interprets and analyzes them.
import os
from dotenv import load_dotenv
load_dotenv()
True
= os.getenv("ANTHROPIC_API_KEY") api_key
import anthropic
= anthropic.Anthropic(
client # defaults to os.environ.get("ANTHROPIC_API_KEY")
=api_key,
api_key )
= client.messages.create(
message ="claude-3-opus-20240229",
model=1000,
max_tokens=0.0,
temperature="You are an expert travel guide",
system=[
messages"role": "user", "content": "Top places to visit in Sydney"}
{
]
)
print(message.content)
[ContentBlock(text="Here are some of the top places to visit in Sydney, Australia:\n\n1. Sydney Opera House - This iconic building is a UNESCO World Heritage site and a must-see attraction.\n\n2. Sydney Harbour Bridge - Take a walk across the bridge or participate in a guided climb for stunning views of the city and harbor.\n\n3. Bondi Beach - One of Australia's most famous beaches, known for its golden sand, surfing, and vibrant beach culture.\n\n4. The Rocks - A historic neighborhood with cobblestone streets, museums, galleries, and markets.\n\n5. Darling Harbour - A waterfront precinct with restaurants, shops, and attractions like the SEA LIFE Sydney Aquarium and the Australian National Maritime Museum.\n\n6. Royal Botanic Garden Sydney - A beautiful oasis in the heart of the city, with diverse plant collections and stunning harbor views.\n\n7. Taronga Zoo - Located on the shores of Sydney Harbour, this zoo is home to a wide variety of native Australian and exotic animals.\n\n8. Sydney Tower Eye - The tallest structure in Sydney, offering 360-degree views of the city from the observation deck and revolving restaurant.\n\n9. Manly Beach - A popular beach located a short ferry ride from the city center, known for its laid-back atmosphere and water activities.\n\n10. Blue Mountains National Park - A scenic park located about 2 hours from Sydney, known for its dramatic rock formations, waterfalls, and hiking trails.\n\nThese are just a few of the many attractions Sydney has to offer, and there are plenty of other neighborhoods, beaches, and cultural experiences to explore in this vibrant city.", type='text')]
from IPython.display import display, HTML, Markdown
0].text)) display(Markdown(message.content[
Here are some of the top places to visit in Sydney, Australia:
Sydney Opera House - This iconic building is a UNESCO World Heritage site and a must-see attraction.
Sydney Harbour Bridge - Take a walk across the bridge or participate in a guided climb for stunning views of the city and harbor.
Bondi Beach - One of Australia’s most famous beaches, known for its golden sand, surfing, and vibrant beach culture.
The Rocks - A historic neighborhood with cobblestone streets, museums, galleries, and markets.
Darling Harbour - A waterfront precinct with restaurants, shops, and attractions like the SEA LIFE Sydney Aquarium and the Australian National Maritime Museum.
Royal Botanic Garden Sydney - A beautiful oasis in the heart of the city, with diverse plant collections and stunning harbor views.
Taronga Zoo - Located on the shores of Sydney Harbour, this zoo is home to a wide variety of native Australian and exotic animals.
Sydney Tower Eye - The tallest structure in Sydney, offering 360-degree views of the city from the observation deck and revolving restaurant.
Manly Beach - A popular beach located a short ferry ride from the city center, known for its laid-back atmosphere and water activities.
Blue Mountains National Park - A scenic park located about 2 hours from Sydney, known for its dramatic rock formations, waterfalls, and hiking trails.
These are just a few of the many attractions Sydney has to offer, and there are plenty of other neighborhoods, beaches, and cultural experiences to explore in this vibrant city.
from IPython.display import Image
'claude_results.jpeg', width=800) Image(
# load image and convert to base64
import base64
from PIL import Image
from io import BytesIO
= base64.b64encode(open('claude_results.jpeg', 'rb').read()).decode('utf-8') image_data
type(image_data)
str
100] image_data[:
'/9j/4AAQSkZJRgABAgEASABIAAD/4QDKRXhpZgAATU0AKgAAAAgABgESAAMAAAABAAEAAAEaAAUAAAABAAAAVgEbAAUAAAABAAAA'
Describe an Image
= client.messages.create(
message ="claude-3-opus-20240229",
model=1024,
max_tokens=[
messages
{"role": "user",
"content": [
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": image_data,
},
},
{"type": "text",
"text": "Describe this image."
}
],
}
],
)print(message)
Message(id='msg_01Xd14cQdncxYH1Cqt7ULGte', content=[ContentBlock(text='This image shows a comparison table of performance metrics across different AI language models.\n\nThe models compared include Claude 3 (in its Opus, Sonnet and Haiku variants), GPT-4, GPT-3.5, and two versions of Gemini 1.0.\n\nVarious metrics are listed down the left side, including:\n- Undergraduate level knowledge (MMLU)\n- Graduate level reasoning (GQPA, Diamond)\n- Grade school math (GSM8K) \n- Math problem-solving (MATH)\n- Multilingual math (MGSM)\n- Code generation (HumanEval)\n- Reasoning over text (DROP, F1 score) \n- Mixed evaluations (BIG-Bench-Hard)\n- Knowledge Q&A (ARC-Challenge)\n- Common Knowledge (HellaSwag)\n\nThe cells contain the score each model achieved on the corresponding metric. The score format varies by row, with some showing x-shot scores, others showing raw scores, F1 scores, or number of shots.\n\nOverall, this table enables a side-by-side comparison of how well these prominent language models perform across a diverse set of NLP benchmarks and capability assessments. The color coding also helps highlight relative strengths and weaknesses.', type='text')], model='claude-3-opus-20240229', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(input_tokens=1640, output_tokens=291))
0].text)) display(Markdown(message.content[
This image shows a comparison table of various metrics across different AI language models, including Claude 3 (in Opus, Sonnet and Haiku variants), GPT-4, GPT-3.5, and two versions of Gemini.
The metrics compared include: - Undergraduate level knowledge (MMLU) - Graduate level reasoning (GPQA, Diamond) - Grade school math (GSM8K) - Math problem-solving (MATH) - Multilingual math (MGSM) - Code generation (HumanEval) - Reasoning over text (DROP, F1 score) - Mixed evaluations (BIG-Bench-Hard) - Knowledge Q&A (ARC-Challenge) - Common Knowledge (HellaSwag)
For each metric, the scores are provided along with the number of “shots” used in the evaluation (e.g. 0-shot, 5-shot, 10-shot etc.). The scores are given as percentages, except for the DROP reasoning metric which uses an F1 score.
Overall, the table allows comparing the performance of these different AI models across a range of knowledge domains and reasoning tasks. The color coding appears to highlight the best performing model for each metric.
Generate Code
= client.messages.create(
message ="claude-3-opus-20240229",
model=1024,
max_tokens=[
messages
{"role": "user",
"content": [
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": image_data,
},
},
{"type": "text",
"text": "Write python code to create a bar plot from the image table. Only Claude columns, dont include other models. x label is all the metrics, y label is the values."
}
],
}
],
)print(message)
Message(id='msg_01THwqDdZn2oWfVF2XNu45w3', content=[ContentBlock(text="Here's the Python code to create a bar plot from the image table, including only the Claude columns:\n\n```python\nimport matplotlib.pyplot as plt\n\nmetrics = ['Undergraduate level knowledge MMLU',\n 'Graduate level reasoning GFQA, Diamond',\n 'Grade school math GSMSB',\n 'Math problem-solving MATH',\n 'Multilingual math MGSM',\n 'Code HumanEval',\n 'Reasoning over text DROP, F1 score',\n 'Mixed evaluations BIG-Bench-Hard',\n 'Knowledge Q&A ARC-Challenge',\n 'Common Knowledge HellaSwag']\n\nclaude_3_opus = [86.8, 50.4, 95.0, 60.1, 90.7, 84.9, 83.1, 86.8, 96.4, 95.4]\nclaude_3_sonnet = [79.0, 40.4, 92.3, 43.1, 83.5, 73.0, 78.9, 82.9, 93.2, 89.0]\nclaude_3_haiku = [75.2, 33.3, 88.9, 38.9, 75.1, 75.9, 78.4, 73.7, 89.2, 85.9]\n\nx = range(len(metrics))\nwidth = 0.25\n\nfig, ax = plt.subplots(figsize=(12, 8))\n\nax.bar(x, claude_3_opus, width, label='Claude 3 Opus')\nax.bar([i + width for i in x], claude_3_sonnet, width, label='Claude 3 Sonnet')\nax.bar([i + 2*width for i in x], claude_3_haiku, width, label='Claude 3 Haiku')\n\nax.set_xlabel('Metrics')\nax.set_ylabel('Values')\nax.set_xticks([i + width for i in x])\nax.set_xticklabels(metrics, rotation=45, ha='right')\nax.legend()\n\nplt.tight_layout()\nplt.show()\n```\n\nThis code does the following:\n\n1. It imports the necessary matplotlib.pyplot module.\n\n2. It defines the metrics as a list of strings.\n\n3. It extracts the values for each Claude model (Opus, Sonnet, and Haiku) into separate lists.\n\n4. It sets up the x-coordinates and width for the bars.\n\n5. It creates a figure and an axis using `plt.subplots()`.\n\n6. It creates the bars for each Claude model using `ax.bar()`, with appropriate offsets for the x-coordinates.\n\n7. It sets the x-label and y-label using `ax.set_xlabel()` and `ax.set_ylabel()`, respectively.\n\n8. It sets the x-tick positions and labels using `ax.set_xticks()` and `ax.set_xticklabels()`, with rotation and horizontal alignment for better readability.\n\n9. It adds a legend using `ax.legend()`.\n\n10. It adjusts the layout using `plt.tight_layout()` to ensure the labels and titles fit nicely.\n\n11. Finally, it displays the plot using `plt.show()`.\n\nWhen you run this code, it will generate a bar plot comparing the performance of the Claude models (Opus, Sonnet, and Haiku) across various metrics.", type='text')], model='claude-3-opus-20240229', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(input_tokens=1670, output_tokens=825))
0].text)) display(Markdown(message.content[
Here’s the Python code to create a bar plot from the image table, including only the Claude columns:
import matplotlib.pyplot as plt
= ['Undergraduate level knowledge MMLU',
metrics 'Graduate level reasoning GFQA, Diamond',
'Grade school math GSMSB',
'Math problem-solving MATH',
'Multilingual math MGSM',
'Code HumanEval',
'Reasoning over text DROP, F1 score',
'Mixed evaluations BIG-Bench-Hard',
'Knowledge Q&A ARC-Challenge',
'Common Knowledge HellaSwag']
= [86.8, 50.4, 95.0, 60.1, 90.7, 84.9, 83.1, 86.8, 96.4, 95.4]
claude_3_opus = [79.0, 40.4, 92.3, 43.1, 83.5, 73.0, 78.9, 82.9, 93.2, 89.0]
claude_3_sonnet = [75.2, 33.3, 88.9, 38.9, 75.1, 75.9, 78.4, 73.7, 89.2, 85.9]
claude_3_haiku
= range(len(metrics))
x = 0.25
width
= plt.subplots(figsize=(12, 8))
fig, ax
='Claude 3 Opus')
ax.bar(x, claude_3_opus, width, label+ width for i in x], claude_3_sonnet, width, label='Claude 3 Sonnet')
ax.bar([i + 2*width for i in x], claude_3_haiku, width, label='Claude 3 Haiku')
ax.bar([i
'Metrics')
ax.set_xlabel('Values')
ax.set_ylabel(+ width for i in x])
ax.set_xticks([i =45, ha='right')
ax.set_xticklabels(metrics, rotation
ax.legend()
plt.tight_layout() plt.show()
This code does the following:
It imports the necessary matplotlib.pyplot module.
It defines the metrics as a list of strings.
It extracts the values for each Claude model (Opus, Sonnet, and Haiku) into separate lists.
It sets up the x-coordinates and width for the bars.
It creates a figure and an axis using
plt.subplots()
.It creates the bars for each Claude model using
ax.bar()
, with appropriate offsets for the x-coordinates.It sets the x-label and y-label using
ax.set_xlabel()
andax.set_ylabel()
, respectively.It sets the x-tick positions and labels using
ax.set_xticks()
andax.set_xticklabels()
, with rotation and horizontal alignment for better readability.It adds a legend using
ax.legend()
.It adjusts the layout using
plt.tight_layout()
to ensure the labels and titles fit nicely.Finally, it displays the plot using
plt.show()
.
When you run this code, it will generate a bar plot comparing the performance of the Claude models (Opus, Sonnet, and Haiku) across various metrics.
import matplotlib.pyplot as plt
= ['Undergraduate level knowledge MMLU',
metrics 'Graduate level reasoning GPQA, Diamond',
'Grade school math GSMSK',
'Math problem-solving MATH',
'Multilingual math MGSM',
'Code HumanEval',
'Reasoning over text DROP, F1 score',
'Mixed evaluations BIG-Bench-Hard',
'Knowledge Q&A ARC-Challenge',
'Common Knowledge HellaSwag']
= [86.8, 50.4, 95.0, 60.1, 90.7, 84.9, 83.1, 86.8, 96.4, 95.4]
claude_3_opus = [79.0, 40.4, 92.3, 43.1, 83.5, 73.0, 78.9, 82.9, 93.2, 89.0]
claude_3_sonnet = [75.2, 33.3, 88.9, 38.9, 75.1, 75.9, 78.4, 73.7, 89.2, 85.9]
claude_3_haiku
= range(len(metrics))
x = 0.25
width
= plt.subplots(figsize=(12, 8))
fig, ax ='Claude 3 Opus')
ax.bar(x, claude_3_opus, width, label+ width for i in x], claude_3_sonnet, width, label='Claude 3 Sonnet')
ax.bar([i + width*2 for i in x], claude_3_haiku, width, label='Claude 3 Haiku')
ax.bar([i
'Values')
ax.set_ylabel(+ width for i in x])
ax.set_xticks([i =45, ha='right')
ax.set_xticklabels(metrics, rotation
ax.legend()
plt.tight_layout() plt.show()
Recipe Generation
= base64.b64encode(open('recipe.jpg', 'rb').read()).decode('utf-8') recipe_data
from IPython.display import Image
#display the image
'recipe.jpg', width=800) Image(
Prompt I used to generate the image is ‘A single perfect éclair, dark chocolate glaze, a single raspberry on top, cracked surface revealing vanilla custard’
= client.messages.create(
message ="claude-3-opus-20240229",
model=1024,
max_tokens=[
messages
{"role": "user",
"content": [
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": recipe_data,
},
},
{"type": "text",
"text": "Generate recipe for the given image."
}
],
}
],
)print(message)
Message(id='msg_01L7JH75i2MPegcQa4b597DS', content=[ContentBlock(text='This image shows an appetizing raspberry chocolate eclair or cream puff dessert. The eclair pastry is golden brown and flaky looking, filled with a luscious vanilla cream filling, and topped with a rich chocolate ganache glaze and fresh raspberries dusted with cocoa powder.\n\nTo make raspberry chocolate eclairs, you would need to:\n\n1. Prepare choux pastry dough, pipe it into eclair shapes, and bake until puffed and golden.\n\n2. Make a vanilla pastry cream filling, let it chill, then pipe it into the cooled eclair shells. \n\n3. Prepare a shiny chocolate ganache glaze and dip or spread it on top of the filled eclairs.\n\n4. Top with fresh raspberries and dust with cocoa powder for the finishing touch. \n\n5. Chill the assembled eclairs for a couple hours to allow flavors to meld before serving.\n\nThe combination of crisp pastry, silky cream, bittersweet chocolate and tart-sweet raspberries makes for an irresistible treat that is sure to impress. While eclairs take some time and technique, the end result of this classic French patisserie is well worth the effort for a special occasion dessert.', type='text')], model='claude-3-opus-20240229', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(input_tokens=1648, output_tokens=279))
0].text)) display(Markdown(message.content[
This image shows an appetizing raspberry chocolate eclair or cream puff dessert. The eclair pastry is golden brown and flaky looking, filled with a luscious vanilla cream filling, and topped with a rich chocolate ganache glaze and fresh raspberries dusted with cocoa powder.
To make raspberry chocolate eclairs, you would need to:
Prepare choux pastry dough, pipe it into eclair shapes, and bake until puffed and golden.
Make a vanilla pastry cream filling, let it chill, then pipe it into the cooled eclair shells.
Prepare a shiny chocolate ganache glaze and dip or spread it on top of the filled eclairs.
Top with fresh raspberries and dust with cocoa powder for the finishing touch.
Chill the assembled eclairs for a couple hours to allow flavors to meld before serving.
The combination of crisp pastry, silky cream, bittersweet chocolate and tart-sweet raspberries makes for an irresistible treat that is sure to impress. While eclairs take some time and technique, the end result of this classic French patisserie is well worth the effort for a special occasion dessert.
Image with system prompt
= client.messages.create(
message ="claude-3-opus-20240229",
model=1024,
max_tokens="You are an expert chef, who can generate recipes from images in french.",
system=[
messages
{"role": "user",
"content": [
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": recipe_data,
},
},
{"type": "text",
"text": "Generate recipe for the given image."
}
],
}
],
)print(message)
Message(id='msg_01QGkczZma6tD84ZQoNccyzW', content=[ContentBlock(text="Voici une recette pour un éclair au chocolat et aux framboises, inspirée de l'image :\n\nPâte à choux :\n- 125 ml d'eau\n- 50 g de beurre\n- 65 g de farine \n- 2 œufs\n- 1 pincée de sel\n\nCrème pâtissière à la vanille :\n- 250 ml de lait\n- 50 g de sucre\n- 30 g de farine\n- 2 jaunes d'œufs \n- 1 gousse de vanille\n\nGanache au chocolat :\n- 100 g de chocolat noir\n- 100 ml de crème fraîche liquide\n\nDécoration :\n- Cacao en poudre\n- Framboises fraîches\n\nInstructions :\n\n1. Préparez la pâte à choux en portant à ébullition l'eau, le beurre et le sel. Hors du feu, incorporez la farine d'un coup puis remettez sur feu doux en mélangeant jusqu'à ce que la pâte se décolle. Laissez tiédir puis incorporez les œufs un par un. Pochez sur une plaque et faites cuire 30 min à 180°C.\n\n2. Pour la crème, portez le lait et la vanille à ébullition. À part, mélangez sucre, farine et jaunes d'œufs. Versez le lait dessus en mélangeant, puis remettez sur feu doux jusqu'à épaississement. Laissez refroidir.\n\n3. Préparez la ganache en faisant fondre le chocolat avec la crème chauffée.\n\n4. Fourrez les éclairs refroidis de crème pâtissière, glacez-les de ganache, saupoudrez de cacao et décorez de framboises fraîches. \n\nDégustez ces délicieux éclairs faits maison, alliant le croquant de la pâte, la douceur vanillée de la crème et l'intense chocolat, rehaussés de framboises acidulées.", type='text')], model='claude-3-opus-20240229', role='assistant', stop_reason='end_turn', stop_sequence=None, type='message', usage=Usage(input_tokens=1664, output_tokens=543))
0].text)) display(Markdown(message.content[
Voici une recette pour un éclair au chocolat et aux framboises, inspirée de l’image :
Pâte à choux : - 125 ml d’eau - 50 g de beurre - 65 g de farine - 2 œufs - 1 pincée de sel
Crème pâtissière à la vanille : - 250 ml de lait - 50 g de sucre - 30 g de farine - 2 jaunes d’œufs - 1 gousse de vanille
Ganache au chocolat : - 100 g de chocolat noir - 100 ml de crème fraîche liquide
Décoration : - Cacao en poudre - Framboises fraîches
Instructions :
Préparez la pâte à choux en portant à ébullition l’eau, le beurre et le sel. Hors du feu, incorporez la farine d’un coup puis remettez sur feu doux en mélangeant jusqu’à ce que la pâte se décolle. Laissez tiédir puis incorporez les œufs un par un. Pochez sur une plaque et faites cuire 30 min à 180°C.
Pour la crème, portez le lait et la vanille à ébullition. À part, mélangez sucre, farine et jaunes d’œufs. Versez le lait dessus en mélangeant, puis remettez sur feu doux jusqu’à épaississement. Laissez refroidir.
Préparez la ganache en faisant fondre le chocolat avec la crème chauffée.
Fourrez les éclairs refroidis de crème pâtissière, glacez-les de ganache, saupoudrez de cacao et décorez de framboises fraîches.
Dégustez ces délicieux éclairs faits maison, alliant le croquant de la pâte, la douceur vanillée de la crème et l’intense chocolat, rehaussés de framboises acidulées.
Multiple Images: Getting Feedback on What Shoe to Wear for a Blue Suit
= base64.b64encode(open('suit.jpeg', 'rb').read()).decode('utf-8')
blue_suit = base64.b64encode(open('black_shoe.jpeg', 'rb').read()).decode('utf-8')
black_shoes = base64.b64encode(open('brown_shoe.jpeg', 'rb').read()).decode('utf-8') brown_shoes
'suit.jpeg', width=500) Image(
'black_shoe.jpeg', width=500) Image(
'brown_shoe.jpeg', width=500) Image(
= base64.b64encode(open('suit.jpeg', 'rb').read()).decode('utf-8')
suit_data = base64.b64encode(open('black_shoe.jpeg', 'rb').read()).decode('utf-8')
black_shoe_data = base64.b64encode(open('brown_shoe.jpeg', 'rb').read()).decode('utf-8') brown_shoe_data
= client.messages.create(
message ="claude-3-opus-20240229",
model=1024,
max_tokens="You are a fashion expert who has excellent knowledge of color combinations and can recommend the best color shoe for the given suit.",
system=[
messages
{"role": "user",
"content": [
{"type": "text",
"text": "Suit:"
},
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": suit_data,
},
},
{"type": "text",
"text": "Shoe 1:"
},
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": black_shoe_data,
},
},
{"type": "text",
"text": "Shoe 2:"
},
{"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": brown_shoe_data,
},
},
{"type": "text",
"text": "For my suit, Look into my suit color, and recommend what color shoe should I wear? Black or brown? Give reasons."
}
],
}
], )
0].text)) display(Markdown(message.content[
Based on the blue plaid suit shown in the image, I would recommend pairing it with the brown leather dress shoes (Shoe 2) rather than the black ones (Shoe 1). Here are a few reasons why:
Brown shoes complement blue suits very well, creating a harmonious and stylish color combination. The warm tones of the brown leather pair nicely with the cool blue hues of the suit fabric.
Wearing brown shoes with a blue suit is a classic and versatile look that can work for both business and more casual occasions. It’s not quite as formal as a black shoe, but still polished and put-together.
The rich, burnished leather of the brown shoes in the photo would add some nice visual texture and depth to offset the plaid pattern of the suit. The broguing details on the shoes also provide an extra touch of style.
Black shoes, while always appropriate with a navy solid suit, can sometimes look a bit stark or severe with lighter shades of blue or bolder patterns like the plaid shown here. The brown option allows the suit to remain the focal point.
So in summary, while you can’t go wrong with either, I’d suggest the brown dress shoes as the optimal pairing to elevate this blue plaid suit. The combination strikes the right balance of dapper and distinctive.