Vision / Visual Counting

Fruit Counting From Image

In the image task, the model counts bananas, apples, and oranges, even when some fruit overlap. It has to identify each fruit type and count carefully instead of giving a rough description.

Prompt

How many fruits of each type are in the image?

Bananas, apples, and oranges arranged on a wooden table.
Max tokens
100K
temperature
0
top_p
1
seed
42
presence_penalty
0
frequency_penalty
0
Reasoning effort
High
Execution
Single-shot via API

Fortytwo Prime

Fortytwo

PASS4 / 4

Based on the image, here is the count of each fruit type:
• Apples: 8
• Bananas: 7 (3 in the bunch on the left, 4 in the bunch on the right)
• Oranges: 5
Total: 20 fruits

Fortytwo identifies apples, bananas, and oranges, including the two banana bunches, and returns the exact answer-key counts.

vs
Qwen 3.7 PlusAlibaba
Kimi K2.7 CodeMoonshotPASSGemini 3.5 FlashGooglePASSChatGPT 5.5OpenAIPASSGemini 3.1 ProGooglePASSKimi K2.5MoonshotPASSClaude Fable 5AnthropicMIXEDQwen 3.7 PlusAlibabaMIXEDMiniMax M3MiniMaxMIXEDGrok 4.3xAIMIXEDClaude Opus 4.7AnthropicMIXEDGemma 4GoogleMIXEDQwen 3.6 PlusAlibabaMIXEDChatGPT 5.4OpenAIMIXEDClaude Opus 4.8AnthropicMIXEDGrok 4.20xAIFAILClaude Opus 4.6AnthropicFAILOpenRouter Fusion (Quality)OpenRouterUNSUPPORTEDOpenRouter Fusion (Budget)OpenRouterUNSUPPORTEDNemotron 3 UltraNVIDIAUNSUPPORTEDDeepSeek V4 ProDeepSeekUNSUPPORTEDDeepSeek V4 FlashDeepSeekUNSUPPORTEDGLM 5.1ZhipuUNSUPPORTEDMiniMax M2.7MiniMaxUNSUPPORTEDDeepSeek V3.2DeepSeekUNSUPPORTED
MIXED3 / 4

Based on the image, hereis the count for each type of fruit:

  • Apples: 8
  • Bananas: 6 (3 in the left bunch, 3 in the right bunch)
  • Oranges: 5 (including the one with leaves)

Qwen 3.7 Plus returns counts for apples, bananas, and oranges and matches the answer key on 8 apples and 5 oranges, but counts only 6 bananas instead of 7.

Model verdicts

ModelVerdictIdentified Fruit TypesCounted ApplesCounted OrangesCounted Bananas
1Fortytwo PrimePASSpasspasspasspass2Kimi K2.7 CodePASSpasspasspasspass3Gemini 3.5 FlashPASSpasspasspasspass4ChatGPT 5.5PASSpasspasspasspass5Gemini 3.1 ProPASSpasspasspasspass6Kimi K2.5PASSpasspasspasspass7Claude Fable 5MIXEDpasspasspassfail8Qwen 3.7 PlusMIXEDpasspasspassfail9MiniMax M3MIXEDpasspasspassfail10Grok 4.3MIXEDpasspassfailpass11Claude Opus 4.7MIXEDpasspasspassfail12Gemma 4MIXEDpasspasspassfail13Qwen 3.6 PlusMIXEDpasspasspassfail14ChatGPT 5.4MIXEDpasspasspassfail15Claude Opus 4.8MIXEDpasspassfailfail16Grok 4.20FAILpassfailfailfail17Claude Opus 4.6FAILpassfailfailfail18OpenRouter Fusion (Quality)UNSUPPORTEDunsupportedunsupportedunsupportedunsupported19OpenRouter Fusion (Budget)UNSUPPORTEDunsupportedunsupportedunsupportedunsupported20Nemotron 3 UltraUNSUPPORTEDunsupportedunsupportedunsupportedunsupported21DeepSeek V4 ProUNSUPPORTEDunsupportedunsupportedunsupportedunsupported22DeepSeek V4 FlashUNSUPPORTEDunsupportedunsupportedunsupportedunsupported23GLM 5.1UNSUPPORTEDunsupportedunsupportedunsupportedunsupported24MiniMax M2.7UNSUPPORTEDunsupportedunsupportedunsupportedunsupported25DeepSeek V3.2UNSUPPORTEDunsupportedunsupportedunsupportedunsupported

Have a complex task to evaluate?

Request a custom evaluation for your use case.

Request a demo →