I did a quick test with some small(-ish) open-source models to check their tool-calling capabilities.
It was totally unscientific - I used a single input containing multiple questions that should trigger several tool calls (retrieval and get_waste_collection), as we use at ZüriCityGPT. I ran the tests on my Mac using Ollama with the default quantizations (typically Q4).
The question was:
Wann ist Papiersammlung in Wiedikon? Und wann karton an der Quellenstrasse? Wer ist die Stadtpräsidentin?
This should answer with 3 tool calls. Two for waste collection, and one for retrieval.
TL;DR: Models larger than 12B seem to handle it well. The others? Not so much. An exception was qwen2.5:1.5b - oddly, it worked (unlike the 3B and 7B versions, which failed due to a wrong type… just a fluke?). Also, phi3-mini:3.8b gave the correct answer but in the wrong format. Maybe better prompting could fix that?
mistral:7b ❌
mistral-nemo:12b ✅
mistral-small3.1:24b ✅
qwen2.5:0.5b ❌
qwen2.5:1.5b ✅
qwen2.5:3b ❌
qwen2.5:7b ✅❌(almost, wrong type for karton)
qwen2.5:14b ✅
cogito:3b ❌
cogito:8b ❌
cogito:14b ✅ ❌(almost, missed retrieval tool)
cogito:32b ✅
granite3.2:2b ❌
granite3.2:8b ❌
phi4-mini:3.8b ✅ (correct answer, wrong format)
llama3.2:1b ❌
llama3.2:3b ❌
Input
{
"model": "mistral-nemo:12b",
"temperature": 0.0001,
"stream": false,
"tools": [
{
"type": "function",
"function": {
"name": "get_waste_collection",
"description": "Hole die zukünftigen Abfallentsorgungs Daten für einen Ort",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "Eine Adresse oder einen Kreis"
},
"type": {
"type": "string",
"enum": [
"cardboard",
"paper",
"organic",
"cargotram",
"waste",
"garbage",
"etram",
"müll",
"müllabfuhr",
"kehricht",
"kehrichtabfuhr"
],
"description": "Welche Abfallart"
}
},
"required": [
"location",
"type"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}
}
},
{
"type": "function",
"function": {
"name": "retrieve",
"description": "Retrieve information related to a query. Query has to be in the form of a question.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "In the form of a question. Today is: 23. April 2025"
}
},
"required": [
"query"
],
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-07/schema#"
}
}
}
],
"stream_options": {
"include_usage": true
},
"messages": [
{
"role": "system",
"content": "Don't make assumptions about what values to plug into functions. Only call functions when you are sure they are relevant.\nOptional properties are not required to be used. \nGiven a question from the user, call one of your functions to complete the question. If the question cannot be answered by your available functions, return __NONE__.\n\nIf the question is ambiguous or unclear, return __NONE__.\nBe very aware, that the word \"Busse\" in german can mean \"buses\" or \"fines\". The later has nothing to do with public transport schedules or timetables or lost and found.\nAlso \"Öffnungszeiten\" has nothing to do with public transport schedules or timetables. \nZone numbers are not stations or stops."
},
{
"role": "user",
"content": "Question: \"Wann ist Papiersammlung in Wiedikon? Und wann karton an der Quellenstrasse? Wer ist die Stadtpräsidentin\""
}
]
}
Output
mistral:7b ❌
{
"id": "chatcmpl-698",
"object": "chat.completion",
"created": 1745442501,
"model": "mistral:7b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": " To answer your question, I will call the `get_waste_collection` function twice to get the waste collection dates for paper and cardboard in Wiedikon. Then, I will use the `retrieve` function to find out who the city president is.\n\n```javascript\nlet location = \"Wiedikon\";\nlet type = \"paper\";\nget_waste_collection({location: location, type: type}, (error, result) => {\n if(error){\n console.log(__NONE__);\n return;\n }\n let paperCollectionDate = result.date;\n let location2 = \"Quellenstrasse\";\n get_waste_collection({location: location2, type: \"cardboard\"}, (error2, result2) => {\n if(error2){\n console.log(__NONE__);\n return;\n }\n let cardboardCollectionDate = result2.date;\n retrieve({query: \"Wer ist die Stadtpräsidentin?\"}, (error3, result3) => {\n if(error3){\n console.log(__NONE__);\n return;\n }\n console.log(\"Die Papier-Sammlung erfolgt am \" + paperCollectionDate + \". Die Karton-Sammlung erfolgt am \" + cardboardCollectionDate + \". Die Stadtpräsidentin ist \" + result3.name);\n });\n });\n});\n```"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 441,
"completion_tokens": 328,
"total_tokens": 769
}
}
mistral-nemo:12b ✅
{
"id": "chatcmpl-296",
"object": "chat.completion",
"created": 1745438820,
"model": "mistral-nemo:12b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_x93i7twr",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Zürich Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_4aogqqrr",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cardboard\"}"
}
},
{
"id": "call_8yjkmvin",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wer ist die Stadtpräsidentin von Zürich?\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 395,
"completion_tokens": 86,
"total_tokens": 481
}
}
mistral-small3.1:24b ✅
{
"id": "chatcmpl-472",
"object": "chat.completion",
"created": 1745438961,
"model": "mistral-small3.1:24b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_fi8tbtsq",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_cyj0ibc8",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cardboard\"}"
}
},
{
"id": "call_28e4dlc7",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wer ist die Stadtpräsidentin von Zürich\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 397,
"completion_tokens": 67,
"total_tokens": 464
}
}
qwen2.5:0.5b❌
{
"id": "chatcmpl-982",
"object": "chat.completion",
"created": 1745439031,
"model": "qwen2.5:0.5b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_80ycdn3f",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Papiersammlung in Wiedikon\"}"
}
},
{
"id": "call_3npb4yd8",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 52,
"total_tokens": 533
}
}
qwen2.5:1.5b✅
{
"id": "chatcmpl-877",
"object": "chat.completion",
"created": 1745439334,
"model": "qwen2.5:1.5b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_d0gk27va",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_5l6y8cxy",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cardboard\"}"
}
},
{
"id": "call_014sw8rc",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wer ist die Stadtpräsidentin?\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 89,
"total_tokens": 570
}
}
qwen2.5:3b ❌
{
"id": "chatcmpl-352",
"object": "chat.completion",
"created": 1745439503,
"model": "qwen2.5:3b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "None of the provided functions can answer the given question."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 12,
"total_tokens": 493
}
}
qwen2.5:7b ✅❌(almost, wrong type for karton)
{
"id": "chatcmpl-62",
"object": "chat.completion",
"created": 1745439540,
"model": "qwen2.5:7b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_9sj642u9",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_ocipgnzr",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cargotram\"}"
}
},
{
"id": "call_tqmic8pm",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wer ist die Stadtpräsidentin?\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 91,
"total_tokens": 572
}
}
qwen2.5:14b ✅
{
"id": "chatcmpl-613",
"object": "chat.completion",
"created": 1745439737,
"model": "qwen2.5:14b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_l23wv1z2",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_v3mm7nm4",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cardboard\"}"
}
},
{
"id": "call_jliwtuy9",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wer ist die Stadtpräsidentin?\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 93,
"total_tokens": 574
}
}
cogito:3b ❌
{
"id": "chatcmpl-847",
"object": "chat.completion",
"created": 1745439782,
"model": "cogito:3b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "I understand this question is asking about waste collection schedules, public transport timetables, and the city mayor. However, I notice that \"Busse\" could refer to either buses or fines in German.\n\nLet me try to answer using available functions:\n\n1. For Papiersammlung (paper recycling) in Wiedikon:\n - Call get_waste_collection function with location as \"Wiedikon\"\n - This will return the future waste collection dates for paper\n\n2. For Karton (cardboard) at Quellenstrasse:\n - Call get_waste_collection function with location as \"Quellenstrasse\"\n - This will return the future waste collection dates for cardboard\n\n3. For Stadtpräsidentin (city mayor):\n - Call retrieve function with query in German\n - This should return information about the current city mayor\n\nHowever, I notice that the question is ambiguous and unclear because it mixes different topics (waste collection, public transport, and politics). Therefore, I will return __NONE__ as the answer.\n\nQuestion: \"Wann ist Papiersammlung in Wiedikon? Und wann karton an der Quellenstrasse? Wer ist die Stadtpräsidentin\""
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 402,
"completion_tokens": 253,
"total_tokens": 655
}
}
cogito:8b ❌
{
"id": "chatcmpl-329",
"object": "chat.completion",
"created": 1745439964,
"model": "cogito:8b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Let me break this down into parts:\n\n1. For the first part about paper collection in Wiedikon, I can use the get_waste_collection function with location \"Wiedikon\" and type \"paper\".\n\n2. For the second part about cardboard collection at Quellenstrasse, I need to know if there's a specific address or just the street name. Since it's not clear, I'll return __NONE__.\n\n3. The third part asks for the city president (Stadtpräsidentin). This is a general question that can't be answered by any of my available functions, so I'll return __NONE__.\n\nFor the first part, here's what I get:\n```javascript\nget_waste_collection({\n location: \"Wiedikon\",\n type: \"paper\"\n})\n```\n\nThe rest of the questions are either unclear or require information that isn't available through my tools."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 402,
"completion_tokens": 186,
"total_tokens": 588
}
}
cogito:14b ✅ ❌(almost, missed retrieval tool)
{
"id": "chatcmpl-25",
"object": "chat.completion",
"created": 1745441056,
"model": "cogito:14b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_1szudg17",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_jljlrong",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cardboard\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 62,
"total_tokens": 543
}
}
cogito:32b ✅
{
"id": "chatcmpl-645",
"object": "chat.completion",
"created": 1745439900,
"model": "cogito:32b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_7vbu1vwu",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Wiedikon\",\"type\":\"paper\"}"
}
},
{
"id": "call_pa15zof8",
"index": 0,
"type": "function",
"function": {
"name": "get_waste_collection",
"arguments": "{\"location\":\"Quellenstrasse\",\"type\":\"cardboard\"}"
}
},
{
"id": "call_1xjqpqjw",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wer ist die Stadtpräsidentin?\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 481,
"completion_tokens": 89,
"total_tokens": 570
}
}
granite3.2:2b ❌
{
"id": "chatcmpl-150",
"object": "chat.completion",
"created": 1745440256,
"model": "granite3.2:2b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "__NONE__\n\nExplanation: The question is ambiguous and contains multiple unrelated topics. It asks for waste collection times (which I don't have data for), a street address, and the name of a city official. As my functions only provide information on waste collection schedules and city officials' names, I cannot answer this question comprehensively."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 428,
"completion_tokens": 72,
"total_tokens": 500
}
}
granite3.2:8b ❌
{
"id": "chatcmpl-533",
"object": "chat.completion",
"created": 1745440456,
"model": "granite3.2:8b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "<tool_call>[{\"arguments\":{\"query\": \"Wann ist Papiersammlung in Wiedikon?\"}}]"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 428,
"completion_tokens": 29,
"total_tokens": 457
}
}
phi4-mini:3.8b ✅ (correct answer, wrong format)
{
"id": "chatcmpl-935",
"object": "chat.completion",
"created": 1745441287,
"model": "phi4-mini:3.8b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Functions called:\n1. get_waste_collection with parameters {\"location\": \"Wiedikon\", \"type\": \"paper\"}\n2. get_waste_collection with parameters {\"location\": \"Quellenstrasse\", \"type\": \"cardboard\"}\n3. retrieve with parameter {\"query\": \"Wer ist die Stadtpräsidentin in Wiedikon?\"}"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 375,
"completion_tokens": 70,
"total_tokens": 445
}
}
llama3.2:1b ❌
{
"id": "chatcmpl-208",
"object": "chat.completion",
"created": 1745441410,
"model": "llama3.2:1b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_3e6er6en",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Papiersammlung in Wiedikon und Karton an der Quellenstrasse\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 491,
"completion_tokens": 38,
"total_tokens": 529
}
}
llama3.2:3b ❌
{
"id": "chatcmpl-466",
"object": "chat.completion",
"created": 1745441379,
"model": "llama3.2:3b",
"system_fingerprint": "fp_ollama",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_xd4r09yc",
"index": 0,
"type": "function",
"function": {
"name": "retrieve",
"arguments": "{\"query\":\"Wann ist Papiersammlung in Wiedikon? Und wann karton an der Quellenstrasse? Wer ist die Stadtpräsidentin\"}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 491,
"completion_tokens": 44,
"total_tokens": 535
}
}