Skip to content

[BUG] LiteLLM path drops token usage for tool-call turns and structured output (response_model) #8679

Closed

Description

@alexmarchenko-crewai

🤖 [agent on alexmarchenko-crewai's behalf]

Description

When an LLM uses the LiteLLM fallback path, two kinds of call never report token usage. That covers LLM(model=..., is_litellm=True) and any model prefix that isn't a native provider, for example litellm_proxy/....

  1. Native tool-call turns. These are calls with tools but no available_functions, which is how the agent executor makes them. They return the tool calls without emitting LLMCallCompletedEvent. This happens in sync and async handlers, streaming and non-streaming.
  2. Structured output through Instructor (response_model, e.g. a task with output_pydantic). LLMCallCompletedEvent is emitted with usage=None, and the tokens are never tracked.

Tracing still records these calls, so traces show requests with 0 tokens and $0 cost. The native OpenAI, Azure and Anthropic providers report usage for both cases.

Steps to Reproduce

  1. Create LLM(model="anthropic/claude-sonnet-4-5", is_litellm=True). Any LiteLLM-routed model shows the same behavior.
  2. Run a crew with a task that has output_pydantic=SomeModel, and a crew whose agent has one tool.
  3. Listen on crewai_event_bus for LLMCallStartedEvent and LLMCallCompletedEvent, and inspect event.usage.

Expected behavior

Every LLM call emits LLMCallCompletedEvent with usage populated, the same as on the native providers:

  • Tool-call turns should emit a tool_call completion.
  • Structured-output calls should carry the completion's token usage.

Screenshots/Code snippets

Observed on 1.15.29 with the LiteLLM path:

Scenario Started Completed Completed with usage
Plain task 1 1 1
Task with output_pydantic 1 1 0 (usage=None)
Agent with a tool 2 1 (the tool-call turn emits no completion) 1

Operating System

macOS

Python Version

3.12

crewAI Version

1.15.29 (also present on main / 1.15.30)

crewAI Tools Version

N/A

Virtual Environment

uv

Evidence

The drop points are in lib/crewai/src/crewai/llm.py:

  • if tool_calls and not available_functions: return tool_calls in the four response handlers.
  • The response_model and self.is_litellm branch, which emits usage=None.

Possible Solution

  • Emit a TOOL_CALL completion with usage before returning tool calls, as the native providers do.
  • For structured output, get the raw completion from Instructor's create_with_completion and report its usage.

Implemented in #8678.

Additional context

None.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions