TL;DR
LLMCompiler applies compiler design principles — dependency analysis, parallel scheduling — to LLM tool calls, achieving up to 3.7x latency and 6.7x cost reductions over sequential ReAct execution.
Summary
LLMs calling external tools sequentially waste latency and tokens even when tasks are fully independent. Inspired by classical compiler design, LLMCompiler introduces a three-stage pipeline — a Function Calling Planner, a Task Fetching Unit, and an Executor — that identifies inter-task dependencies and dispatches independent calls in parallel. Benchmarked against ReAct, it achieves up to 3.7x latency reduction, 6.7x cost reduction, and ~9% accuracy improvement. The architecture is model-agnostic and was published at ICML 2024.
Key contributions
- Proposes LLMCompiler, a three-stage planner-fetcher-executor pipeline for dependency-aware parallel tool dispatch.
- Achieves 3.7x latency reduction and 6.7x cost reduction versus sequential ReAct on function-calling benchmarks.
- Shows that parallel execution reduces error propagation from sequential misorderings, improving accuracy by ~9%.
- Demonstrates a model-agnostic architecture compatible with both open-source and proprietary LLMs.
When to cite
- When arguing that orchestration design can matter more than model choice for cost and latency in agentic systems.
- When applying compiler concepts (dependency graphs, parallelism) to LLM tool-calling pipelines.
- When looking for evidence that parallel tool execution improves both efficiency and accuracy over sequential approaches.
- When evaluating alternatives to ReAct for multi-tool LLM agent frameworks.
LLM Prompt Engineering Computer Architecture ART - Automatic multi-step reasoning and tool-use for large language models