- https://arxiv.org/pdf/2402.17442
- Source: arxiv
TL;DR
Ansible Lightspeed is a domain-specialized LLM service for IT automation that achieves 49% strong acceptance of multi-line YAML suggestions across nearly 11,000 real users, showing that smaller domain-specific models can match large general-purpose ones on narrow tasks.
Summary
Most code generation research targets general-purpose languages, leaving domain-specific automation underexplored. Ansible Lightspeed generates Ansible YAML tasks from natural language prompts, evaluated on real usage data from 10,696 users. The study introduces a Strong Acceptance Rate metric — acceptance where users edit less than 50% without critical changes — finding a 49.08% rate for multi-line suggestions. 30-day retention reached 13.66%, and domain-specialized smaller models proved competitive with large general-purpose models on narrow IT automation tasks.
Key contributions
- Deploys and evaluates an LLM-based Ansible YAML generation service at scale with nearly 11,000 real users.
- Introduces the Strong Acceptance Rate metric that corrects standard acceptance rates for cosmetic edits.
- Establishes the first N-Day retention baseline (13.66% at 30 days) for domain-specific code assistants.
- Demonstrates that domain-specialized smaller models are competitive with large general-purpose LLMs on narrow automation tasks.
When to cite
- When arguing for domain-specialized smaller models over large general-purpose LLMs for narrow task domains.
- When critiquing standard acceptance rate as a misleading metric for code generation adoption.
- When discussing real-world user retention and engagement data for AI coding assistants.
- When evaluating infrastructure-as-code generation as a concrete, auditable LLM use case.