AI Toolkit Plugin Brings LLM Debugging to JetBrains IDEs
An AI Toolkit plugin for JetBrains IDEs, detailed in arXiv:2605.14612, is designed to assist software developers lacking machine learning expertise in testing and debugging LLM-based functionalities. This plugin incorporates tracing and evaluation seamlessly into the Run/Debug cycle. A mixed methods study revealed three primary requirements: consistent evaluation, real-time trace exposure, and reduced setup and context switching. The toolkit offers an IDE-integrated workflow featuring run-initiated trace capture, hierarchical analysis, effortless dataset integration from traces, and unit-test-style evaluations with customizable metrics. The initial release in PyCharm indicates encouraging early results, such as high conversion rates during Run promotions, ongoing engagement from trace users, and minimal churn, suggesting that IDE-native observability enhances developer efficiency.
Key facts
- AI Toolkit plugin for JetBrains IDEs targets developers building AI features on LLMs and agentic workflows.
- Plugin brings tracing and evaluation into the Run/Debug loop.
- Mixed methods study revealed three practitioner needs: regular evaluation, trace exposure at execution, minimal setup.
- Features include run-triggered trace capture, hierarchical inspection, one-click dataset addition, unit-test-like evaluations.
- First release is in PyCharm.
- Early signals show strong conversion at Run, sustained usage, low churn.
- Paper published on arXiv with ID 2605.14612.
- Plugin aims to help product-focused engineers without ML background.
Entities
Institutions
- JetBrains
- arXiv