The coding industry is experiencing rapid changes with the AI coding tools, ranging from basic code completion to full-scale software development lifecycle.
Let's borrow the terminologies from autonomous self-driving cars to categorize these tools into five levels: L1 to L5.
| Level | High-level Approaches | Example Products |
|---|---|---|
| L1 | Code-level Completion IDE with Auto-complete | GitHub Copilot, Codium, Codeium Tabby |
| L2 | Task-level Code Generation Ticket to Code IDE with Chat | ChatGPT, Claude aider, cline, 16x Prompt Cursor, Continue, PearAI, Windsurf |
| L3 | Project-level Generation Ticket to PR Prompt to UI | Codegen, Sweep Pythagora, Plandex, MetaGPT v0 by Vercel |
| L4 | AI Software Engineer PRD to Production | Factory Code Droid, Marblism Devin, Genie, bolt.new |
| L5 | AI Development Teams | AutoDev |
Code Completion (L1)
At the foundational level, L1 tools like GitHub Copilot focus on code completion.
GitHub Copilot excels at code completion
These tools are useful for developers, streamlining the coding process with intelligent suggestions and completions.
They have become the norm in modern development environments nowadays, paving the way for more advanced AI coding tools.
Code Completion (L1) Products
Paid:
Open-source:
Task-Level Automation (L2)
L2 focuses on task-level automation.
We have LLMs like ChatGPT. They are good at handling development tasks such as new features, bug fixes, and refactoring based on descriptive prompts.
ChatGPT is a versatile LLM model that can assist with coding tasks
However, using LLMs for coding tasks can be challenging. We need well-crafted prompts and relevant source code context to generate good quality code. Manually creating these prompts can be time-consuming and tedious.
To address these issues, we have workflow automation tools such as aider (command line tool) and 16x Prompt (desktop application), which aims to streamline the process of generating prompts and getting the best quality code from LLMs.
We also have IDEs and IDE extensions like Cursor, Continue, and PearAI that integrate LLMs into the development environment, providing a seamless experience for developers.
These tools are starting to enter the market, providing a more user-friendly interface for developers to interact with LLMs and automate coding tasks.
Task-Level Automation (L2) Products
LLMs:
Workflow Automation Tools:
- aider (command line tool)
- 16x Prompt (desktop application)
IDE Extensions:
IDEs:
Project-Level Automation (L3)
L3 represents an early stage in project-level automation. Tools such as Codegen, Sweep, and Pythagora demonstrate abilities to take a project, analyze its requirements, and generate relevant pull requests.
A key feature of these L3 tools is the ability to automate multiple parts of the software development process, from requirements, to code generation, to pull request creation, by integrating with project management tools like Jira and source code platforms like GitHub.
Codegen demonstrates ticket to PR automation
However, these systems are at a preliminary stage, capable of managing relatively simple coding projects and generating basic code snippets. They require human intervention to ensure code quality and relevance, limiting their autonomy.
Project-Level Automation (L3) Products
Paid:
Free:
Partially Open-source:
Open-source:
AI Software Engineer (L4)
L4 marks the key transition from human-driven coding to AI-driven software development. Development processes are fully automated, from product requirements to production deployment.
At this level, we have tools like Devin, Marblism and Genie from Cosine.
They envisioned to have access to terminals and deployment tools, capable of managing the entire spectrum of development activities.
Devin benchmark against LLM models
Sample projects made with Marblism
Genie score on SWE-Bench (August 2024)
These advanced systems can interpret product requirements, manage code deployment, and maintain software in production environments, embodying the role of an AI software engineer. They enable non-technical people to go from zero to a fully functional software product in minutes.
AI Software Engineer (L4) Products
Launched:
Upcoming / Waiting List:
AI Development Teams (L5)
At L5, AI coding will most likely involve an AI system with multiple AI Software Engineer as described above. These AI agents can collaborate and work together on a project, each specializing in different aspects of software development.
For example, in the Microsoft paper AutoDev, it was proposed that agents can "receive objectives and conversation histories from the Agent Scheduler, responding with actions specified by the Rules and Actions configuration".
However, only "a single GPT-4 agent" was used for evaluation in the paper.
AutoDev enables an AI Agent to achieve a given objective by performing several actions within the repository. Source: https://arxiv.org/html/2403.08299v1
With the GPT-5 expected to release in 2025, the dream of AI development teams is not far-fetched.
We could see a future where the AI system could replicate entire software development teams, coding and collaborating across different facets of software creation.
AI Development Teams (L5) Products
Which Level is Right for You?
The choice of AI coding tools depends on your needs and the complexity of your projects.
If you just need help with code completion, L1 tools like GitHub Copilot are sufficient.
For more complex tasks like feature development and bug fixes, L2 LLMs like ChatGPT or Claude 3 can be used together with workflow automation tools like aider or 16x Prompt.
If you are adventurous and want to experiment with project-level automation, L3 tools like Codegen or Sweep can be a good starting point.
Also, you don't have to stick to one level. You can use a combination of tools from different levels to suit your specific requirements.
For example, I use GitHub Copilot for simple code completion tasks that can be solve in 5 seconds, ChatGPT and 16x Prompt for more complex tasks that require a few minutes of prompting.
Looking Ahead
In 2024, AI's role in coding is rapidly evolving, from basic syntax assistance to full-scale development lifecycle management.
As AI continues to mature, we can expect more sophisticated tools that will redefine the coding landscape, enabling developers to focus on higher-level tasks while AI handles the routine aspects of software development.