The coding industry is experiencing rapid changes with the AI coding tools, ranging from basic code completion to full-scale software development lifecycle.
Let's borrow the terminologies from autonomous driving to categorize these tools into five levels: L1 to L5.
Code Completion (L1)
At the foundational level, L1 tools like GitHub Copilot focus on code completion.
GitHub Copilot excels at code completion
These tools are useful for developers, streamlining the coding process with intelligent suggestions and completions.
They have become the norm in modern development environments nowadays, paving the way for more advanced AI coding tools.
Code Completion (L1) Products
Paid:
Open-source:
Task-Level Automation (L2)
L2 focuses on task-level automation.
We have LLMs like ChatGPT. They are good at handling development tasks such as new features, bug fixes, and refactoring based on descriptive prompts.
ChatGPT is a versatile LLM model that can assist with coding tasks
However, using LLMs for coding tasks can be challenging. We need well-crafted prompts and relevant source code context to generate good quality code. Manually creating these prompts can be time-consuming and tedious.
To address these issues, we have workflow automation tools such as aider (command line tool) and 16x Prompt (desktop application), which aims to streamline the process of generating prompts and getting the best quality code from LLMs.
These tools are starting to enter the market, providing a more user-friendly interface for developers to interact with LLMs and automate coding tasks.
Task-Level Automation (L2) Products
LLMs:
Workflow Automation Tools:
- aider (command line tool)
- 16x Prompt (desktop application)
Project-Level Automation (L3)
L3 represents an early stage in project-level automation. Tools such as Codegen, Sweep, and Pythagora demonstrate abilities to take a project, analyze its requirements, and generate relevant pull requests.
Codegen demonstrates ticket to PR automation
However, these systems are at a preliminary stage, capable of managing relatively simple coding projects and generating basic code snippets. They require human intervention to ensure code quality and relevance, limiting their autonomy.
Project-Level Automation (L3) Products
Paid:
Free:
Partially Open-source:
Open-source:
AI Software Engineer (L4)
L4 marks the threshold of holistic development lifecycle management. At this level, we have tools like Devin. They envisioned to have access to terminals and deployment tools, capable of managing the entire spectrum of development activities.
Devin benchmark against LLM models
These advanced systems can interpret product requirements, manage code deployment, and maintain software in production environments, embodying the role of an AI software engineer.
AI Software Engineer (L4) Products
AI Development Teams (L5)
At L5, AI coding will most likely involve an AI system with multiple AI agents.
For example, in the Microsoft paper AutoDev, it was proposed that agents can "receive objectives and conversation histories from the Agent Scheduler, responding with actions specified by the Rules and Actions configuration".
However, only "a single GPT-4 agent" was used for evaluation in the paper.
AutoDev enables an AI Agent to achieve a given objective by performing several actions within the repository. Source: https://arxiv.org/html/2403.08299v1
With the GPT-5 release on the horizon, the dream of AI development teams is not far-fetched.
We could see a future where the AI system could replicate entire software development teams, coding and collaborating across different facets of software creation.
AI Development Teams (L5) Products
Which Level is Right for You?
The choice of AI coding tools depends on your needs and the complexity of your projects.
If you just need help with code completion, L1 tools like GitHub Copilot are sufficient.
For more complex tasks like feature development and bug fixes, L2 LLMs like ChatGPT or Claude 3 can be used together with workflow automation tools like aider or 16x Prompt.
If you are adventurous and want to experiment with project-level automation, L3 tools like Codegen or Sweep can be a good starting point.
Also, you don't have to stick to one level. You can use a combination of tools from different levels to suit your specific requirements.
For example, I use GitHub Copilot for simple code completion tasks that can be solve in 5 seconds, ChatGPT and 16x Prompt for more complex tasks that require a few minutes of prompting.
Looking Ahead
In 2024, AI's role in coding is rapidly evolving, from basic syntax assistance to full-scale development lifecycle management.
As AI continues to mature, we can expect more sophisticated tools that will redefine the coding landscape, enabling developers to focus on higher-level tasks while AI handles the routine aspects of software development.