AI for software developers: Why performance reporting is critical (and complicated)

At least 60% of Zenhub’s software developers are using AI in the workplace. And that’s great. In theory.

We want our team to leverage every tool they can to do their work effectively. We’ve also seen some solid evidence that products like CoPilot have a positive impact on busy dev teams. 

But here’s the truth — if someone asked me to measure the impact AI has had on my team to date, I’d be playing a guessing game. 

Right now, the use of artificial intelligence in software development is an experiment. And while we’re always down for innovation and exploration, we have yet to determine exactly how this experiment is going.

Ideally, dev teams should know:

• How we’re interacting with AI
• How it’s changing our workflow
• And how these factors impact our companies

Below, we explore some tools (and important considerations) that could help us get through the weeds.

The goals of implementing AI for software teams

Depending on which tools you use and how your team operates, AI could show up in your day-to-day routine in several ways, but we’re looking at three main buckets — project management, task management, and coding.

When you leverage AI in any of these capacities, the primary goal is the same. AI in software development is all about time saved.

And when we save time, we can generally expect:

• Improved productivity
• Increased capacity
• And a positive impact on the bottom line of our companies

All desirable outcomes, obviously. 

Why evaluating team performance is crucial

For better or worse, artificial intelligence will change the world of software development entirely.

GitHub CEO Thomas Dohmke recently estimated that 80% of all code will soon be AI-generated. And most of us can agree that AI will play a massive role in project and task management, which should be a game-changer given that software developers spend just 43% of their day coding.

That said, we need to be asking ourselves whether some applications of AI (as it exists today) are more harmful than helpful to productivity, and answering this question can also be tricky.

Right now, the AI tools we’re using in software development, like Copilot, don’t offer much in terms of data to help us measure their usefulness or efficacy. 

Ideally, all of these tools would offer reports that tell us:

• How often we interact with them
• How many suggestions they make
• How many of those suggestions we accept 
• How much code they generate
• And how many changes we make to the code or content they create for us

But until that data becomes available, we need to think about how we can measure team performance as AI plays a bigger role in our industry. 

The metrics that can help us evaluate our teams

No matter how you break it down, measuring the impact of AI in software development is complicated. This isn’t the neat-and-tidy answer people want, but that’s just the reality for our industry.

Looking at any one factor in isolation is unlikely to provide meaningful information for your team. If we look only at time spent coding, our conclusions could be misleading. Sure, our devs may spend less time writing code, but they could spend even more time rewriting and adapting the code AI generates.

To get a sense of how much time AI is actually saving us, we’ll need to look at a wide array of metrics and data points over a significant period of time. Depending on your team’s approach to development, there are several measures that could help you judge how things are going.

You may want to consider:

• Completed issue volume
• Pull requests closed
• Time spent on projects 
• # of bugs reported and # of bugs eliminated
• The progression of tasks through pipelines

And eventually, the quality of what your team is able to produce — but AI isn’t quite there yet.

In general, this assessment is going to be easier for teams that track hours and have a good amount of data from their pre-AI days. But even that isn’t a perfect solution. 

Tip: To make this job easier, you can (and should) lean on tools like Zenhub’s reporting suite, which provides tons of valuable insights on performance, capacity, and the habits of your team.

The long game 

I mentioned this before and can’t stress it enough. This assessment won’t be black and white, and if we want to get a concrete sense of how AI impacts software development, we should plan to play a long game. Think weeks or months, at least.

There are countless factors that could cause a shift in performance, including the fact that no two projects are exactly the same. Our teams will also need time to adapt their workflow to AI tools and settle into their new reality, so immediate observations could be misleading.

We need to account for these kinds of variables and leverage all of the tools available to us to get a handle on the impact of AI and where our industry is headed.