How Developers Are Using ChatGPT

To say ChatGPT has been making waves would be an understatement. The natural language chatbot is hugely popular and the topic of dinner table conversation worldwide, and sometimes even a participant (probably). 

It’s proven useful, or at least interesting, in various domains. Some real estate agents swear by it, and many schools have been more than a little concerned by it. It can even handle a task as dynamic, unpredictable, and detailed as running a game of Dungeons and Dragons. 

Unsurprisingly, it has also made its way into developers’ circles and workflows. But how exactly are developers’ using it? What are they finding it useful for? And, perhaps even more instructive, what are they finding it not useful for? 

Every writer knows you can’t edit a blank page, and editing is often where the real magic of writing happens. This is the most common theme of how developers have found ChatGPT helpful. But it also points to the chatbot’s greatest weakness.

The strengths of ChatGPT – according to devs

Broadly, there are three major themes to ChatGPT’s strengths in assisting developers. The first is that it gives them something to work with. 

Getting developers started

Software engineer Joseph Onifade describes the chatbot as “like having a personal assistant.”

For him, the greatest utility ChatGPT provides is for simple day-to-day tasks and research like “understanding how to comment code out on a new IDE and figuring out what programming language would be best for an application or tool in development.”

That echoes how developer Dave Lane of Twin Sun Solutions has been using it: “ChatGPT has been wonderful for prototyping simple scripts,” he says. “It shines at helping you with rote tasks, quickly writing boilerplate code that you don’t want to bother thinking about.” In fact, it’s so good that it usually “can produce a functioning script that does what I want on the first try.” 

Developer Paul Marks says he’s been using it in programming languages he’s already very familiar with, where he finds it “a quick way of producing high-quality code I can then go through and make any changes to” where necessary. He’s also found it handy as a refresher. “If it’s been a while since I’ve done something, I can ask it for an example, and it gives me a relevant example that’s easy to read/understand.”

Filling gaps, piecemeal work, catching bugs

Even if you wouldn’t trust ChatGPT to write your entire codebase for you, some developers have found it helpful in generating smaller pieces of code as needed.

Developer Anthony Bautista has found it useful in text generation and (somewhat ironically) chatbot development. “It is also useful for generating code snippets and fragments.”

For developer Shikha Shah, this capability is one of ChatGPT’s standout features. “ChatGPT can be trained in a specific programming language and used to generate code snippets that conform to a given set of guidelines. This can save time and increase productivity for developers.”

Alongside using ChatGPT “to start the work on a new feature,” developer Ryan Yockey says one of its greatest strengths has been using it to “solve a few bugs in our code base. It has been incredibly good once you give it some basic context.” 

Developer Jan Cizmar found it essentially the same. “It’s great for writing small parts of code and finding bugs in smaller scripts. I can also write tests for code like this.” 

Maintaining momentum

Once developers are rolling and they’ve built some code, ChatGPT has also proven helpful in keeping developers moving. 

That’s essentially how developer Aleksei Kankov has been using it: “For me, the most helpful thing is auto-completing code: ChatGPT can be used to suggest the next line of code to complete a given code snippet.” Similarly, developer Pulkit Bhardwaj primarily leverages the chatbot to generate piecemeal code snippets. 

And for Chris Reynolds, a developer with Pantheon Systems, ChatGPT has helped him keep rowing through uncertain waters. “I’ve used ChatGPT a couple of times to supplement my development, primarily when working in languages that I’m not immediately familiar with,” he says. However, he added he knew the languages well enough to troubleshoot the resulting code afterward.

The drawbacks of ChatGPT for devs 

Ask any developer, or any person even, and they’ll tell you the same thing: ChatGPT is not a human. It sounds painfully obvious (it is), but it’s a crucial piece of context to keep in mind when you’re using it. Don’t ask it for things only a person can provide. 

It can be confidently wrong

The Dunning-Kruger effect explains people’s overconfidence in a field they’re ignorant in because they don’t know enough to know how much they don’t know. Until machines achieve self-awareness, ChatGPT doesn’t have any, so it’s especially vulnerable to this. 

“The biggest limitation of using ChatGPT for code generation,” says developer Ellen Cibula, “is that the output entirely depends on your input. You need to understand the logic of your problem. If you don’t understand what you are trying to do and give ChatGPT incomplete instructions, you will get code that may not do what you want.” Editing is part of the process, but there are limits to that. If the code you have to work with is entirely unworkable, it’s twice as much work taking it all apart and putting it back together.

Sometimes ChatGPT can’t figure out what you want if you aren’t precise in your phrasing, as developer Christopher William explains. “When I was using it for coding help, I found it difficult to phrase questions the right way. When I was not specific, the software would return general coding information that I could have found on the internet. ” 

ChatGPT also struggles in higher-level tasks, in developer Chris Looney’s experience. “It may not be suitable for certain tasks that require a high degree of domain-specific knowledge or understanding.” 

This can also make ChatGPT a treacherous tool for developers still early in their careers, says developer Marat Minulin, as it can confidently return incorrect code. People just starting in programming “will not be able to understand the problem in the code immediately.”

It doesn’t understand context

Because ChatGPT is not human, it can also have trouble parsing questions. Sometimes this means no response when it knows the answer. Sometimes it means a confidently incorrect answer. Devs need to understand exactly what they’re asking to be sure they understand the response. 

In Dave Lane’s experience, “ChatGPT can produce incorrect responses while appearing confident in its answer.” If developers take those at face value, it could lead to “bugs or security vulnerabilities in their software.” 

Paul Marks concurred: “The area I wouldn’t rely on it for is generating code for a concept or language I wasn’t already familiar with,” he says. 

Aleksei Kankov suggested ChatGPT’s most significant drawback may be its lack of understanding of context. “ChatGPT is a language model and does not have a full understanding of the context in which it is generating text. This can lead to errors or inaccuracies in the output,” he says. 

This echoes developer Chris Reynolds’ impressions of it. “The biggest limitation of ChatGPT is not having a holistic understanding of the codebase,” he says. 

High-level, specific, or complex domains are beyond its abilities

Other developers noted that while ChatGPT might shine in simple work, it can be counter-productive in complex contexts.

As Jan Cizmar notes, “the most significant limitation is that ChatGPT doesn’t accept the whole codebase of more extensive projects as input, so it doesn’t know the file structure and relations between files.” 

Or, as developer Antoine Goret says, “It’s ideal for creating boilerplate code, for example, but not so for writing complex, bespoke code for something like a banking application.” 

As a result, some developers say they don’t use it at all. Jan Cizmar ended up dropping it, finding it was slowing him down. Head of Software Engineering at TechStack Ltd., Danylo Tolmachov, found that “ChatGPT didn’t meet my technical needs,” although he does use it to help write performance reviews. And in developer Chris Love’s experience, “most of the code is incomplete, buggy, or just wrong. You have to review and test it thoroughly.” 

Useful, but no replacement for human devs

For many developers, ChatGPT has proven to be a useful tool. While not a revolution that completely upsets how software development is done (yet), it’s been an excellent source of automation. Much like other forms — such as the automations in Zenhub — it helps developers by simplifying and streamlining workflows, letting them focus on higher-level creative work. 

This is a good balance, as for many developers, ChatGPT has demonstrated that it’s by no means capable of replacing a human developer. Letting people focus on the work that requires their creativity and flexibility helps developers work more efficiently, which they love. 

But as developers noted, it’s crucial to understand how ChatGPT works, as this informs what it can and, more crucially, what it can’t do.