Can AI / ChatGPT replace a programmer, or is it a tool that makes a developer more productive?

I needed a simple utility function that I figured would be about 20-30 lines of code. Rather than starting from scratch (or from StackOverflow), I decided to try using ChatGPT. Let’s see how that went….

The Problem

I needed a utility method that would get the specific number of octets of a user’s IP address, and since I might be calling it frequently, performance needed to be considered. For example, if the IP address was 192.168.3.25 and I wanted the first 2 octets, I’d expect it to return 192.168.

ChatGPTs first attempt

I asked ChatGPT to generate the method:

Result: not bad. As a friend told me, ChatGPT is often useful for starting “white sheet” projects, where you need to create something completely from scratch. Once ChatGPT has generated the initial code, you can keep working from there.

But is it efficient?

Looking at the string split and join methods, I was wondering how efficient this method would be, so I asked:

Nice! I could’ve Googled the efficiency of the string split and join methods vs IndexOf, but this was a nearly-instantaneous answer.

But what about …

Edge cases

It often seems that writing code that correctly handles normal data is less than half the work: the problems come when you end up with abnormal data (or abnormal usage). So I asked:

I wrote a battery of unit tests for this method and started running them, which quickly pointed out some additional problems, which I passed on to ChatGPT:

Not quite what I wanted, since I’d assume that if I ask for 4 octets I would get all 4 octets, but this is also an edge case because who would call this method asking for the complete IP address? (Though a programmer in the future could make a mistake and call your method with parameters that don’t make sense–shoot, I’ve done that to my own code!) So we’ll ignore this and move on to the next problem. Since I’ve already encountered a couple edge cases that caused the method to blow up, let’s see if ChatGPT can take care of all possible failures:

My first hint that it hadn’t solved all problems should have been seeing the first sentence of its response state, “to make the method more fail-safe….”

I was curious how expensive the IPAddress.TryParse method was:

That gave me a little pause: since IP address is ultimately coming from an HTTP Request on the web, who knows what a hacker could put into that to cause problems.

I was expecting an IPv4 address as input, and one of my unit tests was an IP6 address, which caused an error. So I asked:

Better, but my battery of unit tests still showed another problem:

(I’ve learned through painful experience to never trust external inputs to my code!)

That addresses the partial IP address, but I was wondering about efficiency again:

End result

So after some prodding ChatGPT has handled some of the edge cases I’ve thrown at it, but I now have a 44 line method, and limited confidence in its ability to not blow up completely if I get some kind of garbage input on an IP address (and I didn’t want it to blow up but just indicate an error and continue).

In the end, I used some of what I’d ferreted out from ChatGPT but instead wrote my own method. On the inputs I decided to just do simple bounds checking on the number of octets to return and a basic format check of the IP address (does it have 3 periods). I used a for loop with the IndexOf(‘.’) function to find the position to substring. I included a try/catch block to handle any other weird edge cases I hadn’t considered in my unit tests. My method ended up being only 27 lines of code.

My takeaway

ChatGPT was quite useful in giving me some direction to start on the function, and quickly answering questions about code efficiency–both things I could’ve derived with a little internet searching but I got the results faster with ChatGPT. The edge cases which I’d setup in my unit tests (I usually have more test cases for various failures than I do for “happy paths”) showed multiple defects in the resulting code, though to be fair they would have probably shown a couple similar defects in my own code if I’d started from scratch.

Although I ultimately ended up throwing away much of the code ChatGPT generated, the time spent going through the process of handling edge cases and finding efficient ways to parse the IP address was not time wasted, since it was all useful for the method I ended up writing.

AI is useful, but developers are still needed!