How reliable is AI lke ChatGPT in giving you code that you request?
ChatGPT is a language model, it’s not intended for code and you’re using it “off label” at your own risk. It can produce working code, which is impressive in itself, but in order to know if it’s decent code you still need to be competent with that language. I had someone run a few prompts for me a while back, it ignored central parts of the query, and its output was basically like a very junior developer - fair enough, but not great or even that good.
Potentially useful, but if you expect it to be more than one part of the “process”, you might be setting yourself up for trouble.
Edit: just like it’s not a coder, it’s not a search engine or knowledge base, either. It just knows language and what seems like it ought to follow a given phrase. Be very aware of this difference, because sometimes it spits out 100% falsehoods with the same level of confidence and authority as the true stuff.
I think it’s important for people to also truly understand that generative machine learning models like ChatGPT also only “know” what they’ve seen before. There’s no interpretation or synthesis. It merely regurgitates what it’s seen, with some sampling from a probability distribution.
This means, if you’re asking for something niche, and it’s only seen what you’re prompting it for once (or, really, the same text repeatedly across multiple websites) , there’s a very good chance that it will just recreate that artifact wholesale.
Which means you need to be cognizant of what the license for that material is before you use it in a product!
I have built several programs with ChatGPT 4 by now. From very basic Python scripts to Python webscrapers and C# in combination with Unity3D.
In the beginning it was much better than it is currently. At the moment context is severely hampered no matter the limit and you’ll be bashing your head against circular arguments and it straight up ignoring stuff you just posted two messages ago.
Trying to troubleshoot code it wrote a few days ago will be a slog and like dragging yourself over nails at times. Here’s what I have found to help and make life better:
- Be very, very, very precise in your instructions. And keep them saved, so you can reuse them later (point 4)
- From the very start plan to build your project with small functions that interact (good policy anyway) which makes troubleshooting and changing these functions much easier and will prevent you running into message limits.
- If it fails to work for you the way you need it, you’ll might have to scrap your entire code and start over with ChatGPT -> Again, the reason for point 2 being very important. Scrapping one functin is much less painful than an entire tool.
- Start new chats when you feel the quality degrading. Sometimes it helps and since the context is garbage at the moment anyway, it doesn’t matter much.
- Post the code it is supposed to fix every single time. It will inevitably refer to other code, code hallucinations, etc. otherwise. Again why point 2 is important.
I agree with the other comments that ChatGPT isn’t really that good for programming, it hallucinates often and you end up working too hard just to try and figure out what it got wrong. However, I have found a good AI engine, phind.com, that has started to replace my google searches. It’s just a wrapper for ChatGPT, but it cites its sources so you can verify or dig deeper, provides search engine results in a sidebar and has upvote/downvote options to help it improve. So it feels like a personal google “agent” that runs off and googles something for you and comes back with a concise report.
Personally I just can’t work with system that lies to me (even for a little) but all the time.
I tried to use chatGpt and Bing bot and phind.com few times and everytime I got answers that looks like real and looks like correct answer but slightly (and few times completely) wrong.
Everytime I have to reread documentation, check links, investigate is there a reason why LLM answered this way, maybe I wrong this time and LLM found something that I did not found…
I agree that phind.com get best results, but every small incorrectness here and there irks me and makes me question myself and answer as whole.
Upd: in general questions, like when you trying to investigate some new field, technology, tooling suite LLM is very, very good. When you want to get something like overview of topic that you interested in.
I quite like GitHub Copilot and use it a lot, but I find ChatGPT not all that useful.
For actual coding, it feels like describing what I want it to do is more complicated than doing it myself.
I can see some uses as a search engine, but I’ve had a lot of bad luck where it suggested code that was plain wrong or not working and often did not even compile, so most of the time, I’d rather look on GitHub, Stackoverflow or sites like that.
I use chatgpt a lot when coding. It’s pretty good and the code is typically usable. But sometimes it messes up hard and it can take a while to realize that. Net benefit though and I’m sure the technology will improve over time.