You are correct in that a computer is only as smart as its programming. The way an LLM works is essentially just a chatbot with tons of “if, then” decision trees. You can feed it all the correct facts you want; it’s not programmed to spit out accurate facts, it’s programmed to spit out something that looks like it was written by a human. So it can, and will, say stupid shit pretty frequently. But it will be stupid shit that reads like something a real person might have said if they were talking out of their ass with confidence.
You are correct in that a computer is only as smart as its programming. The way an LLM works is essentially just a chatbot with tons of “if, then” decision trees. You can feed it all the correct facts you want; it’s not programmed to spit out accurate facts, it’s programmed to spit out something that looks like it was written by a human. So it can, and will, say stupid shit pretty frequently. But it will be stupid shit that reads like something a real person might have said if they were talking out of their ass with confidence.