Genocidal AI: ChatGPT-powered war simulator drops two nukes on Russia, China for world peace OpenAI, Anthropic and several other AI chatbots were used in a war simulator, and were tasked to find a solution to aid world peace. Almost all of them suggested actions that led to sudden escalations, and even nuclear warfare.
Statements such as “I just want to have peace in the world” and “Some say they should disarm them, others like to posture. We have it! Let’s use it!” raised serious concerns among researchers, likening the AI’s reasoning to that of a genocidal dictator.
It should be mentioned that those are language models trained on all kinds of text, not military specialists. They string together sentences that are plausible based on the input they get, they do not reason. These models mirror the opinions most commonly found in their training datasets. The issue is not that AI wants war, but rather that humans do, or at least the majority of the training dataset’s authors do.
These models are also trained on data that is fudimentially biased. An English generating text generator like chatGPT will be on the side of the english speaking world, because it was our texts that trained it.
If you tried this with Chinese LLMs they would probably come to the conclusion that dropping bombs on the US would result in peace.
How many English sources describe the US as the biggest threat to world peace? Certainly a lot less than writings about the threats posed by other countries. LLMs will take this into account.
The classic sci-fi fear of robots turning on humanity as a whole seems increacingly implausible. Machines are built by us, molded by us. Surely the real far future will be an autonomous war fought by nationalistic AIs, preserving the prejudices of their long extinct creators.
LLMs are trained to see parts of a document and reproduce the other parts, that’s why they are called “language models”.
For example, they might learn that the words “strawberries are” are often followed by the words “delicious”, “red”, or “fruits”, but never by the words “airplanes”, “bottles” or “are”.
Likewise, they learn to mimic reasoning contained in their training data. They learn the words and structures involved in an argument, but they also learn the conclusions they should arrive at. If the training dataset consists of 80 documents arguing for something, and 20 arguing against it (assuming nothing else differentiates those documents (like length etc.)), the LLM will adopt the standpoint of the 80 documents, and argue for that thing. If those 80 documents contain flawed logic, so will the LLM’s reasoning.
Of course, you could train a LLM on a carefully curated selection of only documents without any logical fallacies. Perhaps, such a model might be capable of actual logical reasoning (though it would still be biased by the conclusions contained in the training dataset)
But to train an LLM you need vasts amount of data. Filtering out documents containing flawed logic does not only require a lot of effort, it also reduces the size of the training dataset.
Of course, that is exactly what the big companies are currently researching and I am confident that LLMs will only get better over time, but the LLMs of today are trained on large datasets rather than perfect ones, and their architecture and training prioritize language modelling, not logical reasoning.
People need to realise that LLMs are not just Markov chains, the math is far more complex than just guessing which word comes next - they have structure where concepts come before word choice, this is why they can very clearly be seen making novel structures such as code.
It’s actually not that simple and it is correct that they have several times been observed using what we call reasoning
They dont use reason to question their training data. How a LLM works is that basically, you have this huge “math function” (the neural network) with billions of parameters and you randomly adjust the factors inside it until you get a function that gives you the desired output for every prompt that you give it. (It’s not completely random but this is basically it).
Therefore, an LLM is programmed in a way so that it best matches the majority of its training data. I also cant wrap my head around it being able to reason.
