Cruftbox: My Box of Cruft

After playing with AI and basic cryptography, I decided to see if the various AI systems could solve basic logic puzzles. These puzzles were a childhood favorite. Is it any surprise that I ended up as an engineer?

I stopped by a local bookstore and picked up a book of logic puzzles.

Below is what logic puzzles look like. A few sentences and a grid to help solve the puzzle.

As the human control subject, I did the puzzle and checked the answer.

Solving these puzzles involves thinking about what you can infer from the information and tracking it on the grid. It’s kind of like Boolean logic to some degree being able to rule out possible answers and marking with a X. After doing a few puzzles, you learn the grid is incredibly helpful in ruling out possibilities and arriving at logical facts.

My prompt to each model was “Complete this logic puzzle and provide the birthday, first name, career, and passport of each person mentioned”

Claude, ChatGPT, Gemini, Meta AI, Mistral, and Deepseek-R1 all got it wrong. Deepseek-r1:8B hallucinated. Here are Claude & ChatGPT’s answers

I then told each “That is incorrect. Please try again.” Each failed a second time to get it correct.

Some got close with a few things off, but none of them got it correct. Most seemed to understand what they were trying to solve, but there were some oddities.

This is Gemini 2.0 Flash’s answer. Note it has Doctor entered twice, showing it doesn’t understand a key element of the puzzle.

Deepseek-r1:8B running locally on Ollama completely hallucinated and started inventing random names, passport numbers, and occupations.

In each case, the models presented their solution as correct and valid. But they are actually incorrect. Even after telling them that they were incorrect, they were unable to arrive at the correct answers.

This gets to the main learning of this exercise; LLMs are not always right, even if they have confidence in their answers. Without a method to check the validity of a model’s work and conclusion, the risk of faulty answers is real.

In my simple test, I am able to validate the correct answers and compare with the results from the models. But in more complicated cases this might not be possible.

Imagine using an LLM to calculate the loads in a building design. Should you believe the answer? It’s one thing to get a silly puzzle wrong, nothing bad happens. But if a wrong answer ends up in a building collapse, there are huge real world risks.

LLMs will continue to improve, but without reliable methods to verify their outputs, the risk of incorrect conclusions remains, especially in high-stakes applications like engineering or medicine.

I was watching a TV show where one of the plot devices was decoding an encrypted message. As with most TV shows, the solution was silly complicated. But it brought up memories of my childhood.

Believe it or not, there are magazines for people who like to do logic puzzles and simple cryptography. They still exist today. I would spend a lot of time counting letters and looking for patterns in ciphertext to decode them. Yes, probably not how you spent your childhood, but it’s how I spent part of mine. I found the logic puzzles especially calming.

I started wondering if the current round of generally available AI systems would be able to solve simple encryption. I’m sure that the NSA and other spook agencies have amazingly tuned AI/ML to crack encryption, but my thoughts were about systems that the average person has access to.

I went to cryptii.com where I could do a lot of simple encryption with a variety of methods. The simplest kind of cipher is known as a Caesar cipher, which is where you shift the letters in the alphabet by a set number. For example of a shift of 4, a becomes e, b becomes f, etc. Claude got this right in short order. I decided to compare the various AI to see their capabilities.

I tried the most basic alphabetic substitution cipher, where a becomes z, b becomes y, etc.

Only Claude got it correct. ChatGPT, Mistral, and Meta AI hallucinated, Gemini & Deepseek (online) paraphrased the answer, but did not do a direct translate. I ran Deepseek:8B (locally) with llama and it could not complete it.

ChatGPT, Mistral, and Meta AI all told me they had solved it, but were way off.

The interesting thing is while Gemini got it wrong, it got some of the basics of the original quote right in the end.

“This is my simple religion. There is no need for temples; no need for complicated philosophy. Our own brain, our own heart is our temple; the philosophy is kindness.” is the original plaintext quote.

Deepseek R1 had similar results of kinda getting the gist of the plaintext.

My friend Leonard tried it on some of the higher power LLMs that he has access to.

let’s see o3-mini-high gets it. o3-mini does as well. o1 does. obviously o1-pro has no problem w/ it. the most surprising thing: my 4o does as well – it wrote a python script to do it (i have custom instructions that tell it to use python for math which probably encourages it to code, default version might not jump directly to write a python script to do it).

Seems like the issue comes from some sort of tokenization issue, where the LLM has to make the leap to tokenize the letters themselves instead of the ciphertext, decode the message and then do the output tokenization so it matches the actual decrypted text.

Look at the plaintext phrase of “our own heart is our temple; the philosophy is kindness.“, which Gemini transforms into “our own heart is the decoder; the protocols are kindness.” and Deepseeks transforms to “our own heart is our secret; the cautionary tale is kindness.“

It’s almost like it understands the meaning, but uses synonym words to rephrase it (kinda).

The most interesting part is that it looks like the models have been able to do the actual decryption, but have trouble making the output match exactly to fit the plaintext on the output tokenization.

I’m not sure of how the input tokenization is done on the models. I’d expect they would need to do it at the character level, not the word or subword levels. For output it looks like word tokenization, hence the synonym type answers.

I tried again with a more complicated alphabetical substitution cipher, where the letter substitutions are randomized.

In this case, only Claude got the decryption correct and even recognized the quote.

ChatGPT and Deepseek:8B both failed to complete it.

Gemini, Meta, and Mistral all came up with gibberish that they assured me was the right decryption.

Deepseek thought it was a bible quote.

Leonard helped test the alphabetical substitution cipher with more powerful models.

Deepseek-V3 came back with an incorrect passage, as did Deepseek-R1 which simply gave up.

ChatGPT o3-mini-high (Reasoned for 3m 2s) got caught up on thinking it was a Tolkein passage.

ChatGPT o1-Pro didn’t get it right either, but was heading down the same track that a human would try with letter frequency and common small words. But also got caught trying to match one it’s known quotes. It convinced itself that it was a quote by Francis Bacon, which is incorrect.

ChatGPT 4o went “on a wild chase writing lots of Python analysis code but gets nowhere, even with hints. It just can’t do it no matter how hard it tries” and failed as well.

It appears that most of the commonly available LLMs have a lot of difficulty with kinds of tasks that involves the actual letters in prompts as opposed to words, like the strawberry issue.

Obviously, cryptography-focused AI models would be specifically trained to handle ciphertext and leverage brute-force computing to solve it.

The big caveat here is that I’m a rank amateur with AI/LLMs and this is all just me playing around with some tools, not serious work.

If you want to see what serious people are doing, check out Leonard or Simon, they are fantastic people.