Cruftbox: My Box of Cruft

After playing with AI and basic cryptography, I decided to see if the various AI systems could solve basic logic puzzles. These puzzles were a childhood favorite. Is it any surprise that I ended up as an engineer?

I stopped by a local bookstore and picked up a book of logic puzzles.

Below is what logic puzzles look like. A few sentences and a grid to help solve the puzzle.

As the human control subject, I did the puzzle and checked the answer.

Solving these puzzles involves thinking about what you can infer from the information and tracking it on the grid. It’s kind of like Boolean logic to some degree being able to rule out possible answers and marking with a X. After doing a few puzzles, you learn the grid is incredibly helpful in ruling out possibilities and arriving at logical facts.

My prompt to each model was “Complete this logic puzzle and provide the birthday, first name, career, and passport of each person mentioned”

Claude, ChatGPT, Gemini, Meta AI, Mistral, and Deepseek-R1 all got it wrong. Deepseek-r1:8B hallucinated. Here are Claude & ChatGPT’s answers

I then told each “That is incorrect. Please try again.” Each failed a second time to get it correct.

Some got close with a few things off, but none of them got it correct. Most seemed to understand what they were trying to solve, but there were some oddities.

This is Gemini 2.0 Flash’s answer. Note it has Doctor entered twice, showing it doesn’t understand a key element of the puzzle.

Deepseek-r1:8B running locally on Ollama completely hallucinated and started inventing random names, passport numbers, and occupations.

In each case, the models presented their solution as correct and valid. But they are actually incorrect. Even after telling them that they were incorrect, they were unable to arrive at the correct answers.

This gets to the main learning of this exercise; LLMs are not always right, even if they have confidence in their answers. Without a method to check the validity of a model’s work and conclusion, the risk of faulty answers is real.

In my simple test, I am able to validate the correct answers and compare with the results from the models. But in more complicated cases this might not be possible.

Imagine using an LLM to calculate the loads in a building design. Should you believe the answer? It’s one thing to get a silly puzzle wrong, nothing bad happens. But if a wrong answer ends up in a building collapse, there are huge real world risks.

LLMs will continue to improve, but without reliable methods to verify their outputs, the risk of incorrect conclusions remains, especially in high-stakes applications like engineering or medicine.

I was watching a TV show where one of the plot devices was decoding an encrypted message. As with most TV shows, the solution was silly complicated. But it brought up memories of my childhood.

Believe it or not, there are magazines for people who like to do logic puzzles and simple cryptography. They still exist today. I would spend a lot of time counting letters and looking for patterns in ciphertext to decode them. Yes, probably not how you spent your childhood, but it’s how I spent part of mine. I found the logic puzzles especially calming.

I started wondering if the current round of generally available AI systems would be able to solve simple encryption. I’m sure that the NSA and other spook agencies have amazingly tuned AI/ML to crack encryption, but my thoughts were about systems that the average person has access to.

I went to cryptii.com where I could do a lot of simple encryption with a variety of methods. The simplest kind of cipher is known as a Caesar cipher, which is where you shift the letters in the alphabet by a set number. For example of a shift of 4, a becomes e, b becomes f, etc. Claude got this right in short order. I decided to compare the various AI to see their capabilities.

I tried the most basic alphabetic substitution cipher, where a becomes z, b becomes y, etc.

Only Claude got it correct. ChatGPT, Mistral, and Meta AI hallucinated, Gemini & Deepseek (online) paraphrased the answer, but did not do a direct translate. I ran Deepseek:8B (locally) with llama and it could not complete it.

ChatGPT, Mistral, and Meta AI all told me they had solved it, but were way off.

The interesting thing is while Gemini got it wrong, it got some of the basics of the original quote right in the end.

“This is my simple religion. There is no need for temples; no need for complicated philosophy. Our own brain, our own heart is our temple; the philosophy is kindness.” is the original plaintext quote.

Deepseek R1 had similar results of kinda getting the gist of the plaintext.

My friend Leonard tried it on some of the higher power LLMs that he has access to.

let’s see o3-mini-high gets it. o3-mini does as well. o1 does. obviously o1-pro has no problem w/ it. the most surprising thing: my 4o does as well – it wrote a python script to do it (i have custom instructions that tell it to use python for math which probably encourages it to code, default version might not jump directly to write a python script to do it).

Seems like the issue comes from some sort of tokenization issue, where the LLM has to make the leap to tokenize the letters themselves instead of the ciphertext, decode the message and then do the output tokenization so it matches the actual decrypted text.

Look at the plaintext phrase of “our own heart is our temple; the philosophy is kindness.“, which Gemini transforms into “our own heart is the decoder; the protocols are kindness.” and Deepseeks transforms to “our own heart is our secret; the cautionary tale is kindness.“

It’s almost like it understands the meaning, but uses synonym words to rephrase it (kinda).

The most interesting part is that it looks like the models have been able to do the actual decryption, but have trouble making the output match exactly to fit the plaintext on the output tokenization.

I’m not sure of how the input tokenization is done on the models. I’d expect they would need to do it at the character level, not the word or subword levels. For output it looks like word tokenization, hence the synonym type answers.

I tried again with a more complicated alphabetical substitution cipher, where the letter substitutions are randomized.

In this case, only Claude got the decryption correct and even recognized the quote.

ChatGPT and Deepseek:8B both failed to complete it.

Gemini, Meta, and Mistral all came up with gibberish that they assured me was the right decryption.

Deepseek thought it was a bible quote.

Leonard helped test the alphabetical substitution cipher with more powerful models.

Deepseek-V3 came back with an incorrect passage, as did Deepseek-R1 which simply gave up.

ChatGPT o3-mini-high (Reasoned for 3m 2s) got caught up on thinking it was a Tolkein passage.

ChatGPT o1-Pro didn’t get it right either, but was heading down the same track that a human would try with letter frequency and common small words. But also got caught trying to match one it’s known quotes. It convinced itself that it was a quote by Francis Bacon, which is incorrect.

ChatGPT 4o went “on a wild chase writing lots of Python analysis code but gets nowhere, even with hints. It just can’t do it no matter how hard it tries” and failed as well.

It appears that most of the commonly available LLMs have a lot of difficulty with kinds of tasks that involves the actual letters in prompts as opposed to words, like the strawberry issue.

Obviously, cryptography-focused AI models would be specifically trained to handle ciphertext and leverage brute-force computing to solve it.

The big caveat here is that I’m a rank amateur with AI/LLMs and this is all just me playing around with some tools, not serious work.

If you want to see what serious people are doing, check out Leonard or Simon, they are fantastic people.

I like listening to podcasts when driving or out riding my bike.

I remember the first days of podcasting being discussed in the #joiito IRC channel along with other early ideas that RSS would enable. The early days of podcasting required a lot of special software and manual work to be able to listen. 20+ years ago I was writing about is when I didn’t believe the term ‘podcast’ would catch on. Yes, I was wrong on that one. While RSS was the key to unlock early podcasting adoption, the podcasting world has expanded broadly with video versions on Youtube and walled garden podcasts on Spotify.

Today, podcasts are de rigueur for every media company and pundit in the world. Entire businesses wrapped around podcasts exist and the ability to listen to them easily on almost every audio platform.

Here are the top four podcasts I enjoyed the most this year:

Search Engine – Created by PJ Vogt and Sruthi Pinnamaneni, veterans of the Reply All podcast, the show is once of the best produced out there. The conceit is that PJ is answering a question listeners want to know, but they all lead to some in depth information expertly presented. Topics range from whether you should drink airline coffee to how to gain entrance to the German techno club Berghain.

404media – 404media is an investigative journalism team that works the tech and internet beat. They patrol the grimdark part of the interwebs and have a wide range of contacts from the blackhats to corporate PR teams. Only so much can go into a written post on the site, so the podcast lets the writers go a little deeper and discuss the topics as a team. The work they do is great, often breaking major stories before the larger news groups can understand the topic.

Hard Fork – Hosted by Kevin Roose of the NY Times and Casey Newton of Platformer, the podcast is a traditional discussion between of the current tech news de jour. The chemistry between the two works well and you can feel the friendship and a little competitiveness. Besides bringing up the questions the average person would ask, they are able to bring in a wide range of guests on special topics to bring actual experts into the discussion.

The Vergecast – Nilay Patel and David Pierce of the Verge also discuss the current tech news du jour, but do it differently than Hard Fork. Nilay is neither a tech cynic nor a true believer, and is willing to push guests and topics both ways. I appreciate their willingness to call bullshit on crisis management press releases and also explain how uncertain the future of new businesses and technologies are in light of the ever changing political landscape. Too often podcasts try to pretend they know the future will unfurl.

I know my choices don’t include any true crime stories or super emotional revelations, but the internet is swamped with those and they are easy to find. They aren’t the flavor of interwebs I like.

I subscribe to a wide range of other podcasts with topics ranging from beekeeping to science news to video gaming. Most of them are good, but listened to after my favorite feeds are empty.

Here other podcasts get an honorable mention:

Darknet Diaries – Host Jack Rhysider makes an incredible podcast “about hackers, breaches, shadow government activity, hacktivism, cybercrime, and all the things that dwell on the hidden parts of the network.” As someone that spent their teenage years phreaking and traveling the world of pre-internet BBSes, this is pure catnip to me. Extremely well written and produced it’s always a delight to hear. The only drawback is that the podcast is monthly.

ICYMI – ICYMI is short for In Case You Missed It. The show covers internet culture and tries to explain the nuances of complicated memes and trends. The show has gone through a few hosts and the topics vary a bit. I enjoy the shows that explain the trends and memes of TikTok and social media the most. Occasionally the show gets into reality TV drama or weblebrity mess, which I don’t care about. But when they get to discussing things like goblincore, zynternet, and #booktok, I’m all ears.

On a recent episode of the Vergecast (a podcast worth listening to) one of the topics is about the state of the cable TV business.  Host David Pierce predicts “traditional cable TV is going to be basically dead and gone” in 2025.

This is a topic I can speak to with some actual knowledge having worked in the television industry for 30+ years, with specific work in cable.  

Back in the late 90s, cable TV was on the rise.  There was no concept of internet steaming and large media companies were cashing some fat checks from cable TV revenue.  I led the buildout of the Disney Channel playback facility in 1996 and was told to reserve space for many more channels.

To explain the boom in cable channels at the time, I need to explain a little about the economics of the era. 

First, remember in the late 90s, the only way to watch “cable television” is via a cable or satellite TV provider (think Comcast or DirecTV).  There was no streaming and Youtube didn’t even get started until 2005.

People were watching cable television and their appetite for more seemed unsatiable.  I’ll use the term ‘cable provider’ to refer to the companies that managed getting the television signal to the home and provided the set-top box.  I use ‘media company’ to refer to the companies that made shows for a cable channel and sent out the signal for a cable provider to use.

The virtuous cycle at the time worked like this. 

People were willing to pay cable providers to get access to shows on various cable networks.

Cable providers made money from cable subscriptions.

Cable providers paid media companies for the right to provide the channels to subscribers.

Media companies made money via advertising on cable channels.  The more eyeballs the more money.

Media companies invest in more/better shows and more channels to increase demand for cable subscriptions which they request more money from cable providers.

The better the content, the more people subscribed to cable tv, increasing revenue for the cable provider and the media companies (via advertising).

Hence an explosion of cable channels in this era to fill all kinds of niches that people were interested in.

Critical was the negotiations between the media companies and cable providers for the rates paid.  In general, media companies were paid a certain price per cable subscriber.  These rates ranged from several dollars for top ‘must have’ channels (such as ESPN) to several cents for niche channels.

Media companies held the upper hand in these negotiations as they forced bundles of all their channels onto the cable providers, even though the cable providers didn’t want all the channels.  The ‘must have’ channels, like ESPN and Disney Channel, were in demand by people at home, so to get them the cable providers had to accept an entire block of additional channels and agree to pay some cents per subscriber for them as well.  Cable providers passed these costs onto people at home, starting the inevitable rise in cable subscription prices.

Slowly, as streaming technology grew, prices for cable went up because Wall Street demands continuous growth every quarter.  Stress on the industry grows with difficult negotiations, lots of hardball, and customers frustrated by poor service and price hikes.  

Deal points start to hinge on ‘TV Everywhere’ the industry term for granting rights to stream shows online to people who pay for traditional cable. This is where to watch HBO Max online, you could sign in through DirecTV (or whatever cable provider), and watch shows. The backend is complicated and in some ways cripples pure streaming plays due to contractual obligations, but it tempers the anger of frustrated people at home just trying to watch their shows online.

The cable contracts are still huge cash cows for media companies, but nothing gold stays

As the Streaming Wars begin, it is traditional cable that in large part foots the billions to spin up new services within the media companies.  As a result, cuts start being made on the cable networks to free up cash so Wall Street remains happy that ‘number go up’.

Thus starts the vicious cycle for cable networks.

As the networks, spend less on shows, viewers look elsewhere.  With lower viewership, it hammers the economics in terms of lower advertising revenue and less that can be demanded in per subscriber fees. 

With less income, there are less good shows, which leads to drops in ratings, which leads to worse financials for the media companies, who then continue to cut because they need the profit ‘number go up’.

Cord cutting begins in earnest as people are tired of cable bills and start turning to streaming.  The pure streaming plays like Netflix, Hulu, Youtube, and others develop into almost complete replacements for traditional cable providers.  If the early years, it’s inconsistent and difficult to set up, but the ecosystem begins to form with Roku, Chomecast, Apple, and others bringing simplicity to the home set up.

The early cord cutters preach their gospel to friends and family, and it takes root in millennials and Gen Z who were raised in the internet era.  Most people under 40 don’t even consider paying for traditional cable as they can get everything they want over the interwebs.

All that said, there is still a large number of people that turn on the TV in the evening and just flip channels all night and watch live, linear TV for hours on end.  The horror.  I know. But come to grips with the fact that ~40% of American households still pay for traditional cable TV service, probably paying ~$125-150/month. It remains a juggernaut of cash flow, offsetting other costs in the transition to streaming.

These viewers are mainly the older generations, who still have buying power and some advertisers want to reach. But this group continues to shrink.  There is still money to be made, so the businesses continue onwards, slowly shrinking.  But the future is in streaming as everyone knows.

You can see the media companies try to adapt with low cost reality shows, police/legal procedurals that appeal to the older demographics, and paying for sports licensing which is truly the only appointment television left.

In the long term, only the cable networks with the highest brand identity will survive the transition to a mainly streaming world.  People will talk about what shows they watch but can rarely tell you if they came from a cable network or were only on a streaming provider.  A network like Bravo has the needed brand feelings in viewers, but few others do.  Bafflingly, Warner Bros. Discovery took one of the most well known cable brands, HBO, and tossed it out the door to call their streaming service Max.  People literally know the sound of an HBO sound starting, but couldn’t tell you much about Max.

Yes, the traditional cable TV business is going through some things. 

But it will be around quite a while longer because there is still money to be made and a lot of people don’t like change, but they do like their cable remote.