My attempt to train an LLM at home, and why it failed

Last week I was scrolling TikTok, as one does, and saw this video by Sangeetha Bhatath, a software engineer. She was discussing that Andrej Karpathy had released the code for microGPT, an extremely simple version of the code used to train large language models. Karpathy is a co-founder of OpenAI and one of the leading thinkers in the space.

TikTok – Make Your Day

No Description

Sangeetha’s point is that you can try training a LLM yourself, and see what’s in the inside the black box to some degree. I was intrigued and decided to give it a try.

After a bit of chatting with Claude (the web chat AI from Anthropic), we agreed to use nanoGPT as it was able to take advantage of GPU processing. As a PC gamer, I have a reasonable video card (Nvidia 4070 Super w/12GB VRAM) that would greatly speed the training. GPUs do a lot of vector math to make video games work and coincidentally LLM training is basically the same kind of vector math. I hated linear algebra in engineering school, so I’m glad we have chips to do this for me.

Interior view of a gaming PC build showing an ASUS motherboard with Corsair CPU cooler, green RGB RAM modules, GeForce RTX graphics card, and custom cable management with braided cables inside a case with RGB lighting.


The plan was to use the GPT-2 weights that are publicly available with as much data as I could gather of my own writing and speaking. In short, a plan to make a Cruftbot or CruftGPT. Claude made a detailed four phase plan that I could understand and was clear direction for Claude Code (Anthropic’s focused developer AI app) to execute.

The text you used to train a LLM is reflected in the way the LLM writes. Train a lot of Shakespeare, you get a LLM that talks like an Elizabethan. Train a lot of legal documents, you get a LLM that talks like a lawyer.

I’ve been in the interwebs for a long time and have 25 years of posting and over 300 videos of my various antics. Claude helped me write several scripts to scrape data from my weblog, Medium stories, Bluesky posts, and transcripts of my videos. Reddit has an export function, which made that easy. I have a lot of posts on Twitter, but I haven’t been posting there for a couple years now. It used to be easy to get an export of posts, but under the current management it’s extremely difficult.

I set Claude Code to work on setting up the NanoGPT code on my desktop. As an aside, wsl2 (Ubuntu linux) under Windows works very well. I fed the personal data to Claude Code and it formatted it for me. 25+ years on the internet equaling 699K tokens of data. Good, but not great.

Another aside: LLMs process text using tokens, which are the numerical building blocks of text input. Instead of reading full words, a tokenizer breaks text down into common chunks of characters. For example, the word ‘apple’ might be one token, while a complex word like ‘bioluminescence’ might be split into three or four tokens. The tokenizer assigns each unique chunk a specific number, the word ‘apple’ might be ‘27149’.

Training is essentially the LLM learning the mathematical relationships between these numbers. Since computers excel at math but don’t ‘read’ like humans, turning language into a giant game of statistics and geometry (technically it’s vector math) is what makes the magic happen.

Claude started a few training runs and tried both GPT-2 small (124M) and GPT-2 medium (345M) parameter sets to see what worked best with my personal dataset. After a bit of GPU time, it found the GPT-2 medium worked best to provide the best ‘val loss trajectory’. I learned that ‘val loss trajectory’ is tracking the validation loss number, which kinda means how well the personal data is overlaying with the base language data.

Since I want CruftBot to sound like me, it’s important the training results in my personal data being more apparent than the base language that the GPT-2 set provides.

Before bed, I told Claude to continue training and to continue without asking me for approval. The GPU was pegged at 99% but not overheating, which was great.

The next morning the training was done and Claude stood up Gradio to act as a UI with CruftBot.

The results were underwhelming.

Here's detailed alt text for this image: Alt text: Screenshot of a web application titled "cruftbox GPT-2" with a dark gray background and orange accent color. The subtitle reads "Fine-tuned on cruftbox.com blog posts, Reddit comments, Bluesky posts, and YouTube transcripts." The interface shows a "Prompt" input text area containing the text "Checked the hives this morning and" followed by a large orange submit button (label not visible, cropped at right edge). Below that is a "Generated text" section displaying the AI-generated continuation of the prompt in white text on a slightly lighter dark gray background: "Checked the hives this morning and they are strong. I'm not sure what happened to the bees but they shouldn't be laying so much in the first place. I've been keeping bees in a jar in the crawlspace behind the garage since last June and keeping the temperature as low as I can" The generated text is nonsensical and humorous, demonstrating the imperfect output of a fine-tuned language model. The text is cut off at the bottom, suggesting it continues beyond the visible area.


The output used words I use, but was put together in nonsense fashion. You could see CruftBot trying, but it was just guessing at words.

Claude explained “This is the fundamental limitation of a fine-tuned model this size: it’s not a knowledge model or a chat assistant, it’s a text completion engine trained on your writing patterns. It doesn’t understand questions, it just continues text in a direction that statistically resembles your corpus.”

Claude went on to explain that what I really needed was a lot more tokens of my own data.

Screenshot of a text response (likely from an AI assistant) displayed in white serif text on a black background. The response analyzes the poor output of a fine-tuned GPT-2 model. The text reads: "Yeah that's not good. 'Bees in a jar in the crawlspace' and 'my daughter was born in March' are confabulations — the model is mixing beekeeping vocabulary with unrelated fragments from your corpus and inventing details that aren't true. This is a ceiling you've hit with GPT-2 small at 699K tokens. The model has learned surface patterns — beekeeping words, first-person voice, casual tone — but it doesn't have enough depth to maintain coherent factual continuity within a passage. It strings plausible-sounding sentences together without any underlying understanding. To get meaningfully better results you'd need one or more of:" Followed by three bold-labeled recommendations: More data — ideally 2M+ tokens, heavily weighted toward long-form writing rather than short social media posts A larger base model — GPT-2 medium or large would have more capacity to maintain coherence, though you'd hit the same ceiling eventually A different approach entirely — rather than fine-tuning GPT-2 from scratch, using a modern instruction-tuned model like Llama 3 or Mistral with your writing as context or LoRA fine-tuning would produce dramatically better results. These models already understand the world — you'd just be steering their voice toward yours


My own data means things I’ve written, talks I’ve given, and videos I’ve made. Asking for triple of what it took me 30 years on the internet to write, and I’m prolific compared to most netizens, is humbling. There just doesn’t exist three times more ‘me’ of data out there.

In short, I learned it’s just guessing words based on patterns of tokens in the data it was trained on and it needs a lot more data to train on. There is some truth to the idea that AIs are ‘word guessing machines’ but at the leading edge they guess as well as almost any expert human would on topics.

If I really wanted to take this further, there are other approaches to improve the result, but in the end they would all pale in comparison to the current frontier models that you can try for free.

There’s a huge value in doing technical things yourself and seeing what is involved. I learned a tremendous amount about the basics of LLM training and what kind of issues would be involved with scaling.

When I worked at NBC, we used the same Nvidia A100 & H200 cards for video editing that are now used for LLM training. They are enormously powerful GPUs. At the time, our competition in buying them was from cryptocurrency groups, not AI companies. The idea that thousands of these cards are needed to train the frontier AI modules shows me the gigantic amount of tokens that are crunched to get today’s AI bots.

Looking at this from a professional point of view, it’s easy to extrapolate from my experiment how a business might want to build its own LLM, trained on a large corpus of knowledge important to that business. It’s probably a spreadsheet of costs comparing doing it yourself with servers, GPUs, and data centers compared to paying an existing AI company to train your data on top of their models. On top of all that, does the cost of a well trained AI system pay for itself in terms of productivity and improvements? The answer on that is still undetermined, despite the current hype cycle.

We are all in the very early days of AI, despite the feeling that it’s taking over our personal worlds and most businesses. My 24-hour experiment only scratched the surface and it’s clear there’s a long way to go before any of us (developers, businesses, or society) truly understand how this technology will reshape our world.

If you are technically minded, do yourself the favor and try training your own model. It won’t end up being very usable, but you will learn a lot.

26 Years of Blogging: From Dial-Up to AI Slop

Today marks the 26th anniversary of starting this very weblog. I had a personal web site since 1997, but 2000 marks when I began traditional blogging.

My first post was celebrating getting a blogging system known as NewsPro running. In the beginning I was mainly blogging about Ultima Online, the MMO I was playing at the time.

The internet was very different at the time. There was no Facebook, Twitter, Instagram, or Wikipedia. There were no smartphones, streaming video, podcasts, or mainstream broadband. Storage was measured in megabytes, not gigabytes.

In January 2000, we were still in the era of dial-up internet, desktop computers, flip phones, DVDs, and broadcast television.

The change over the last 26 years is mind-blowing when you step back and look at it.

I changed over the years as well, going from the father of toddlers to an empty nester. My blogging evolved with me, from video games and daily routines to writing about the nascent social media and blogging scene.

Sometime in 2002 I moved from NewsPro to MovableType. At the time, the software was revolutionary.

Beyond this personal weblog, I was also experimenting in the corporate world, getting my maintenance team to write about what they did on their shifts online as opposed to paper logbooks.

I remember sending a check for $200 to Ben & Mena Trott to license the Walt Disney Company to use Movable Type. I spoke at conferences about using blogging in the workplace.

When digital cameras became affordable I was able to incorporate more images and even post video.

My first page to go viral was about making a smoker from a trash can. Another was about hacking my Tivo.

In those days, comments were de rigueur on weblogs and the first appearance of spam and bad actors arrived, and the endless attempts to counter them were met with varying levels of success. At the time, the dopamine from nice comments outweighed the headache of spammers.

I wrote about everything from experiments I was doing making gunpowder to keeping track of Halloween costumes.

As the world of microblogging on Twitter, Facebook, and other sites took off, my blogging started to dwindle.

I started posting on Medium for a bit, but was mainly stuck on Twitter.

Making video content became easier and easier and I started making videos about my hobbies like beekeeping and video games on YouTube.

Places to share on Reddit, Discords, and Slacks were abundant. But you can be at the mercy of moderators of varying attitudes and commenters that try to make you feel bad.

As social media became a slurry of AI slop, influencers, and bots, I realized I needed a space I could actually control, one not beholden to CEOs chasing the latest hype cycle.

In 2024, thanks to help from my friend Greg, we got this blog up and running again. Greg helped me move to WordPress, which is the de facto standard these days.

I still make videos and post them here, but also make actual blog posts about things that interest me.

I’m not trying to make money or become a dadfluencer, just happy to have a little space on the net for myself.

There is no amazing revelation or realization after 26 years of blogging.

I have no idea if people are reading what I write, and it really doesn’t matter.

It gets the ideas out of the whirlwind in my head so I can make space for new things.

I’m just happy to keep my little corner of the interwebs tidy.

Twenty-six years on, I’m still writing, not because it’s strategic or visible, but because I enjoy it.

We made a zine

As holiday cards began to roll in, Michele, my wife, and I discussed whether we were going to do a Christmas card ourselves. After being married 30+ years with our kids grown and out of the house, our life doesn’t lead to photos of far-flung travel and excitement.

Recently, our neighbor Emily put out a zine featuring my 20 years of Halloween costume data. I was honored and thrilled to see an actual paper zine. For those that don’t know, a zine is a small self-published booklet often made by hand.

When I saw it, I had thoughts about what I would put into a zine.

Two copies of the Haddonfield Journal vol 5 special edition featuring "The Pusateri Archive (2005-2025)" with Halloween pumpkin icons, alongside a detailed article about Michael Pusateri's 20-year Halloween tradition in Los Angeles.

We considered various card concepts until the idea clicked: make our own zine. Not a holiday themed one, but just filled with work and ideas from both of us.

I found a great template on Sinoun Chea’s site. We went with an eight page zine template.

We started filling up the pages with small bits we thought might be interesting to the people on our Christmas card list. Some images of Michele’s work, several best of 2025 lists, a short essay about buying a smart TV, and a few things we had no idea if others would find interesting.

Once we finalized the content and design, I looked into printing. A few pages have color images and I found out quickly that color printing is still pretty expensive at local print shops. At over $2 in printing costs per issue, the total was adding up quickly to a big number. We decided to double down and buy our own color laser printer, as we’d probably be making zines again, not just for the holidays.

I felt very DIY as I folded and stapled the issues together. The physical work of putting a zine together is far more rewarding than uploading images to a print company on the internet. Michele helped with the card list and labels. We got a lot of feedback from friends and family and they loved it. One couple told us they’re going to make one themselves.

Two copies of a personal zine called "Cruft Manor Volume 1" spread on a blue tie-dye fabric, featuring content about platform decay theory, 2025 entertainment lists, and accompanying graphics.

If you’d like a copy, you can print it from this PDF.

When printing, choose two-sided printing and flip on the short edge.

A printing options interface showing "Two-sided" settings with "Print on both sides" checkbox checked and a dropdown menu set to "Flip on short edge".

We had a blast making a zine. If you have some spare time, you might consider making one yourself.

Going analog in a digital world is refreshing.

Generating Alt Text with a right-click

I’ve been trying to be more consistent with alt text, but let’s be honest, writing it manually can be a chore.

Most AI chatbots do a good job of generating the alt text for me if I give them an image. They often catch details I overlook and add in information that I, as a human, would typically leave out.

To make it simpler, I wanted to right-click any image to automate getting AI-generated alt text.

It took about 15 minutes to build.

A Windows file explorer window showing thumbnail images of various photos and files, with a right-click context menu open displaying options like "Open," "Edit with Paint," "Generate Alt Text," and other file management tool

I used Claude Code to make a small Python script to send the image to the Claude API and return the result to a local web page to copy the text easily.

A web page showing AI generated alt text of an image with a Copy to Clipboard button.

It was strikingly simple to do this. The script operates on Windows 11 using the Claude API.

To use it, you need to run Python, a few dependencies, and make a few registry edits to add the right-click menu option. Claude Code did this for me directly, but it can be done manually.

I had Claude Code create a GitHub package with documentation and uploaded it here: https://github.com/cruftbox/image-alt-text

If you are using MacOS or another LLM, it shouldn’t be hard to modify it to work with your preferred setup.

This kind of ‘vibe coding’ feels great to scratch my own itch with ideas that are idiosyncratic to me.

Proof by Cornelius Eady at the inauguration of Zohran Mamdani

I did my best to transcribe:

Proof by Cornelius Eady at the inauguration of Zohran Mamdani

Proof.

You have to imagine it.
Who said you were too dark, too large, too queer, too loud?
Who said you were too poor, too strange, too fat?

You have to imagine it.
Who said you must keep quiet?
Who heard your story, then rolled their eyes?
Who tried to change your name to invisible?

You’ve got to imagine.
Who heard your name and refused to pronounce it.
Who checked their watch and said, “Not now.”

James Baldwin wrote, “The place in which I’ll fit will not exist until I make it.”

New York City of invention, roiling town, refresher and renewer.
New York City of the real will.
The canyons whisper in a hundred tongues.

New York, where your lucky self waits for your arrival,
Where there is always soil for your root.

This is our time.
The taste of us, the spice of us,
the colors and the rhythms and the beats of us,
In the echo of our ancestors who made certain we know who we are.

City of insistence.
City of resistance.

You have to imagine an army that wins without firing a bullet,
A joy that wears down the rock of “no,”
Up from insults,
Up from blocked doors,
Up from trick bags,
Up from fear,
Up from shame,
Up from the way it was done before.

You have to imagine that space they said wasn’t yours.
That time they said you’d never own.
The invisible city lit on its way.

This moment is our proof.

A hard summer

I learned a lot about pain, pride, and empathy this summer.

The entire episode started with a Sunday walk in May with my wife Michele. She has a five mile loop she does and I decided to come along. I’d done it before.

On the walk, we climb a small hill, which didn’t seem much of a problem, but as we got down to the base of the hill to the flats, I started feeling a pain in my lower back and down my left leg. We stopped for a bit to stretch and continued on. The pain got worse and worse the further we went. Instead of stopping and getting a ride home, I forced myself to keep going, stopping every few hundred yards when the pain became intense. Even after the walk was over, the pain continued for several days. In hindsight, this was incredibly stupid.

My childhood lessons about perseverance and pushing through discomfort, values I was taught to prize, ultimately caused lasting harm. Being taught to be tough and endure difficulty is good, but taken too far can be damaging.

The doctor said it looked like sciatica and put me on a regimen of anti-inflammatory drugs and physical therapy. The pain slowly receded.

By June I was feeling better and went back to my usual tasks and tinkering. I decided to repaint my curb numbers and got to work sanding and painting. Bending over the curb was painful, but I was focused on getting the job done, and kept working. I got the final bits done and my lower back was screaming at me.

That evening the pain got worse and worse and I had trouble even getting comfortable to sleep. The next morning, even walking was extremely painful and I went to Urgent Care the first thing. The doctor was very direct, all she could do was prescribe painkillers. I’d have to see orthopedic doctors to see what the underlying issue was and how to fix it. They gave me a bunch of opioids and sent me on my way. I really don’t like taking opioids but needed them for a couple days to get through the pain and be able to sleep. It took days to get comfortable enough to walk more than a few steps.

In July, I tried everything from physical therapy and chiropractic work to acupuncture, finally culminating in an MRI and a steroid injection to help reduce the inflammation.

This all helped reduce the pain and I slowly increased my walking to be able to go a few hundred yards before having to stop. I was hopeful I was past the worst, but I was wrong.

At the end of July I received the report on the MRI and it basically said there was a fragment of a disc pressing on a nerve root at my L5-S1 vertebrae. Reading it, I realized the complexity of the problem and that it wasn’t going away with rest. After reading the MRI report, my physical therapist told me I’d likely need surgery and she didn’t want me to do more work until I’d seen the spine surgeon and got clearance.

Axial MRI scan of lumbar spine showing spinal canal stenosis. The main image displays a cross-sectional view with annotations pointing to a disc fragment compressing a nerve root and the location where the nerve root is being compressed. An inset image in the lower left shows a comparison view of a normal spinal canal for reference.

Slowly the pain diminished and I was able to do small things around the house. Toward the end of August, I was bending over the dishwasher to empty it and felt a twinge. The twinge shortly gave way to pain that kept increasing.

The pain kept escalating to a level I couldn’t believe. Words do not do justice to describe the mind-shattering and crippling pain in my back and down my left leg. There was only one position I could get relief, my back flat on the floor and my legs up in a L shape, on a chair or ottoman. Even with the opioids, even going to the bathroom was an exercise in agony, having to lie on the bathroom floor to recover.

I’ve led an active life and have broken multiple bones, torn ligaments, and had shoulder and knee surgery. I thought I knew what pain was, but again, I was wrong.

The revelation that this level of pain existed was eye-opening. I had heard people complain about back pain and having to stay home from work due to it, but never could imagine this is what back pain could be like.

After two days of this, Michele took me to the ER. She put down the back seats and I lay in the back of the car as she drove. The ER doctor shot me up with even more painkiller and told me the same thing I had been told previously, all he could do was give me more painkillers and anti-inflammatories. I’d have to see the spine surgeon to fix this.

Back home, I tried to rest and hoped the anti-inflammatories would help get me through to the appointment with the spine surgeon. Once again, I was wrong.

A few days after the ER visit I took a shower. Getting out, I was drying off and felt a twinge that rapidly became pain. Naked, I laid down on the floor, hoping to get the pain under control. But the pain just kept getting worse, reaching levels more intense than I had ever felt before. Even my recovery position and stress breathing did nothing. My brain was panicking because I knew I couldn’t endure this for long. Michele was with me and I told her to call the paramedics. We didn’t know what else to do.

Soon I could hear the sirens in the distance and before long the paramedics arrived. They did their best to extract me carefully from the bathroom and get me into the truck but I was hit with waves of pain. I answered their questions and explained the history as they placed an IV in my arm. They assured me they could help. And sure enough, they gave me a dose of IV fentanyl and the pain switched off, like they had flipped a switch. By the time I got to the emergency room, I was relaxed and in relative comfort.

Yet again, the ER doctor explained the limits of what they could do and before long I was home. More days of lying on my back followed.

I stopped taking the opioids three days after the ER visit. I found myself looking forward to my next pill and knew I had to stop.

A man with gray hair lies on his back on a patterned rug, taking a selfie from above. He wears a gray t-shirt with a graphic design featuring what appears to be sci-fi helmets or masks. Behind him, a large dog with tan and black coloring is sprawled out on the same rug, lying belly-up with legs extended. The image captures a relaxed moment of companionship between person and pet.
Flat on my back

In those days following the ER, lying on the floor and staring at the ceiling, my perspective on ‘strength’ began to shift.

Our medical insurance is good and we weren’t being saddled with debt, unlike so many others the poor or no insurance at all. Going through this kind of painful ordeal and worrying about the cost is why so many people self-medicate with drugs and alcohol.

I used to judge those who became addicted to prescription painkillers as weak-willed. However, experiencing this level of pain myself, and realizing the luxury of not having to work during it, has shown me that for many, opioids aren’t a choice, but a desperate necessity. I now see how easily anyone could fall into that cycle.

I’m ashamed it took me suffering personally to have empathy for the millions of other people dealing with chronic pain issues. The sad truth is that the arsenal of tools to deal with chronic pain too often comes down to ever-increasing amounts of addictive drugs.

My appointment with the spine surgeon was in September and we decided to proceed with a microdiscectomy to remove the disc fragment and “clean up” the surrounding area.

The surgery was on October 13th taking only about an hour. They did the entire procedure through a 1.5 inch incision. The surgeon, the OR team, and the medical technology were absolutely incredible.

There was about two days of pain afterwards, mainly from them cutting me open a bit, but the sciatic pain was gone.

After an entire summer of pain, it was gone. Gone like it never existed.

Realizing this filled me with enormous gratitude for everyone who helped me through this. Michele, my daughter Zoe and her boyfriend Christian, the nurses and doctors at the urgent care and emergency rooms, the surgical team, the inventors of the medical tools, the friends and family that let me know they were thinking of me. It’s still overwhelming now, even as I type this with tears in my eyes.

If you are dealing with back pain and hesitating to get help with it, I urge you to see a doctor and find out what can be done. Pain makes everything in life worse, even if you think you are managing it.

My recovery has gone well, gaining strength and confidence, but being careful not to create a new injury.

A man with gray hair stands in a warehouse, holding a small Christmas tree. He wears a dark blue textured sweater over jeans and has a slight smile. Behind him are industrial metal shelving units filled with storage bins and boxes. The concrete floor shows signs of wear, and the warehouse has high ceilings with exposed ductwork and beige walls.
Healed and lifting an enormous tree

Getting myself to think differently remains a challenge. While being able to endure some pain and hardship is important in life, taking it too far, as I had done, is a bad thing. Deciding to take two trips to bring in the groceries from the car instead of trying to do it in one, getting help to lift a heavy box, and limiting my time sitting in front of the computer don’t come easily. I have to make the conscious decision to not revert to my old ways of thinking about pain.

It was a hard summer, but I am grateful to come away changed, having more empathy for others.