books

Fine-Tuning an AI – Part I

by

This post (Part I) walks you through a Git repo I put together to allow anyone to fine-tune a model to write emails in their own ‘voice’.

I did this to use a small model that I could run at home to help me draft emails that didn’t suck.

You don’t need a hefty GPU, but you will need a credit card to rent one on vast.ai for the training (note: affiliate link – I will benefit if you sign up and spend). It costs me about $1 per run to create a useful model that I can use on my home (non-GPU) server.

Part II will dive deeper into how the script works and what’s going on.

What It Does

At the very highest level, the script creates a new large language model (LLM), building from a base model, and training on sample emails you give it. Here’s some sample output (truncated to 40 tokens):

INPUT:
Write an email to a friend arranging a drink:
OUTPUT:
Hey, how are you? I was thinking of going out for a drink this evening. Are you free? Let me know what time and where we could meet.
Ian
On 10 Jul

You can see how it’s put my name (Ian) at the bottom and a (spurious) date, because that’s why a typical email looks like from me.

Compare this to the ‘base’ model this was built from (Qwen/Qwen2.5-3B-Instruct), which gives a far more ‘generic’ and formal response (also truncated to 40 tokens):

INPUT:
Write an email to a friend arranging a drink:
OUTPUT:
The date is 10/24/10 at 6:00pm.
Subject: Exciting Plans for Our Drink Date!

Dear [Friend's Name],

I hope

That, by contrast, does not sound like me at all.

…in a bit more detail

Here’s what the script (run_fine_tune.sh) does:

  • Picks the cheapest available GPU from vast.ai that fits the criteria needed, and provisions it
  • Copies up a script to run on the machine (build_and_push_model.sh), the gmail samples you want to train it on (compressed), and a huggingface token (to store the resulting models)
  • Runs build_and_push_model.sh on the GPU, which:
    • Installs the needed software (LLaMA-Factory, llama.cpp)
    • Logs into huggingface
    • Creates the huggingface repos (if needed)
    • Trains the new model
    • Merges the new model
    • Converts the new model to .gguf format (allowing it to be run in llama.cpp)
    • Quantizes the new model (so it can be run more easily on a non-GPU machine)
    • Pushes the merged model, and the .gguf, to huggingface repos
  • Destroys the vast.ai instance (asking you first)

How To Run It

The latest instructions will be maintained here:

  1. Clone the repo, and navigate to root folder
  2. Set up the virtual environment, install required pips
  3. Use Google Takeout to download your sent email to a .mbox file. You probably only need your ‘Sent’ emails, as they contain your responses. If you don’t use gmail, the script expects a .mbox file. Then run the python script to extract the emails and put them in the correct format. Then compress the result with xz
  4. Get a huggingface write token
  5. Set up vast.ai api key
  6. Run the script
    • Here’s an example invocation, with placeholders in caps:

./run_fine_tune.sh --max-steps 1 --hf-repo qwen2.5-3b-YOUR_NAME_sft --hf-user YOUR_HUGGING_FACE_ACCOUNT

If you like this, you might like one of my books:
Learn Bash the Hard Way
Learn Git the Hard Way
Learn Terraform the Hard Way

If you feel this was useful, consider buying me a coffee to say thanks!

I’m Writing a New Book!

by

I’m excited to announce that I’m writing a new book for O’Reilly:

Follow the Money: Understanding the Financial Architecture of Software.

This is a project that’s been quietly brewing for several years. 👇

The book’s primary goal is to help those engineers and managers whose efforts to change their organisations with technology hit the buffers as they scale. The direct application of sweat and resultant problem-solving worked well when you’re a more junior engineer, but as you try to make more fundamental and wide-ranging change seems to work less and less effectively. This book is intended to cast light on the invisible barriers that block your progress, and give you the tools to understand how to change them.

Its secondary goal is to be a more practical guide to making change within technical organisations, and even across wider businesses. While many books and blog posts exist on what the ideal state of various aspects of your organisation (culture, team structure, test practices, platform technologies etc) not enough of them give practical advice on how to get there, which is a far messier process. It might even help your career by allowing you to see how why – and just how challenging – it will be to get there, and perhaps consider setting yourself up for success elsewhere.

I hope the book will distil the experiences I’ve had over the past five years as a Could Native consultant with Container Solutions, and the last 30 years as an engineer, to give the reader a head start in navigating these murky waters.

The book will be replete with practical advice and real-world case studies to illustrate and provide inspiration (and caution) about what lies ahead for those that want to make positive change.

I hope you’ll join me on this journey. If these themes resonate, stay tuned. I’ll be sharing updates, excerpts, and insights here along the way. In the meantime, this video (‘How Money Flows Trump Conway’s Law) shares some of my more embryonic thoughts from a couple of years ago.

Your thoughts, questions, and experiences are always welcome!

Let’s follow the money together.