What Does It Take to Build an AI Model? Let's Look at OLMo

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. When we talk about training AI models, it can sound like you just put some training data in, and some chatbot comes out. Or for open models, we just download one, run it via Ollama, and boom, magic happens. We get some output from all the hard work done by our own GPU. We only see the product, but we never see what is behind it. To see what actually goes into building a model, it helps to look at projects that publicly release not just the model, but also their code, data, configurations, evaluations, and other artifacts. So, to gain an understanding of how such AI models are being made, we will explore one such model. It's called OLMo. You can clone it from here: https://github.com/allenai/olmo When you open this repository, you can see many folders and files. Let's explore each part one by one from the POV of building an AI model. Step 1: You need data, which has to be cleaned and tidied up as well Before any of the training happens, you need to have sufficient data. Loads of data. Matter of fact, AI models don't continuously learn from new data once they are trained. When new data needs to be incorporated, the model needs to be updated or retrained. So, how do you know what data was used to train a model? Open the README, and you can see the dataset mixes used by OLMo-2. OLMo-mix-1124: the big, mostly web-based dataset used for the bulk of training. Dolmino-mix-1124: a smaller, high-quality, targeted dataset used later in training. Now, the next file you want to look at is inside configs/official-1124/provenance.csv. https://github.com/allenai/OLMo/blob/main/configs/official-1124/provenance.csv This gives more detailed information about the provenance of the datasets and data directories used by the project. Now that the data is sorted out, let's check the next thing. Step 2: Turn text into numbers: The tokenizer Go over to olmo/tokenizer.py for the tokenizer implementation, and olmo_data/tokenizers for the tokenizer files. If you didn't know earlier, models don't directly process words as words. They receive token IDs, which are integers. These IDs are then mapped into vectors that the neural network can work with. A tokenizer is the tool that breaks text into smaller units called tokens. These tokens aren't necessarily complete words. Depending on the tokenizer, a token can represent a whole word, part of a word, a character, whitespace, or another text unit. So the olmo/tokenizer.py file: https://github.com/allenai/OLMo/blob/main/olmo/tokenizer.py contains the logic for tokenizing text. The actual vocabulary and the mapping between tokens and their IDs are stored in the tokenizer artifacts. If we take one of the files under olmo_data/tokenizers, for example: https://raw.githubusercontent.com/allenai/OLMo/refs/heads/main/olmo_data/tokenizers/allenai_dolma2.json You can see the actual lookup table. Here, we map token strings to numbers. You can see mappings like: "let":1169, "DE":1170, "red":1171 Step 3: Design the architecture In olmo/model.py: https://github.com/allenai/OLMo/blob/main/olmo/model.py You can see a large file. This contains things like the Transformer definition, attention layers, feed-forward blocks, layer normalization, and how they all stack together. The things you see aren't necessarily built from scratch. They are built upon good examples and are combinations of existing techniques with their own tweaks. The OLMo implementation was adapted from MosaicML and minGPT. Step 4: Write the training loop Now we have the data and the model. Now we need something to connect both. That's where we have the training loop. Here we have olmo/train.py and olmo/optim.py. https://github.com/allenai/OLMo/blob/main/olmo/train.py https://github.com/allenai/OLMo/blob/main/olmo/optim.py In train.py, you can find the training loop. And in optim.py, you can find the optimizer. The training loop doesn't simply connect the data and model. It orchestrates things like running the model forward, calculating the loss, performing backpropagation, updating the model parameters, logging progress, and saving checkpoints. This is not a once-run-and-forget loop. Depending on the training run, it can perform millions or even billions of training steps across distributed hardware. Step 5: Configure the run For configurations, you can check configs/official-1124/ for a real released model. Or, if you want a toy example, check: configs/tiny/ Every training run needs precise settings: what learning rate to use, how many tokens to train on, which dataset files to use, and dozens of other knobs. The distributed training environment also needs to provide the required hardware and resources. Open the configs/ folder, and you'll find these choices stored as YAML files for different experiments and official model configurations. Step 6: Checkpoints In olmo/checkpoint.py, you can see the code that saves and loads the training progress. The project provides checkpoints in both OLMo and Hugging Face-compatible formats. Step 7: A second, shorter training stage OLMo doesn't stop after the large-scale training stage. For OLMo-2, there is a second, much smaller training stage using carefully chosen, targeted data, including the Dolmino-mix-1124 dataset from Step 1. The released OLMo-2 models used different Stage 2 setups, including multiple training runs. The resulting model checkpoints could then be merged, a process often referred to as model souping. You can find the relevant configurations by looking for Stage 2 configurations inside the config subfolder. So the basic idea is: After large-scale pretraining, the model continues training on a smaller, more targeted dataset. Think of it as taking a rough draft and then working on it further with a more focused set of examples. Step 8: Post-training the model But if you are looking at the final OLMo-2 Instruct models, there is another important part. After pretraining and the additional training stages, the models go through post-training. This includes techniques such as: Supervised Fine-Tuning (SFT) Direct Preference Optimization (DPO) Proximal Policy Optimization (PPO) The OLMo-2 Instruct models use post-training data and methods associated with the Tülu 3 work. This is an important distinction. Pretraining teaches the model broad language capabilities along with statistical patterns and other capabilities learned from the training data. Post-training is used to make the model more useful for following instructions and producing the kind of responses we expect from an assistant. So, when we download an instruction-following model and chat with it, we are seeing the result of more than just the initial pretraining stage. Step 9: Convert the model so others can use it In the hf_olmo folder, there is a file called convert_olmo_to_hf.py. An OLMo checkpoint has its own format. But Hugging Face Transformers uses its own model, configuration, and tokenizer representations. So this script provides the bridge between them. It converts an OLMo checkpoint into Hugging Face-compatible files so that it can be loaded by tooling that supports that format. This is more of a release and distribution step than part of the actual training process. Step 10: Testing that it actually works In olmo/eval and the related evaluation components, you can find the code used for evaluating the model. Finally, a model needs to be evaluated using benchmarks, task-specific tests, human evaluation, safety evaluations, and other measurements to understand how well it actually performs. Wrapping up When we ignore the scale and the wide amount of jargon, from this example, building a model can be boiled down to this sequence: Collect and document your data Tokenize it into a format the model can process Design the architecture (often adapting known techniques) Write a training loop that learns from the data Configure the exact run so it's reproducible Checkpoint constantly so progress isn't lost Continue training on a smaller, targeted dataset and potentially merge multiple resulting checkpoints Post-train the model using techniques such as SFT, DPO, and PPO Convert the result into a widely usable format Evaluate it using benchmarks and other evaluation methods Hope you got a good walkthrough experience of how such models are built. It was interesting to me when I first stumbled across it, and it felt like suddenly a lot of the vagueness disappeared. See you in another article. Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: HexmosTech / LiveReview Blast-Radius Aware AI Code Review for Business-Critical Systems       LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. blast-radius-demo.mp4 LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score How does Blast Radius scoring work? (a more technical explanation) Here's the goal: A 3-line fix in a function used by 40 other files, that also writes to a database, should score high. A 300-line UI change in one file, fully covered by… View on GitHub Click below to try LiveReview with your codebase: