← News

PROJECT BEDROCK / DAY 1–2

Day 1–2: From tokens to generalization

Starting with CS336 Lecture 1 and the Transformer paper, working through gradient descent by hand, and turning what we learn into questions worth pursuing.

Token representations, layered computation and a gradient-descent surface

The first two days of Project Bedrock began with an existing foundation in elementary mathematics, calculus, linear algebra and some probability and statistics. We completed Stanford CS336 Lecture 1, building an initial picture of language models, tokenizers and training resources, while returning to supervised learning, gradient descent and learning rates. The most exciting change was working through formulas by hand: familiar symbols acquired concrete meanings, and an update rule that could once simply be called became something we could explain step by step.

Lecture 1 made efficiency a new lens for reading design decisions. What model can be trained with a given supply of data, hardware and computation? Parameter count, training tokens and computational work have to be considered together. Many architectural and systems choices respond to these constraints. Efficiency became a way to understand what bottleneck a design addresses and which trade-offs it makes—a thread that led back to Attention Is All You Need.

Comparing tokenizers introduced character-, byte- and word-level representations. I was particularly drawn to the uniformity and simplicity of bytes. Yet one token per byte gives a compression ratio of 1 when measured as UTF-8 bytes per token, producing long sequences. This leaves a natural direction for further reading: how can byte coverage be retained while grouping, compression or multiscale processing reduces the computational burden? The next step is to compare these intuitions with the concrete designs of existing subword methods and byte-level models.

When rereading the Transformer paper, I paused at its starting point: recurrent models impose sequential dependencies across positions, limiting parallelism within a training example. Extended Neural GPU, ByteNet and ConvS2S also pursued reductions in sequential computation. I began to read the Transformer as a choice about computational organization, dependency paths and representation, while distinguishing parallel computation during training from the step-by-step production of tokens in autoregressive generation.

The mathematical work started by identifying the objects behind the notation: the mapping defined by a model, the distinct roles of parameters and inputs, and what a loss measures. Deriving gradient descent also made the limits of first-order information concrete. A zero derivative identifies a stationary point, but minima, maxima and other stationary points can share that information. In higher dimensions, saddles, flat regions, ill-conditioned curvature and vanishing gradients add further difficulties. Convexity, curvature and step-size stability have become connected questions for the next stage of study.

A more fundamental question then emerged: why should a lower training loss imply a better model? A training set contains only finite observations, while what matters is performance on unseen data. Why can solutions with similarly low training loss generalize differently? This distinction shifts attention from reducing an objective to understanding its relationship with the real task. It also prompts questions about how people use prior knowledge to build transferable understanding from limited experience.

The optimizer itself therefore becomes part of the learning problem. Initialization, update rules, learning rates and data order can send training along different trajectories toward different solutions; implicit bias offers an important lens on that selection. I want to study how learning-rate schedules affect stability and efficiency, beginning with tractable examples whose conditions can be made explicit, then examining what carries over to neural networks. Questions about global optima also need to be considered alongside the model, objective, computational budget and generalization goal.

With finite training data, how do optimization dynamics shape the geometry and generalization of the solution reached?

These reflections currently converge on a question for deeper study: with finite training data, how do optimization dynamics shape the geometry and generalization of the solution reached? It is a starting point for further reading, derivation and small practical investigations. We have also organized the PS01 v3 consolidation collection: Papers A and B, their learning solutions, four design and learning-analysis reports, a review guide, and a Chinese learning translation of the Transformer paper. All ten PDFs are available, bringing understanding back to calculations and proofs that can be reconstructed independently.

Consolidation should create new questions as well as strengthen answers. Retain an independent attempt, consult the starting approach and full derivation, cover the solution and reconstruct it, change one condition, then return after a delay. The achievement of these first two days is that ideas are beginning to connect and questions are becoming more precise. Project Bedrock will continue from those concrete points of difficulty, connecting reading, mathematics and practice in work that can be shared and developed further.

DAY 1–2 / PS01

Bring understanding back to work you can reconstruct.

The full learning record, all ten original PDFs, a complete download and a step-by-step consolidation path.

Read the log & all resources ↗