Character-level
One character per unit, with character-set size and coverage to consider.
PROJECT BEDROCK / LEARNING LOG 01
DAY 1–2
Starting with CS336 Lecture 1 and the Transformer paper, working through gradient descent by hand, and turning what we learn into questions worth pursuing.
E4B8AD→RepresentationBEGIN WITH WHAT WE KNOW
The first two days of Project Bedrock began with an existing foundation in elementary mathematics, calculus, linear algebra and some probability and statistics. We completed Stanford CS336 Lecture 1, building an initial picture of language models, tokenizers and training resources, while returning to supervised learning, gradient descent and learning rates. The most exciting change was working through formulas by hand: familiar symbols acquired concrete meanings, and an update rule that could once simply be called became something we could explain step by step.
Elementary mathematics · Calculus · Linear algebra · Some probability and statistics
Existing foundations: elementary mathematics, calculus, linear algebra, and some probability and statistics.
We completed the first lecture of Stanford CS336 and established an initial overview: tokenizers, the number of training tokens required at a given model scale, how to estimate the computation, and the role of efficiency in design decisions. Many choices are fundamentally made with training efficiency in mind. We learned about byte-based, character-based, and word-based tokenization. Returning to the Transformer paper, I found its efficiency trade-offs convincing. Its starting problem—the inherent sequential nature of recurrent computation prevents parallelization within training examples, especially for long sequences—makes this a good moment to pause and read the classic attention paper again. Extended Neural GPU, ByteNet, and ConvS2S also pursued the reduction of sequential computation. I want to return to that history and ask which other architectures began from efficiency as their motivation.
Byte-based encoding is elegant, even though a compression ratio of one produces very long sequences. Could a method retain its many advantages while addressing this compression problem? We then studied mathematical methods and notation, derived gradient descent by hand, and revisited supervised learning and learning rates. This was exciting. I had not previously worked through these formulas by hand; doing so changed my understanding at a fundamental level and raised many questions that made me curious.
For gradient descent, I also saw the limitation of first-order information: when the derivative is zero, that information alone cannot distinguish minima, maxima, and other stationary points, including saddles in higher dimensions. This led naturally to convex optimization and the question of finding global optima. My mathematical understanding then extended to the difficulties encountered in high-dimensional neural networks: saddle points, flat regions, ill-conditioned curvature, and vanishing gradients.
I also began to distinguish optimizing the training loss from obtaining a good model. Why should the smallest loss mean the best model? What about generalization? Why can different low-training-loss solutions generalize differently? How should the learning rate be adjusted in seeking a global optimum efficiently? These questions deserve investigation and might lead to new ideas. I may study the mathematics of this small branch in the coming days, or move on for now; either way, I find the questions interesting.
Further exploration helped me see why minimizing training loss need not produce the best model: the loss measures fit to the observed training set, while the world extends far beyond those observations. We care about performance on unseen data. A data-driven training process cannot possess all possible data, so what exactly are we trying to train? How could we build a machine with very strong generalization? Looking at people, why can humans generalize so effectively from limited experience?
We also learned that the optimization algorithm is itself part of the learning process. This connects to implicit bias and learning rates. How might learning-rate choices help us approach a global optimum rather than settle into a local one? Could the rate be adjusted dynamically? What common problems do current schedules face, and could an efficiency-driven approach improve both speed and accuracy? These remain questions to investigate, with the relevant conditions still to be understood.
These questions come together in a possible research theme: with finite training data, how do optimization dynamics affect the geometry and generalization of the final solution? This may help us understand solutions that generalize better. I will leave the question here for now and return to explore it later.
01 / REPRESENTATION & EFFICIENCY
Lecture 1 made efficiency a new lens for reading design decisions. What model can be trained with a given supply of data, hardware and computation? Parameter count, training tokens and computational work have to be considered together. Many architectural and systems choices respond to these constraints. Efficiency became a way to understand what bottleneck a design addresses and which trade-offs it makes—a thread that led back to Attention Is All You Need.
Comparing tokenizers introduced character-, byte- and word-level representations. I was particularly drawn to the uniformity and simplicity of bytes. Yet one token per byte gives a compression ratio of 1 when measured as UTF-8 bytes per token, producing long sequences. This leaves a natural direction for further reading: how can byte coverage be retained while grouping, compression or multiscale processing reduces the computational burden? The next step is to compare these intuitions with the concrete designs of existing subword methods and byte-level models.
One character per unit, with character-set size and coverage to consider.
A uniform byte representation; one byte per token gives one UTF-8 byte per token.
Intuitive semantic units, with trade-offs around vocabulary size, rare words and unseen words.
C denotes training FLOPs, N the parameter count and D the number of training tokens. This is a common rough estimate; the detailed calculation depends on the architecture and training setup. Allocating model size and data also requires considering the goal and available resources.
When rereading the Transformer paper, I paused at its starting point: recurrent models impose sequential dependencies across positions, limiting parallelism within a training example. Extended Neural GPU, ByteNet and ConvS2S also pursued reductions in sequential computation. I began to read the Transformer as a choice about computational organization, dependency paths and representation, while distinguishing parallel computation during training from the step-by-step production of tokens in autoregressive generation.
02 / UNDERSTAND THE UPDATE
The mathematical work started by identifying the objects behind the notation: the mapping defined by a model, the distinct roles of parameters and inputs, and what a loss measures. Deriving gradient descent also made the limits of first-order information concrete. A zero derivative identifies a stationary point, but minima, maxima and other stationary points can share that information. In higher dimensions, saddles, flat regions, ill-conditioned curvature and vanishing gradients add further difficulties. Convexity, curvature and step-size stability have become connected questions for the next stage of study.
The learning rate controls step size. Direction, stability and convergence each require their own conditions.
f(x) = x²Minimumf(x) = −x²Maximumf(x) = x³Stationary inflectionf(x,y) = x² − y²SaddleAll four examples have zero derivative or gradient at the origin. That information alone does not identify a minimum.
03 / BEYOND THE TRAINING SET
A more fundamental question then emerged: why should a lower training loss imply a better model? A training set contains only finite observations, while what matters is performance on unseen data. Why can solutions with similarly low training loss generalize differently? This distinction shifts attention from reducing an objective to understanding its relationship with the real task. It also prompts questions about how people use prior knowledge to build transferable understanding from limited experience.
The optimizer itself therefore becomes part of the learning problem. Initialization, update rules, learning rates and data order can send training along different trajectories toward different solutions; implicit bias offers an important lens on that selection. I want to study how learning-rate schedules affect stability and efficiency, beginning with tractable examples whose conditions can be made explicit, then examining what carries over to neural networks. Questions about global optima also need to be considered alongside the model, objective, computational budget and generalization goal.
With finite training data, how do optimization dynamics shape the geometry and generalization of the solution reached?
Next, I want to separate this question into objects that can be studied: optimization trajectories, step-size rules, solution geometry and evaluation on unseen data. Clear assumptions and comparisons will guide future reading and practical work. I will leave the question here for now and return to explore it later.

PS01 / CONSOLIDATE BY RECONSTRUCTING
All ten supplied PDFs are available in their original form, totaling 103 pages. PS01 is a Project Bedrock learning collection focused on independent attempts, correction and transfer. Each exam has 21 questions and a maximum of 150 points; 120 minutes is a timed-retake guideline, while the first learning pass may be completed in segments.
Paper A · Reconstruct definitions and methods ↗
Review guide · 12 mathematical topics ↗
Attention Is All You Need · Chinese translation ↗21 questions, 150 points: from notation, functions, matrices and loss to backpropagation, expressivity and error propagation. Work in segments initially; use 120 minutes for a timed retake.
21 questions, 150 points: reuse the same methods with new objects, constraints and proofs, connecting XOR, square approximation, convergence conditions, generalization bounds and low-rank approximation.
Each of 21 problems includes a starting approach, full solution, error correction, takeaway, reconstruction task and changed-condition variant; page 16 provides variant checks.
Full derivations and a changed-condition variant for every problem, with attention to assumptions and intermediate steps. Rebuild without the solution, then consult the checks on page 16.
Twelve topics and eight knowledge chains connect the A/B papers. Revisit the relevant definition, derivation, condition or counterexample, then reconstruct the original problem.
Explains each problem’s purpose, observable learning evidence and withdrawal of support, with cross-problem links and the boundaries of adaptations from classic papers.
Explains how the same knowledge is reused with new constraints, objects or proofs, including teaching cases on finite-class generalization, PL convergence, low-rank approximation and residual analysis.
Track errors, corrective actions, transfer checks and delayed reconstruction. S0–S3 are learning states with fields for your own records.
Check explanation, reconstruction, variants and delayed retention across eight knowledge chains, using changed conditions to diagnose transfer.
A Chinese learning translation of the Transformer paper, retaining its eight original authors and references for studying architecture, parallelism and attention.
The supplied Chinese translation retains the original authorship of Vaswani et al. PS01 solutions and reports support study and consolidation.
MAKE IT YOUR OWN
Compare x², −x² and x³ at the origin, then x²−y² in two dimensions: identical first-order information can accompany different local geometry.
Consider two functions that agree on the finite training points but differ elsewhere. Agreement on a training set and agreement over a domain are different claims.
Compare coverage, vocabulary, sequence length and computational organization separately. Can a different computational structure retain the strengths of the representation?
The central achievement of these two days is that ideas are beginning to connect and questions are becoming concrete. Further progress will remain part of the Bedrock record.