Lesson 1

Tensors, shapes, and broadcasting

Introduction

Two sequences of three token IDs form a tensor with shape two by three. Selecting the middle position from every row returns values 1 and 0, showing how shape and axis meaning guide every later model operation.

Learning goal

Create tensors, name each axis, predict indexing results, and detect unsafe broadcasting or memory sharing.

Before you start

Python lists, indexing, integer and decimal values, and simple nested lists.

Lesson plan

  1. Read tensor shape, data type, device, and the meaning assigned to each axis.
  2. Index a text batch and predict both selected values and resulting shapes.
  3. Compare broadcasting directions, safe tensor copying, and NumPy memory sharing behavior.

You only need Python lists and indexing for this lesson. If items[0] and items[-1] are familiar, you are ready.

Why tensors matter

Neural networks calculate with numbers. A word, image, sound clip, or model parameter must become numbers before a model can use it. A PyTorch tensor stores those numbers in a regular grid.

The tensor also records its shape, data type, and device. These facts are part of the input contract. A tensor can contain the correct six numbers in the wrong arrangement, and the program may then solve the wrong task.

One text example is the letter h. A computer cannot use the letter directly, so we assign it an integer ID, perhaps 8. A batch of three letters can then become [8, 9, 8]. The IDs are labels; they do not mean that token 9 is “larger” or “better” than token 8.

Words and shapes

ScalarOne number. Its shape is ().
VectorA one-dimensional list. Shape (3,) means three values.
MatrixA two-dimensional table. Shape (2, 3) means two rows and three columns.
DimensionOne axis of a tensor. A matrix has two dimensions.
dtypeThe kind of stored number, such as torch.int64 or torch.float32.
deviceWhere the tensor lives. This lesson uses cpu.

Shape is read from the outside inward. If token IDs have shape (2, 3), there are two sequences in the batch and three tokens in each sequence. The tensor does not know these meanings. We assign the meaning “sequence” to axis 0 and “position” to axis 1.

PyTorch normally uses integer token IDs for lookup operations. It uses floating-point values for learned weights. A model operation usually requires all participating tensors to use the same device.

Worked example: inspect a text batch

Suppose h = 0, i = 1, and a space is 2. We place two three-token sequences into one tensor. Indexing with tokens[0] selects the first sequence. Indexing with tokens[:, 1] selects position 1 from every sequence; the colon means “all rows.”

Predict before running: write the shapes of tokens, tokens[0], and tokens[:, 1]. Then predict the two values selected by tokens[:, 1].

import torch

tokens = torch.tensor([[0, 1, 2],
                       [1, 0, 1]], dtype=torch.long)

print(tokens)
print("shape:", tuple(tokens.shape))
print("dtype:", tokens.dtype)
print("device:", tokens.device.type)
print("first sequence:", tokens[0].tolist())
print("middle tokens:", tokens[:, 1].tolist())

# Expected result:
# tensor([[0, 1, 2],
#         [1, 0, 1]])
# shape: (2, 3)
# dtype: torch.int64
# device: cpu
# first sequence: [0, 1, 2]
# middle tokens: [1, 0]

torch.long is another name for 64-bit integers. The answers are (2, 3), (3,), and (2,). The middle-token values are [1, 0]. We selected one position from each of the two sequences.

The tensor has six values, but its shape is not (6,). The two-dimensional arrangement preserves the batch structure. You can reshape values with view or reshape, but the total number of values must stay the same.

Broadcasting: same valid shape, different calculation

Broadcasting lets PyTorch repeat a smaller tensor across a compatible axis. It checks sizes. It does not check what an axis means.

scores = torch.tensor([[10.0, 20.0],
                       [30.0, 40.0],
                       [50.0, 60.0]])  # shape (3, 2)

per_feature = torch.tensor([1.0, 2.0])       # shape (2,)
per_example = torch.tensor([[1.0], [2.0], [3.0]])  # shape (3, 1)

print(scores + per_feature)
# Expected: [[11, 22], [31, 42], [51, 62]]

print(scores + per_example)
# Expected: [[11, 21], [32, 42], [53, 63]]

Before reading the results, choose one cell: row 1, column 0. The original value is 30. Feature broadcasting adds 1, so the result is 31. Example broadcasting adds the row value 2, so the result is 32.

A common wrong answer is that both additions produce the same result because both smaller tensors contain small offsets. Their shapes choose different repetition directions. Print one known cell when a broadcasted result looks plausible but wrong.

Creation, copying, and NumPy sharing

torch.tensor(existing_data) makes a new tensor by copying the data. torch.from_numpy(array) usually shares CPU memory with the NumPy array. If one side changes, the other can change too. Use .clone() when you need an independent tensor. This distinction matters in data preparation because an unexpected shared mutation can alter training examples.

import numpy as np
import torch

source = np.array([1.0, 2.0], dtype=np.float32)
shared = torch.from_numpy(source)
copied = torch.tensor(source)
source[0] = 9.0

print(shared.tolist())  # Expected: [9.0, 2.0]
print(copied.tolist())  # Expected: [1.0, 2.0]

Common pitfalls

  • Do not pass floating-point token IDs to nn.Embedding; it expects integer indices.
  • Do not confuse a scalar of shape () with a one-item vector of shape (1,).
  • Check shape, dtype, and device near the point where an error occurs.
  • A reshape changes the view of the data, not its meaning. Keep track of what every axis represents.
  • Do not accept a broadcast only because PyTorch accepts it. Write the meaning of each axis and check one result by hand.

Try it

Create a tensor for three sequences, each containing four token IDs. Print its shape, then select the final token from every sequence. What shapes should those two tensors have?

Reveal the worked answer 
import torch

x = torch.tensor([[0, 1, 2, 1],
                  [2, 2, 0, 1],
                  [1, 0, 0, 2]], dtype=torch.long)
last = x[:, -1]
print(tuple(x.shape))     # Expected: (3, 4)
print(last.tolist())      # Expected: [1, 1, 2]
print(tuple(last.shape))  # Expected: (3,)

The first axis counts three sequences. The second counts four positions. Selecting one position removes that position axis, so the result is a vector with one value per sequence.

Recap

A tensor is a numbered grid plus metadata. Shape gives each axis size. Your program gives each axis meaning. Data type tells you the number representation. Device tells you where computation happens.

Predict shapes first. Then check one value by hand. In the next lesson, autograd will attach a calculation history to floating-point tensors and use it to compute gradients.

Reference: PyTorch tensor tutorial.