Tensors and computation graphs

Tensors are multi-dimensional arrays that are the basic data structures used in deep learning frameworks like TensorFlow and PyTorch. They generalize matrices to higher dimensions and are used to represent inputs, weights, outputs, and gradients in neural networks.

A computation graph is a directed graph where nodes represent operations or variables and edges represent the flow of tensors. It defines how data flows through operations, making it easier to compute gradients through backpropagation using automatic differentiation.

Tensors have different ranks (or dimensions):

• 0D tensor: Scalar (e.g., 7)
• 1D tensor: Vector (e.g., [1, 2, 3])
• 2D tensor: Matrix (e.g., [[1, 2], [3, 4]])
• 3D and above: Higher-dimensional data (e.g., image batches)

Example of a computation graph representing a neural network operation

Here's an example using PyTorch to demonstrate tensors and computation graph creation:

import torch

# Create tensors
x = torch.tensor([2.0], requires_grad=True)
y = torch.tensor([3.0], requires_grad=True)

# Build computation graph
z = x ** 2 + y ** 3  # z = x² + y³

# Compute gradients
z.backward()

# Display gradients
print(f"dz/dx: {x.grad.item()}")  # Should be 2 * x
print(f"dz/dy: {y.grad.item()}")  # Should be 3 * y²

In this code, PyTorch automatically builds a computation graph and computes gradients for tensors using backward propagation.