{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" }, "colab": { "name": "notebook0_numpy_pytorch_intro.ipynb", "provenance": [], "collapsed_sections": [ "cv3E-G1kNoQL", "p3HcanX9NoQT", "rgGOtkJINoQc", "hp91uxDGNoQl", "OyL9SyUeNoQs", "KbaYOCL_NoQ0", "3RMLqrgQNoQ6", "aGUeZ09kNoRD", "TWVS8OFJNoRM", "zGju50FlNoRR", "R99jJOwGNoRX", "sZCthYVhNoRc", "urKJVOFHNoRg", "Na3ky9YANoRm", "BUsGZUFPNoRv" ] }, "accelerator": "GPU" }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "47vQbATHNoPV" }, "source": [ "# Introduction to NumPy and PyTorch" ] }, { "cell_type": "markdown", "metadata": { "id": "0ezJ5PikNoPY" }, "source": [ "Numpy is a widely used Python library for scientific computing with\n", "multidimensional arrays." ] }, { "cell_type": "markdown", "metadata": { "id": "HFavUXRENoPb" }, "source": [ "PyTorch is a popular library used for Deep Learning research and applications. It is similar to numpy in that it is built around the manipulation of multidimensional arrays, but with a few additional features:\n", "* GPU support\n", "* Automatic differentiation\n", "* Other utilities to facilitate building and training neural network" ] }, { "cell_type": "markdown", "metadata": { "id": "blojpu_0NoPc" }, "source": [ "This notebook contains examples some of the essential features of NumPy and PyTorch. Examples from each framework will be presented side-by-side to highlight the similarities (and occasional differences) between their APIs.\n", "\n", "In particular, we will first explore some of the core operations involving the central data structures in each library, the NumPy `ndarray` and the PyTorch `Tensor`.\n", "\n", "Finally, we will look at the basics of automatic differentiation in PyTorch.\n", "\n", "The official documentation is a good place to learn more:\n", "* https://docs.scipy.org/doc/numpy/user/index.html\n", "* https://pytorch.org/docs/stable/index.html\n", "* https://pytorch.org/tutorials/" ] }, { "cell_type": "markdown", "metadata": { "id": "_1T-SkUvNoPe" }, "source": [ "To start, it is typically a good idea to set a random seed to facilitate reproducibility of experiments. So we first import our libraries and set the random seed in both:" ] }, { "cell_type": "code", "metadata": { "id": "Bh3MLTaUNoPn" }, "source": [ "import numpy as np\n", "import torch\n", "\n", "np.random.seed(0)\n", "torch.manual_seed(0)\n", "\n", "use_cuda = torch.cuda.is_available()\n", "device = torch.device('cuda' if use_cuda else 'cpu')\n", "use_cuda, device" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "iZGaGFi9NoPy" }, "source": [ "### Creating arrays\n", "\n", "We can create arrays from lists of values, or alternatively use a number of helper functions to create specific types of arrays. In the latter case, we typically pass in the desired size of the array as the first argument (see examples below - should be fairly self-explanatory)." ] }, { "cell_type": "markdown", "metadata": { "id": "PIzWnQJ6NoP1" }, "source": [ "##### From list" ] }, { "cell_type": "code", "metadata": { "id": "vjHpp9CCNoP2" }, "source": [ "a = np.array([1, 2, 3])\n", "a" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "e-N2svCDNoP6" }, "source": [ "a_ = torch.Tensor([1, 2, 3])\n", "a_" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "4wcdIWbTNoP-" }, "source": [ "a_long = torch.LongTensor([1, 2, 3])\n", "a_long" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "N7meswjYNoQC" }, "source": [ "#### Empty array" ] }, { "cell_type": "code", "metadata": { "id": "O6Mn791GNoQD" }, "source": [ "b = np.empty((3, 2))\n", "b" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "IPI-PHzcNoQH" }, "source": [ "b_ = torch.empty((3, 2))\n", "b_" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "cv3E-G1kNoQL" }, "source": [ "#### Zeroes" ] }, { "cell_type": "code", "metadata": { "id": "ZoJiTYheNoQM" }, "source": [ "c = np.zeros((2, 3))\n", "c" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "7jDofBcuNoQQ" }, "source": [ "c_ = torch.zeros((2, 3))\n", "c_" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "p3HcanX9NoQT" }, "source": [ "#### Ones" ] }, { "cell_type": "code", "metadata": { "id": "4-w7RhWFNoQU" }, "source": [ "d = np.ones(3)\n", "d" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "_ksEHq8yNoQY" }, "source": [ "d_ = torch.ones(3)\n", "d_" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "rgGOtkJINoQc" }, "source": [ "#### Samples from Uniform[0, 1]" ] }, { "cell_type": "code", "metadata": { "id": "eQMUG3mQNoQe" }, "source": [ "e = np.random.random((2, 3))\n", "e" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "twRYL6Y-NoQg" }, "source": [ "e_ = torch.rand((2, 3))\n", "e_" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "IT6tqpBdNoQk" }, "source": [ "### Basic properties of arrays" ] }, { "cell_type": "markdown", "metadata": { "id": "hp91uxDGNoQl" }, "source": [ "#### Shape" ] }, { "cell_type": "code", "metadata": { "id": "v-fp0J2cNoQm" }, "source": [ "print(b)\n", "b.shape" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "h-DbxHgPNoQq" }, "source": [ "print(b_)\n", "b_.shape" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "OyL9SyUeNoQs" }, "source": [ "#### dtype" ] }, { "cell_type": "code", "metadata": { "id": "M5rufy7BNoQt" }, "source": [ "print(a.dtype)\n", "print(e.dtype)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ThpXztgTNoQw" }, "source": [ "print(a_.dtype)\n", "print(a_long.dtype)\n", "print(e_.dtype)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "TGDVMwWPNoQz" }, "source": [ "### Indexing" ] }, { "cell_type": "markdown", "metadata": { "id": "KbaYOCL_NoQ0" }, "source": [ "#### Integer indexing" ] }, { "cell_type": "code", "metadata": { "id": "VXrvNwFBNoQ1" }, "source": [ "print(b)\n", "print(b[0])\n", "print(b[0, 0])" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "LBJ4dAtPNoQ4" }, "source": [ "print(b_)\n", "print(b_[0])\n", "print(b_[1])\n", "print(b_[0, 0])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "3RMLqrgQNoQ6" }, "source": [ "#### Slicing" ] }, { "cell_type": "code", "metadata": { "id": "L_hJb1qvNoQ6" }, "source": [ "print(b)\n", "print()\n", "print(b[:2])\n", "print(b[2:])\n", "print(b[1:3])\n", "print(b[:, :1])\n", "print(b[:2, :2])" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "qIRo5w2RNoQ-" }, "source": [ "print(b_[:2])\n", "print(b_[1:3])\n", "print(b_[:, :1])\n", "print(b_[:2, :2])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "aGUeZ09kNoRD" }, "source": [ "#### Boolean indexing" ] }, { "cell_type": "code", "metadata": { "id": "CpTnLRo1NoRE" }, "source": [ "print(a)\n", "print()\n", "idx = a >= 2\n", "print(idx)\n", "print(a[idx])" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "9kIPJMvsNoRJ" }, "source": [ "idx_ = a_ >= 2\n", "print(idx_)\n", "print(a_[idx_])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "QJjEmTraNoRM" }, "source": [ "### Mathematical operations" ] }, { "cell_type": "markdown", "metadata": { "id": "TWVS8OFJNoRM" }, "source": [ "#### Sum" ] }, { "cell_type": "code", "metadata": { "id": "O1O85R6zNoRO" }, "source": [ "print(e)\n", "print()\n", "print(e.sum())\n", "print(np.sum(e))\n", "print(e.sum(axis=0))\n", "print(e.sum(axis=1))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "2MyuWb1HNoRP" }, "source": [ "print(e_)\n", "print()\n", "print(e_.sum())\n", "print(torch.sum(e_))\n", "print(e_.sum(dim=0))\n", "print(e_.sum(dim=1))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "zGju50FlNoRR" }, "source": [ "#### Elementwise sum" ] }, { "cell_type": "code", "metadata": { "id": "NhWol32INoRT" }, "source": [ "print(a)\n", "print(d)\n", "print()\n", "print(a + d)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "DdptaCWnNoRV" }, "source": [ "print(a_)\n", "print(d_)\n", "print()\n", "print(a_ + d_)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "R99jJOwGNoRX" }, "source": [ "#### Elementwise multiplication" ] }, { "cell_type": "code", "metadata": { "id": "PFPbJdfgNoRY" }, "source": [ "print(a)\n", "print(d)\n", "print()\n", "print(a * d)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Nx-fJcOgNoRZ" }, "source": [ "print(a_)\n", "print(d_)\n", "print()\n", "print(a_ * d_)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "sZCthYVhNoRc" }, "source": [ "#### Dot product" ] }, { "cell_type": "code", "metadata": { "id": "t2WSLosnNoRc" }, "source": [ "print(a)\n", "print(d)\n", "print()\n", "print(np.dot(a, d))\n", "print(a.dot(d))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Tp0zhvNQNoRe" }, "source": [ "print(a_)\n", "print(d_)\n", "print()\n", "print(torch.dot(a_, d_))\n", "print(a_.dot(d_))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "urKJVOFHNoRg" }, "source": [ "#### Matrix multiplication" ] }, { "cell_type": "code", "metadata": { "id": "m92t7na4NoRh" }, "source": [ "print(a)\n", "print(b)\n", "print()\n", "print(np.matmul(a, b))\n", "print(a @ b)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "3gHYSMlwNoRi" }, "source": [ "print(a_)\n", "print(b_)\n", "print()\n", "print(torch.matmul(a_, b_))\n", "print(a_ @ b_)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "0tK24UceNoRl" }, "source": [ "### Broadcasting" ] }, { "cell_type": "markdown", "metadata": { "id": "Na3ky9YANoRm" }, "source": [ "#### Compatible shapes" ] }, { "cell_type": "code", "metadata": { "id": "59bi9SpQNoRm" }, "source": [ "f = torch.rand((2, 1, 3))\n", "g = torch.ones((3, 3))\n", "print(f)\n", "print(g)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Ymeo3SGQNoRo" }, "source": [ "print(f)\n", "print(g)\n", "print(f.shape)\n", "print(g.shape)\n", "print()\n", "print(f + g)\n", "print(f * g)\n", "\n", "print((f + g).shape)\n", "print((f * g).shape)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "BUsGZUFPNoRv" }, "source": [ "#### Incompatible shapes" ] }, { "cell_type": "code", "metadata": { "id": "pUo4_wORNoRv" }, "source": [ "h = np.random.random((2, 3))\n", "i = np.random.random((2, 2))\n", "print(h.shape)\n", "print(i.shape)\n", "\n", "# Raises error\n", "h + i" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "BxUQM7LpNoRy" }, "source": [ "### Converting between NumPy and PyTorch" ] }, { "cell_type": "code", "metadata": { "id": "MkWCR4k-NoRy" }, "source": [ "arr_np = np.random.random((5, 5))\n", "arr_th = torch.rand((5, 5))\n", "\n", "# From numpy\n", "torch.Tensor(arr_np)\n", "print(torch.from_numpy(arr_np))\n", "\n", "# To numpy\n", "print()\n", "print(arr_th.numpy())\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "r95RBNtpNoRz" }, "source": [ "### Autograd basics" ] }, { "cell_type": "code", "metadata": { "id": "prNyiAYxNoR1" }, "source": [ "W = torch.randn((7, 5), requires_grad=True)\n", "x = torch.randn(5)\n", "y = torch.matmul(W, x)\n", "z = y.sum()\n", "print(W)\n", "print(x)\n", "print(z)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "G2NBAPIGNoR3" }, "source": [ "z.backward()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "cmIO3O2UNoR5" }, "source": [ "W.grad" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Pk-TpbXEHmY7" }, "source": [ "x.grad is None" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "hvRJUyWaNoR6" }, "source": [ "# Another example involving differentiating through a for-loop\n", "W = torch.randn((5, 5), requires_grad=True)\n", "x = torch.randn(5)\n", "for i in range(3):\n", " x = torch.matmul(W, x)\n", "z = x.sum()\n", "z.backward()\n", "print(W.grad)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "OizNOSFiIIs9" }, "source": [ "" ], "execution_count": null, "outputs": [] } ] }