{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Analysis via Least-Squares Fitting \n", "\n", "## Overview, Objectives, and Key Terms\n", " \n", "A ubiquitous task in engineering and other technical disciplines is the analysis of data. Data comes from many sources, including measurements in the laboratory and numerical simulations. In this lecture and the next, we'll apply what we've learned so far to produce *models* from data that we can use to explore trends and make conclusions. Specifically, in this lecture, we'll develop models based on **least-squares** fitting of model parameters to existing data. Such fitting may be the right choice when the data in contaminated by noise or when the model need only capture the main features of the observed phenomenon.\n", "\n", "\n", "### Objectives\n", "\n", "By the end of this lesson, you should be able to\n", "\n", "- Explain (and demonstrate) what is meant by a least-squares fit of a linear model $ax+b$ to a set of measured points $(x_i, y_i),\\, i = 0, 1, \\ldots$.\n", "- Use built-in tools to perform linear, least-squares fitting of data to models\n", "- Use built-in tools to perform nonlinear, least-squares fitting of data to models\n", "\n", "### Key Terms\n", "\n", "- error\n", "- least-squares\n", "- normal equation\n", "- `np.linalg.solve`\n", "- `np.polyfit`\n", "- `np.polyval`\n", "- `scipy.optimize.curve_fit`\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Noisy Data and $y = at + b$\n", "\n", "We'll start simple and explore a task with which most students ought to be familiar: running a line through some scattered points of the form $y = at+b$ through several, scattered points $(t_i, y_i),\\, i = 0, 1, \\ldots$. To illustrate the problem, suppose a number of measurements $y_i$ were taken at evenly spaced times $t_0, t_1, \\ldots$, as show below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEcNJREFUeJzt3X+M5PVdx/HXa7kaurQNkDuQXzPTEqwlVUE2BCU2tYih\nSAtqSCRjg9q4RluKlUSpYywxGUNQqRpN41qwGCcYpRiwkCrBKqEx2D0KLXgqpJ3ZUq7ctVBFp0mL\n+/aPnT33jtvbmbn5fr7f+X6ej2QzM5+Z2+977i7zms/38+PriBAAIF8LZRcAACgXQQAAmSMIACBz\nBAEAZI4gAIDMEQQAkDmCAAAyRxAAQOYIAgDI3K6yCxjH7t27o9VqlV0GAMyVvXv3fi0i9uz0urkI\nglarpdXV1bLLAIC5Ynswzus4NQQAmSMIACBzBAEAZI4gAIDMEQQAkDmCAAAK1uv11Gq1tLCwoFar\npV6vV3ZJh5mL6aMAMK96vZ6Wl5c1HA4lSYPBQMvLy5KkdrtdZmmH0CMAgAJ1Op1DIbBpOByq0+mU\nVNGrEQQAUKC1tbWJ2stAEABAgRqNxkTtZSAIAKBA3W5Xi4uLh7UtLi6q2+2WVNGrEQQAUKB2u62V\nlRU1m03ZVrPZ1MrKSmUGiiXJEVF2DTtaWloKNp0DgMnY3hsRSzu9jh4BAGSOIACAzBEEAJA5ggAA\nMkcQAEDmCAIA2ar6ZnCpsOkcgCzNw2ZwqdAjAJClqm8Gl7K3Qo8AQJaqvBlc6t4KPQIAWaryZnCp\neysEAYAsVXkzuNS9FYIAQJaqvBlc6t4KQQAgW+12W/1+X+vr6+r3+5UIASl9b4UgAICKSd1bYRtq\nAKgptqEGAIyFIACAzBEEAJA5ggAAMkcQAKiNFPvz1HHHUvYaAlALKfbnqeuOpUwfBVALrVZLg8Hg\nVe3NZlP9fn9ujjFLpU8ftX2O7U/b3mf7ads3jtpPtf2Q7WdGt6cUVQOAfKTYn6fKO5YejyLHCF6R\ndFNEvEXSJZLeZ/t8STdLejgizpP08OgxAByXFPvzVHnH0uNRWBBExP6IeHx0/2VJ+ySdJelqSXeN\nXnaXpGuKqgFAPlLsz1PlHUuPR5JZQ7Zbki6U9Jik0yNiv7QRFpJOS1EDgHpLsT9PlXcsPR6FDxbb\nfp2kf5LUjYh7bX8jIk7e8vxLEfGqcQLby5KWJanRaFx0tAEaAMD2Sh8sHhXxGkmfkNSLiHtHzS/Y\nPmP0/BmSDhztz0bESkQsRcTSnj17iiwTALJW5KwhS7pD0r6IuH3LU/dLun50/3pJ9xVVAwBgZ0X2\nCC6V9B5J77D9xOjnSkm3Srrc9jOSLh89BjBH6ri6NmdFzhp6NCIcEd8bEReMfh6MiK9HxGURcd7o\n9sWiagAwe5uraweDgSLi0OraWYcBYZMOK4sBTCTF6tojt3KQNqZp1mGGTkqVGCwGUD/TrK6d9Nt9\np9M5LAQkaTgcqtPpTF4wdkQQAJjIpKtrpzmVVNetHKqKIAAwkUlX107z7b6uWzlUFUEAYCKTrq6d\n5tt9XbdyqCoGiwEUatrB5V6vp06no7W1NTUaDXW7XQaKJ8RgMYCxFD1Nc9pv9+12W/1+X+vr6+r3\n+4RAgQgCIGMp1gTUdaO2OuHUEJCxebviFibDqSEAO2KaJiSCAMga0zQhEQRA1pimCYkgALLGQC4k\nBosBoLYYLAYAjIUgAIDMEQQAkDmCAAAyRxAAQOYIAgDIHEEAAJkjCAAgcwQBAGSOIACAzBEEAJA5\nggAAMkcQAEDmCAIAyBxBAACZIwgAIHMEAQBkjiAAgMwRBACQOYIAADJHEABA5ggCAMgcQQAAmSMI\nACBzhQWB7TttH7D91Ja2W2x/xfYTo58rizo+kKNer6dWq6WFhQW1Wi31er2yS8IcKLJH8HFJVxyl\n/SMRccHo58ECjw9kpdfraXl5WYPBQBGhwWCg5eVlwgA7KiwIIuIRSS8W9fsBHK7T6Wg4HB7WNhwO\n1el0SqoI86KMMYL32/786NTRKdu9yPay7VXbqwcPHkxZHzCX1tbWJmoHNqUOgo9KOlfSBZL2S/q9\n7V4YESsRsRQRS3v27ElVHzC3Go3GRO3ApqRBEBEvRMT/RsS6pD+VdHHK4wOzUsVB2W63q8XFxcPa\nFhcX1e12S6oI8yJpENg+Y8vDH5f01HavBaqqqoOy7XZbKysrajabsq1ms6mVlRW12+1S60L1OSKK\n+cX23ZLeLmm3pBckfXj0+AJJIakv6RciYv9Ov2tpaSlWV1cLqROYVKvV0mAweFV7s9lUv9+f6bF6\nvZ46nY7W1tbUaDTU7Xb5YMfYbO+NiKWdXrerqAIi4rqjNN9R1PGAVFINym72PDZnAm32PCQRBpgp\nVhYDE0o1KMt0UKRCEAATSjUoy3RQpEIQABNKNSjLdFCkQhAAU2i32+r3+1pfX1e/3y/knD3TQZEK\nQQBUFNNBkUph00dniemjADC5caeP0iMAgMwRBACQOYIASKSK+xMBUoEriwH8P1YJo8qOOVhs+0RJ\nV0n6IUlnSvqmNjaKeyAink5SoRgsxvxLuT8RsOm49xqyfYukd0n6R0mPSTog6URJ3yXp1lFI3BQR\nn59FwUCdsUoYVXasU0OfjYhbtnnudtunSWKJIzCGRqNx1B4Bq4RRBdsOFkfEA5Jk+9ojn7N9bUQc\niAjO1wBjYJUwqmycWUMfGrMNwDZYJYwqO9YYwTslXSnpLNt/uOWpN0h6pejCgLppt9t88KOSjjVG\n8LykvZLePbrd9LKkDxZZFAAgnW2DICKelPSk7V5EfDthTQCAhLYdI7D9t7bftc1zb7L9W7Z/rrjS\ngHRY9YucHevU0M9L+hVJH7H9kqSDkl4rqSXpWUl/FBH3FV4hUDBW/SJ3O25DbfsGSY9qYzHZNyX9\nR0QMj/mHZoyVxSgSq35RV7Pchvp0SX+tjQHi79RGGAC1wapf5G7HIIiI35B0nqQ7JP2MpGds/7bt\ncwuuDUiCawMjd2NtQx0b54++Ovp5RdIpku6xfVuBtQFJsOoXudsxCGx/wPZeSbdJ+oyk74mIX5R0\nkaSfLLg+oHCs+kXuxrkewW5JPxERh42mRcS67auKKQtIi1W/yNmOQRARv3mM5/bNthwAQGpcqhIA\nMkcQAEDmCAIkxVYOQPVw8Xokw1YOQDXRI0AynU7nUAhsGg6H6nQ6Mz0OvQ5gMvQIkEyKrRzodQCT\no0eAZFJs5ZCq1wHUCUGAZFJs5cAGcsDkCAIkk2IrBzaQAyZHECCpdrutfr+v9fV19fv9mZ+3ZwM5\nYHKFBYHtO20fsP3UlrZTbT9k+5nR7SlFHR95YgM5YHI7XqFs6l9sv03Sf0v684h466jtNkkvRsSt\ntm+WdEpE/NpOv4srlAHA5GZ5hbKpRMQjkl48ovlqSXeN7t8l6Zqijg8AGE/qMYLTI2K/JI1uT9vu\nhbaXba/aXj148GCyAgEgN5UdLI6IlYhYioilPXv2lF0OSsIqYaB4qVcWv2D7jIjYb/sMSQcSHx9z\nhFXCQBqpewT3S7p+dP96SfclPj7mCKuEgTSKnD56t6R/lvRm28/Zfq+kWyVdbvsZSZePHgNHxSph\nII3CTg1FxHXbPHVZUcdEvTQaDQ0Gg6O2A5idyg4WA6wSBtIgCFBZrBIG0iAIaqou0y6L3psIABem\nqSWmXQKYBD2CGmLaJYBJEAQ1xLRLAJMgCGoo1cVZ6jIOAeSOIKihFNMuN8chBoOBIuLQOARhAMwf\ngqCGUky7ZBwCqI/CLkwzS1yYpnoWFhZ0tP87trW+vl5CRQCOVPqFaVBvXCQeqA+CAFNh+wegPggC\nTIXtH4D6YIwAAGqKMQIAwFgIAgDIHEEAAJkjCAAgcwQBAGSOIACAzBEEAJA5ggAAMkcQAEDmCAIA\nyBxBAACZIwgAIHMEAQBkjiAAgMwRBACQOYIAADJHEMyBXq+nVqulhYUFtVot9Xq9sksCUCO7yi4A\nx9br9bS8vKzhcChJGgwGWl5eliQuCwlgJugRVFyn0zkUApuGw6E6nc5Mj0OvA8gXPYKKW1tbm6h9\nGvQ6gLzRI6i4RqMxUfs0UvU6AFQTQVBx3W5Xi4uLh7UtLi6q2+3O7Bgpeh0AqosgqLh2u62VlRU1\nm03ZVrPZ1MrKykxP2aTodQCorlKCwHbf9hdsP2F7tYwa5km73Va/39f6+rr6/f7Mz9un6HUAqK4y\newQ/HBEXRMRSiTUctzrMtknR6wBQXZwa2mLSD/XN2TaDwUARcWi2zbyGQZG9DgDV5YhIf1D7S5Je\nkhSS/iQiVo7ymmVJy5LUaDQuGgwGhdZ05BRKaeP0yLG+GbdaLR2trmazqX6/X1SpADAW23vHOetS\nVhCcGRHP2z5N0kOSboiIR7Z7/dLSUqyuFjuUMM2H+sLCgo7292db6+vrsy4RACYybhCUcmooIp4f\n3R6Q9DeSLi6jjq2mmULJbBsAdZA8CGyfZPv1m/cl/aikp1LXcaRpPtSZbQOgDsroEZwu6VHbT0r6\nF0kPRMSnSqjjMNN8qDPbBkAdlDJGMKkUYwTSxoBxp9PR2tqaGo2Gut0uH+oA5lalB4snlSoIAKBO\nKj1YnLM6LEADUC9sQ50Q2z0DqCJ6BAmx3TOAKiIIEmK7ZwBVRBAkxAI0AFVEECTEAjQAVUQQJMQC\nNABVxDoCAKgp1hEAAMZCEABA5ggCAMgcQQAAmSMIACBzBAEAZI4gAIDMEQQAkDmCAAAyRxAAQOYI\nAgDIXG2DgEtCAsB4anmpSi4JCQDjq2WPgEtCAsD4ahkEXBISAMZXyyDgkpAAML5aBgGXhASA8dUy\nCLgkJACMj0tVAkBNcalKAMBYCAIAyBxBAACZIwgAIHMEAQBkbi5mDdk+KGkw5R/fLelrMyxnHvCe\n88B7zsPxvOdmROzZ6UVzEQTHw/bqONOn6oT3nAfecx5SvGdODQFA5ggCAMhcDkGwUnYBJeA954H3\nnIfC33PtxwgAAMeWQ48AAHAMtQ4C21fY/nfbz9q+uex6imb7HNuftr3P9tO2byy7phRsn2D7c7Y/\nWXYtKdg+2fY9tv9t9G/9A2XXVDTbHxz9n37K9t22Tyy7plmzfaftA7af2tJ2qu2HbD8zuj2liGPX\nNghsnyDpjyW9U9L5kq6zfX65VRXuFUk3RcRbJF0i6X0ZvGdJulHSvrKLSOgPJH0qIr5b0vep5u/d\n9lmSPiBpKSLeKukEST9VblWF+LikK45ou1nSwxFxnqSHR49nrrZBIOliSc9GxBcj4luS/lLS1SXX\nVKiI2B8Rj4/uv6yND4izyq2qWLbPlvRjkj5Wdi0p2H6DpLdJukOSIuJbEfGNcqtKYpek19reJWlR\n0vMl1zNzEfGIpBePaL5a0l2j+3dJuqaIY9c5CM6S9OUtj59TzT8Ut7LdknShpMfKraRwvy/pVyWt\nl11IIm+SdFDSn41Oh33M9kllF1WkiPiKpN+VtCZpv6T/jIi/L7eqZE6PiP3Sxhc9SacVcZA6B4GP\n0pbFFCnbr5P0CUm/HBH/VXY9RbF9laQDEbG37FoS2iXp+yV9NCIulPQ/Kuh0QVWMzotfLemNks6U\ndJLtny63qnqpcxA8J+mcLY/PVg27k0ey/RpthEAvIu4tu56CXSrp3bb72jj19w7bf1FuSYV7TtJz\nEbHZ07tHG8FQZz8i6UsRcTAivi3pXkk/WHJNqbxg+wxJGt0eKOIgdQ6Cz0o6z/YbbX+HNgaX7i+5\npkLZtjbOHe+LiNvLrqdoEfGhiDg7Ilra+Pf9h4io9TfFiPiqpC/bfvOo6TJJ/1piSSmsSbrE9uLo\n//hlqvkA+Rb3S7p+dP96SfcVcZBdRfzSKoiIV2y/X9LfaWOWwZ0R8XTJZRXtUknvkfQF20+M2n49\nIh4ssSbM3g2SeqMvOF+U9LMl11OoiHjM9j2SHtfGzLjPqYYrjG3fLentknbbfk7ShyXdKumvbL9X\nG4F4bSHHZmUxAOStzqeGAABjIAgAIHMEAQBkjiAAgMwRBACQOYIAmNJoF9BfKrsO4HgRBMD0TpZE\nEGDuEQTA9G6VdK7tJ2z/TtnFANNiQRkwpdEOr58c7ZEPzC16BACQOYIAADJHEADTe1nS68suAjhe\nBAEwpYj4uqTPjC6ozmAx5haDxQCQOXoEAJA5ggAAMkcQAEDmCAIAyBxBAACZIwgAIHMEAQBkjiAA\ngMz9H2ym8y7SCp8/AAAAAElFTkSuQmCC\n" }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from data_fitting_examples import scattered_points\n", "t, y = scattered_points()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll *assume* that the observed quantity $y(t)$ varies *linearly* with $t$, i.e., we define $y(t) = at + b$. The goal is to find the model parameters $a$ and $b$ yielding a model that *best* matches the observed values $y_i$. \n", "\n", "Before defining what exactly *best* means, try to define $a$ and $b$ by inspection. One easy way to estimate $a$ and $b$ is to use the first and last measured points, i.e., defined the intercept to be $b = y[0]$ and the slope to be $a = (y[-1]-y[0])/(t[-1]-t[0])$. \n", "\n", "The result is not bad:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW5x/HPE3CLWqUXtNY2M9ZKQNGrkKpVq16su9QV\nt9S9xg23WhcclWqNeG0LLqDeVFBaUutaxQ1QVEBqlbCoyCK0ZqKALIpWjbLld//4JaxZJmHOOZM5\n3/frNa8kJ5M5T5TXPHme32bOOUREJL4Kog5ARESipUQgIhJzSgQiIjGnRCAiEnNKBCIiMadEICIS\nc0oEIiIxp0QgIhJzSgQiIjHXMeoAMtG5c2eXTCajDkNEpF2ZMmXKUudcl5ae1y4SQTKZpKqqKuow\nRETaFTNLZ/I8tYZERGJOiUBEJOaUCEREYk6JQEQk5pQIRERiTolARCRglZWVJJNJCgoKSCaTVFZW\nRh3SetrF9FERkfaqsrKSsrIyamtrAUin05SVlQFQWloaZWhrqCIQEQlQKpVakwQa1NbWkkqlIopo\nY0oEIiIBqqmpadX1KCgRiIgEqKioqFXXo6BEICISoPLycgoLC9e7VlhYSHl5eUQRbUyJQEQkQKWl\npVRUVJBIJDAzEokEFRUVOTNQDGDOuahjaFFJSYnTpnMiIq1jZlOccyUtPU8VgYhIzCkRiIjEnBKB\niEjMKRGIiMScEoGISMwpEYhIbOX6ZnBh0aZzIhJL7WEzuLCoIhCRWMr1zeDCrFZUEYhILOXyZnBh\nVyuqCEQklnJ5M7hUKsXBtbXMALrWXwuyWlEiEJFYytnN4KqrGZxO8xLQAfjOOt8KqlpRIhCRWMq5\nzeC+/RZuvx26d+dIM64H9gLW3WUtqGpFYwQiElulpaW5MUNo9Gi4/HKYNw/69mXMQQcxpH9/Vq4z\nmB1ktaKKQEQkKuk0nHQSHH00FBTA2LHw+OOceMUVoVYr2oZaRCRsy5fDH/4A5eVgBjffDFdfDVts\nkdXbZLoNtVpDIiJhGjPGt4HmzoVTToE//hEinqmk1pCISBhqauDkk+Goo3wVMGYMPPFE5EkAlAhE\nRIK1fDkMHAjdusFLL8Edd8C778IRR0Qd2RpqDYmIBGXsWOjXz7eBTjoJBg2CRCLqqDaiikBE8kYY\n+/NkdI+aGt//P/JIcM5XAk89lZNJAADnXM4/evXq5UREmjNy5EhXWFjogDWPwsJCN3LkyPDusXy5\ncwMHOldY6NxWWzl3++3Offtt1u7fWkCVy+A9VtNHRSQvJJNJ0un0RtcTiQTV1dXB3+Ohh3wbaM4c\nOPFEGDw48gog0+mjgbWGzOyHZvaamc0ys/fN7Mr66981s5fNbG79x05BxSAi8RHGbqKNvdbOwF3p\nNBx+OKxaBS++CE8/HXkSaI0gxwhWAdc457oD+wOXmdnuwA3AOOfcbsC4+q9FRDZJGLuJrvtamwHX\nArOBX5jB734HM2b4VcLtTGCJwDm30Dk3tf7zL4FZ+OR5PDCi/mkjgBOCikFE4iOM3UQb7tEbeAe4\nC3itQwdGDxoEN90EW26ZtXuFKZRZQ2aWBPYB3gJ2dM4tBJ8sgB3CiEFE8lsYu4mWHnIIM/fck3HA\n5sB5XbrwnxEjOOGqq7J2jygEPlhsZtsA44Fy59zTZva5c277db6/zDm30TiBmZUBZQBFRUW9Ghug\nEREJxYoVcM89cOutsHo19O8P112X8xVA5IPF9UFsBjwFVDrnnq6/vMjMdqr//k7A4sZ+1jlX4Zwr\ncc6VdOnSJcgwRUSa9uqrsPfe/o3/sMNg5ky45ZacTwKtEeSsIQOGAbOcc4PW+dYo4Jz6z88Bng0q\nBhGRNps/H04/3b/5L18Ozz0Hzz4Lu+wSdWRZF2RFcCBwFtDbzKbXP44B7gQON7O5wOH1X4tIOxLG\nCt7IrFzpt4ju1s2/8d96K7z/Phx3XNSRBSeTVWdRP7SyWCR3hLGCt+E+iUTCmZlLJBJZf/1Gvfqq\nc7vv7hw416ePc//6V/D3DBAZriyO/E0+k4cSgUjuSCQS6yWBhkcikcjaPcJKNmvMn+/cGWf4t8Rd\ndnFu1Khg7hOyTBOBNp0TkVZpywre1raSUqkUteuc1wtQW1tLKpVqfcDNWbnS7whaXOxXAw8Y4NtA\nffpk9z45TttQi0irFBUVNbrfTlMreCsrKykrK1vzxp5OpykrKwNoco5/GNtFMH48XHaZf+M/9lg/\nPXTXXbP3+u2IKgIRaZXWruBty1/3gW4XsXAhlJbCoYfC11/DqFHw/POxTQKgRCAirdTaFbxt+es+\nkO0iVq70O4IWF/uzAW65xa8JiFkbqFGZDCRE/dBgsUj71dbB5azOGnr9def22MMPBh99tHNz57b9\ntdoRNFgsIpkIek1AW/+6Ly0tpbq6mrq6Oqqrq9u2Z9DChfDLX/o20FdfwTPPwAsvwI9/3PrXymeZ\nZIuoH6oIRIKRt2sCVq50bvBg57bd1rnNN3fuppuc+/rrYO+Zg9AJZSLSkjBO9QrdxIl+NtB77/kz\ng++7D3bbLeqoIpETm86JSG4LZZpmWD75BM46Cw4+GL74wq8LeOml2CaB1lAiEImxME71CtyqVX4N\nQHExPP443Hijnw104olgFnV07YISgUiMhXGqV6DeeAN69YKrroL99/ftoPJy2HrrqCNrV5QIRGIs\njFO9ArFoEZxzDvzsZ7BsmV8XMHo0dO0adWTtkgaLRaT9WLUKHngAbr4ZamvhN7+BVEoVQBMyHSzW\nXkMi0j5MmuRnA73zDhx+uJ8NVFwcdVR5Qa0hEcltixfDeefBQQfBp5/Ck0/CmDFKAlmkRCAiuWn1\nahg61Pf9Kyvhhhtg9mw4+WTNBsoytYZEJPe8+aZvA02bBj//uW8DdesWdVR5SxWBiOSOxYvh/PPh\ngANgyRJ44gkYO1ZJIGBKBCISvdWr4f77fd//L3+B66+HWbPglFPUBgqBWkMiEq1120CHHebbQN27\nRx1VrKgiEJFoLFkCF1zg20CLF8Njj8HLLysJRECJQETCtXq1XxTWtSv8+c9w3XV+NtCpp6oNFBG1\nhkQkPG+9BZdeClOnQu/eMGSIKoAcoIpARIK3dClceKHfGO6TT+Bvf4NXXlESyBFKBCISnNWr4cEH\nfRvokUfgmmt8G+i009QGyiFqDYlIMN5+27eBpkzxZwYPGQJ77BF1VNIIVQQikl1Ll0JZmW8DLVgA\nf/0rvPqqkkAOUyIQkexYvRoqKvyisOHD4de/9m2gM85QGyjHqTUkIptu8mS/KGzyZDjkEN8G6tEj\n6qgkQ6oIRKTtPv0ULroI9tsPPvrI7xL62mtKAu2MEoGItF5dHfzpT3420LBh/szgOXPgzDPVBmqH\n1BoSkdapqvJtoLff9mcGDx0Ke+4ZdVSyCVQRiEhmPvsMLrkE9t0X0mm/S+j48UoCeUCJQESaV1fn\n2z9du/p20JVX+jbQL3+pNlCeUGtIRJo2dapfFPbWW/7M4KFDYa+9oo5KskwVgYhs7LPPfAIoKYHq\nar9L6IQJSgJ5SolARNaqq/OLwYqL4f/+D664wreBzjpLbaA8FlgiMLPhZrbYzGasc+23ZjbfzKbX\nP44J6v4icVRZWUkymaSgoIBkMkllZWXmPzx1Khx4oD8sprjYf3333bDddsEFLDkhyIrgEeCoRq4P\nds7tXf94McD7i8RKZWUlZWVlpNNpnHOk02nKyspaTgbLlkG/fvCTn8C//w0jRsDEifDf/x1O4BK5\nwBKBc24C8FlQry8i60ulUtTW1q53rba2llQq1fgP1NXBww/7v/4feMCPCcyZA2efrTZQzEQxRtDP\nzN6tbx11aupJZlZmZlVmVrVkyZIw4xNpl2pqajK/Pm2anwV0/vmw225+q+j77oPttw84SslFYSeC\nB4Bdgb2BhcAfm3qic67COVfinCvp0qVLWPGJtFtFRUUtX//8c7j8cj8baN48XxFMnAh77x1SlJKL\nQk0EzrlFzrnVzrk64E/AvmHeXyRbNmlQNiDl5eUUFhaud62wsJDy8nLfBhoxwreB7r9/bRvo3HOh\nQJMH4y7UBWVmtpNzbmH9lycCM5p7vkguahiUbejHNwzKApSWlkYWV8O9U6kUNTU1FBUVUV5eTmmP\nHnDwwTBpEvz0pzB6NOyzT2RxSu4Jcvroo8CbQLGZfWxmFwB3mdl7ZvYu8D/A1UHdXyQorR6U3QSt\nrTxKS0uprq6mrq6O6unTKX37bejZ0//1P3w4vPGGkoBsJLCKwDl3RiOXhwV1P5GwtGpQdhO0ufJw\nDkaOhGuvhSVL4OKL4fbboVOTczMk5tQcFGmljAZls6BNlce77/o20NlnQzLpTwwbOlRJQJqlRCDS\nSs0OymZRqyqPL77wh8P07AmzZvldQv/xD/+1SAuUCERaqbS0lIqKChKJBGZGIpGgoqIi6wPFGVUe\nzvlzAYqL4d57oawMPvgAfvUrzQaSjOlfikgbrDcoW10dyGyhFiuP997zB8WffTYkEv7EsPvvh+9+\nN+uxSH5TIhDJUU1WHn36wK9/7Wf/zJzp20BvvukXiYm0gQ6mEclhpaWla6sN5+Cvf/VtoEWLfBuo\nvBz+67+iDVLaPSUCkfZgxgx/YPyECX6X0OeeUwUgWaPWkEgu+89/4Jpr/F5AM2ZARQX8859KApJV\nSgQiIWnVKuGGNlC3bjB4sN8l9IMP4MILNRtIsk6tIZEQtGqV8Pvv+zbQ+PH+L/9nnoF9tT+jBMec\nc01/02xL4DjgZ8D3gW/wG8W94Jx7P5QIgZKSEldVVRXW7USyLplMkk6nN7qeSCSorq72X3z5Jdx6\nK9xzD2y7LQwc6NcDdOgQbrCSN8xsinOuxT5ikxWBmf0W6AO8DrwFLAa2BLoCd9YniWucc+9mI2CR\nfNbsKmHn4LHH/FjAggX+zX/gQOjcOeQoJa6aaw1Nds79tonvDTKzHYDsbq4ikqeKiooarQh6f+97\ncNhh8Npr0KsXPP007LdfBBFKnDU56uScewHAzPpu+D0z6+ucW+ycU79GJAMbrhLeBhjcsSNjFy+G\n6dP9mcFvvaUkIJHIZPpB/wyviUgT1qwSLiridGBuhw5ctWoVBeee688KuPhijQVIZJobIzgaOAbY\n2czuXedb3wFWBR2YSL4p7dmT0h//GGpqYK+9/L5A++8fdVgizVYEC4ApwLf1Hxseo4Ajgw9NJE98\n9RVcd51/85861Z8PMHmykoDkjCYrAufcO8A7ZlbpnFsZYkwi+cE5eOIJv0Hc/Plw3nlw552www5R\nRyayniYrAjN7zsz6NPG9H5nZbWZ2fnChiYSntWcDt2j2bDj8cDjtNOjSxR8SM3y4koDkpOamj14I\n/BoYbGbLgCXAVkASmAcMcc49G3iEIgFr89nAjfnqK38+8KBBUFgIQ4ZoIFhyXrMriwHM7HLgDfxi\nsm+AD5xztc3+UJZpZbEEKaNVvy1xDp56Cq6+Gj7+GM49F/73f1UBSKQyXVmcyfTRHYEngKuB7+GT\ngUjeaNXZwI2ZMweOPBL69vVnA7zxBjz8sJKAtBstJgLn3E3AbsAw4FxgrpndYWa7BhybSCgyOhu4\nMV9/DTfeCHvu6Y+JvO8+qKqCAw8MIEqR4GS0n63z/aNP6h+rgE7Ak2Z2V4CxiYSixbOBN+Sc3wqi\ne3e/J9CZZ/qqoF8/6KgNfaX9aTERmNkVZjYFuAuYBOzpnLsE6AWcHHB8IoFr8mzgxgaKP/gAjj4a\nTj4ZOnWCiRPhkUdgxx1Dj1skWzL586UzcJJzbr3RNOdcnZkdF0xYIuFa72zgxnz9NdxxB/zhD7Dl\nln6r6EsvVQUgeaHFf8XOuVua+d6s7IYjkmOcg7//Ha66Cj76CM4+288G+t73oo5MJGt05p1IU+bO\nXdsG2n57f3D8iBFKApJ3lAhENlRbCzfdBD16wJtvwt13+z2CfvazqCMTCYQSgYQq61s5ZFNDG6h7\ndygv99tDzJkDV16psQDJa/rXLaHJ6lYO2TZvHlxxBbz0kq8Exo+Hgw+ONiaRkKgikNCkUqk1SaBB\nbW0tqVQqq/dpVdVRWws33wx77OFXBA8a5NtASgISI6oIJDSbvJVDBjKuOpyDUaN82yedhtJS+P3v\nYaedshaLSHuhikBC0+atHFoho6rjX/+C446DE06AbbaB11+HkSOVBCS2lAgkNK3eyqENmq06vvkG\nBgzwbaCJE30baNo0OOSQrN1fpD1SIpDQtGorhzZqqro4r3NnnwBuu82vC5g9228ZvdlmWbu3SHvV\n4nkEuUDnEUimNhwj2AUYUlDAMXV1sPvu/rzgQw+NNEaRsGTzPIK2BjDczBab2Yx1rn3XzF42s7n1\nHzsFdX+Jp4aqo+sPf8gAYBZw+Oab+z2Cpk9XEhBpRJCtoUeAoza4dgMwzjm3GzCu/muRrCrdbjvm\nbLYZvwW2OP10Nps3D665Rm0gkSYElgiccxOAzza4fDwwov7zEcAJQd1fYujDD+EXv4A+fWCLLWDc\nOHj0Udh556gjE8lpYQ8W7+icWwhQ/7HJs/zMrMzMqsysasmSJaEFKO3Qt9/6QeDdd4dXX/XrAaZP\nh969o45MpF3I2VlDzrkK51yJc66kS5cuUYcjEWlxlfALL/jZQAMGwPHH+9lAv/kNbL55NAGLtENh\nryxeZGY7OecWmtlOwOKQ7y/tSLOrhA84wJ8RMGoUdOsGr7wChx0WZbgi7VbYiWAUcA5wZ/3HZ0O+\nv7Qjja0SXl1byyeXXQbLl0OHDnDXXX6bCFUAIm0WWCIws0eBQ4HOZvYxMACfAB43swuAGqBvUPeX\n9m/DVcJHAfcBP/7iCzj1VPjjH+EHP4gkNpF8ElgicM6d0cS3VL9LRoqKikin0ySAwcCJwGygdIcd\nqHzssWiDE8kjOTtYLDJwwAB+u9lmzAKOAK4H9t9qK44ZNCjiyETyi7ahltw0ejRn3HEHrFzJC4WF\nXFJbS0EiwdDy8ugPsRHJM6oI8lROHwnZnHQaTjrJHxpfUABjxnDs119T4xzV1dVKAiIBUEWQh3L6\nSMimLF/uB39vvx3MYOBAvzvoFltEHZlI3tPuo3komUySTqc3up5IJKiurg4/oJaMHQv9+sHcuX6L\n6EGDIIuH1YjEVeS7j0p0wjgSMitqauCUU+DII/3Xo0fDk08qCYiETIkgD4VxJCRswjjEihVw553Q\nvTu8+CKUl8N7761NCCISKiWCPBTGkZAN4xDpdBrn3JpxiBaTwcsvw157Qf/+/o1/1iy48UaNBYhE\nSIkgD4VxJGRGh8Sv66OPoG9fOOIIWL3aVwJPPw2JRNZiEpG20WCxtElBQQGN/dsxM+rq6tZeWLEC\nBg/220Q7B6mUPyRmyy1DjFYknjRYLIHKaBzilVd8G+iGG3wlMHOmTwRKAiI5RYlA2qTZcYiPP4bT\nToPDD4dVq/yZAX//OyST0QQrIs3SgjJpk4bxhlQqRU1NDUVFRQy89VbOmD/fnw+werVvB117rSoA\nkRynMQLJjnHj/KKw2bP9SWGDB8Muu0QdlUisaYxAwjF/Ppx+Ovz8535g+Pnn4ZlnlARE2hElAmmb\nFSv8IfHFxfDss3DrrfD++3DssVFHJiKtpDECab1XX/VtoFmzoE8fuPtu+NGPoo5KRNpIFYFkbv58\nOOMMf0j8t9/Cc8/5w+OVBETaNSUCadnKlX6L6G7d/DTQAQN8G+i446KOTESyQK0had5rr/k20MyZ\nvv9/zz2w665RRyUiWaSKQBq3YAGceSb07g21tb4F9PzzSgIieUiJQNa3cqU/GKa42G8Kd8stvhro\n0yfqyEQkIGoNyVrjx8Nll/n+/zHHwL33qgIQiQFVBAILF0JpKRx6KHz1lV8QpjaQSGwoEcTZypV+\nK4jiYn9E5E03+TbQ8cf7A+RFJBbUGoqrCRN8G2jGDDjqKN8G2m23qKMSkQioIoibTz6Bs86CQw6B\n//zHrwt48UUlAZEYUyKIi1Wr/BqA4mJ4/HF/QMysWXDCCWoDicScWkNxMHGibwO9954/MP6++1QB\niMgaqgjy2aJFcM45cPDB8MUXfl3ASy8pCYjIepQI2oHKykqSySQFBQUkk0kqKyub/4FVq/xf/cXF\n8OijcOONfjbQiSeqDSQiG1FrKMdVVlZSVlZGbW0tAOl0mrKyMmDtcZHrmTQJLr0U3n3Xnxk8ZAh0\n7RpmyCLSzqgiyHGpVGpNEmhQW1tLKpVa/4mLFsG558JBB8GyZX5dwJgxGSeBVlcdIpI3VBHkuJqa\nmuavr1oFDz7oF4PV1kL//n5G0NZbZ3yPVlcdIpJXVBHkuKKioqavT5oEJSVw+eWw775+VtAdd7Qq\nCUArqg4RyUtKBDmuvLycwsLC9a4lttqKcUVFvg306afwxBO+DVRc3KZ7tFh1iEheUyLIcaWlpVRU\nVJBIJOgI3NypEx+Yses//wk33OAXhZ1yyibNBmq26hCRvBdJIjCzajN7z8ymm1lVFDG0J6WlpVQ/\n+igr996b25YtY/MDDvCzggYOhG222eTXb6zqKCwspLy8fJNfW0RyX5QVwf845/Z2zpVEGMMmC3y2\nzeLFcP75cMABsGSJ3x5i7Fh/fnCWrFt1mBmJRIKKigoNFIvEhXMu9AdQDXTO9Pm9evVyYRg5cqRL\nJBLOzFwikXAjR45s8fmFhYUOWPMoLCxs8ecysmqVc0OHOrf99s517Ojcddc59+WXm/66IhIbQJXL\n5D05kydl+wF8CEwFpgBlTTynDKgCqoqKigL6z7RWW97UE4nEes9veCQSiU0L5s03nevZ0//v6d3b\nuZkzN+31RCSWMk0E5p8bLjP7vnNugZntALwMXO6cm9DU80tKSlxVVbBDCclkknQ6vdH1RCJBdXV1\noz9TUFBAY//9zIy6urrWB7FkiV8HMGwYfP/7/tCYvn21LYSItImZTXEZtN8jGSNwzi2o/7gY+Duw\nbxRxrKstUyizNttm9Wq/KKy4GEaMgGuvhdmz4dRTlQREJHChJwIz29rMtm34HDgCmBF2HBtqy5t6\nVmbbvP027LcfXHIJ7L03vPMO3HUXbLtt5q8hIrIJoqgIdgTeMLN3gLeBF5xzoyOIYz1teVPfpNk2\nS5dCWRnsvz8sWOB3CR03DnbffVN/FRGRVolkjKC1whgjAD8VNJVKUVNTQ1FREeXl5dmfQrl6tR8D\n6N/fnxFw1VVwyy3wne9k9z4iEnuZjhEoEYRp8mS/RXRVlT8zeMgQ6NEj6qhEJE/l9GBx7Hz6KVx0\nEey3H9/MncsVnTtTMH48yeOO03bPIhI5JYIg1dVBRYU/E2DYMGYddRS7rFjBfUuX4li73bOSgYhE\nSYkgKJMn+4Hgiy7y7Z/p0zl65kwWffPNek/Tds8iEjUlgmz79FO4+GI/JfSjj2DkSHj9dejRQ9s9\ni0hOUiLIlro6eOghvyjsoYfgyiv9orDS0jWLwrTds4jkIiWCbJgyBX76U7jwQr8OYNo0vz3Edtut\n9zRt9ywiuUiJYFN89plfEfyTn0A6DX/5C4wfD3vu2ejTtd2ziOQirSNoi7o6ePhhuP56WLYM+vWD\n227bqAIQEYmS1hEEZepUf0jMr34F3bv7NtA99ygJiEi7pUSQqWXL4LLLoKQEPvwQ/vxnmDAB9tor\n6shERDaJEkFLGtpAXbv6raIvvxzmzIGzztIW0SKSF5QImjNtGhx0kD8zuGtXPzvonntg++2jjkxE\nJGuUCBrz+ef+L/+SEpg3Dx55BCZO9OcFiIjkmY5RB5BT6ur8FNBrr/UrhC+9FH73O1UAIpLXlAga\nTJ/uB4P/8Q+/R9CYMbDPPlFHJSISOLWGPv8crrgCevWCDz6A4cNh0iQlARGJjfhWBM6tbQMtXeo3\nirv9dujUKerIRERCFc9E8O67vg30xhu+DfTSS9CzZ9RRiYhEIl6toYYzgnv29DuDDhvm20BKAiIS\nY3mbCCorK0kmkxQUFJBMJPjHJZf4LaLvvRfKyvyisPPPh4K8/U8gIpKRvGwNVVZWUlZWRm1tLT2A\noTU1HPDggyzddVc6v/CCHxgWEREgTyuCVCrlj4AEpgG7AxcAP1m5UklARGQDeVkRNBz9+G/gT0AK\nWAbYRx9FGJWISG7Ky4qg4ejHR4FL8Ulg3esiIrJWXiYCHQkpIpK5vEwEOhJSRCRzOqpSRCRP6ahK\nERHJiBKBiEjMKRGIiMScEoGISMwpEYiIxFy7mDVkZkuAdBt/vDOwNIvhtAf6neNBv3M8bMrvnHDO\ndWnpSe0iEWwKM6vKZPpUPtHvHA/6neMhjN9ZrSERkZhTIhARibk4JIKKqAOIgH7neNDvHA+B/855\nP0YgIiLNi0NFICIizcjrRGBmR5nZHDObZ2Y3RB1P0Mzsh2b2mpnNMrP3zezKqGMKg5l1MLNpZvZ8\n1LGEwcy2N7MnzWx2/f/rn0YdU9DM7Or6f9MzzOxRM9sy6piyzcyGm9liM5uxzrXvmtnLZja3/mOn\nIO6dt4nAzDoAQ4Gj8adVnmFmu0cbVeBWAdc457oD+wOXxeB3BrgSmBV1ECG6BxjtnOsG/Dd5/rub\n2c7AFUCJc64H0AE4PdqoAvEIcNQG124AxjnndgPG1X+ddXmbCIB9gXnOuX8751YAfwOOjzimQDnn\nFjrnptZ//iX+DWLnaKMKlpn9ADgWeCjqWMJgZt8BDgaGATjnVjjnPo82qlB0BLYys45AIbAg4niy\nzjk3Afhsg8vHAyPqPx8BnBDEvfM5EewMrHtI8cfk+ZviuswsCewDvBVtJIG7G7gOqIs6kJD8CFgC\nPFzfDnvIzLaOOqggOefmA38AaoCFwBfOubHRRhWaHZ1zC8H/oQfsEMRN8jkRWCPXYjFFysy2AZ4C\nrnLO/SfqeIJiZscBi51zU6KOJUQdgZ7AA865fYCvCahdkCvq++LHA7sA3we2NrNfRhtVfsnnRPAx\n8MN1vv4BeVhObsjMNsMngUrn3NNRxxOwA4FfmFk1vvXX28xGRhtS4D4GPnbONVR6T+ITQz77OfCh\nc26Jc24l8DRwQMQxhWWRme0EUP9xcRA3yedEMBnYzcx2MbPN8YNLoyKOKVBmZvje8Szn3KCo4wma\nc66/c+6Oi0/ZAAABWElEQVQHzrkk/v/vq865vP5L0Tn3CfCRmRXXXzoMmBlhSGGoAfY3s8L6f+OH\nkecD5OsYBZxT//k5wLNB3KRjEC+aC5xzq8ysHzAGP8tguHPu/YjDCtqBwFnAe2Y2vf7ajc65FyOM\nSbLvcqCy/g+cfwPnRRxPoJxzb5nZk8BU/My4aeThCmMzexQ4FOhsZh8DA4A7gcfN7AJ8QuwbyL21\nslhEJN7yuTUkIiIZUCIQEYk5JQIRkZhTIhARiTklAhGRmFMiEGmj+l1AL406DpFNpUQg0nbbA0oE\n0u4pEYi03Z3ArmY23cx+H3UwIm2lBWUibVS/w+vz9Xvki7RbqghERGJOiUBEJOaUCETa7ktg26iD\nENlUSgQibeSc+xSYVH+gugaLpd3SYLGISMypIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIR\nkZhTIhARiTklAhGRmPt/UmH+x4CuF2IAAAAASUVORK5CYII=\n" }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "b = y[0]\n", "a = (y[-1]-y[0])/(t[-1]-t[0])\n", "y_a = a*t + b\n", "plt.plot(t, y, 'ko', t, y_a, 'r')\n", "plt.xlabel('t')\n", "plt.ylabel('y(t)')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## What \"Best\" Means\n", "\n", "To define a \"best\" line, however, requires we define what \"best\" means. In general, any values of $a$ and $b$ will result in a model that differs from the measured values. This difference is the error. More formally, let the error be defined as\n", "\n", "$$\n", " e_i = y(t_i) - y_i = a t_i + b - y_i \\, \\quad i = 0, 1, \\ldots\n", "$$\n", "\n", "Here, a model provides a best fit if it minimizes the $L_2$ norm of the error, which is defined as\n", "\n", "$$\n", " || \\mathbf{e} ||_2 = \\sqrt{ |e_0|^2 + |e_1|^2 + \\ldots} = \\sqrt{\\sum_{i=0} |e_i|^2} \\, .\n", "$$\n", "\n", "Equivalently, such a model is called the **least-squares** fit to the data because the sum of the errors squared is minimized.\n", "\n", "For the case of the linear model $y(t) = at + b$, the error can be written in matrix form:\n", "\n", "$$\n", "\\mathbf{e} = \n", "\\left[\n", " \\begin{matrix} \n", " x_0 & 1 \\\\\n", " x_1 & 1 \\\\\n", " \\vdots & 1 \\\\\n", " x_{n-1} & 1\n", " \\end{matrix}\n", "\\right]\n", "\\left[\n", "\\begin{matrix}\n", " a \\\\\n", " b \n", "\\end{matrix}\n", "\\right] -\n", "\\left[\n", "\\begin{matrix}\n", " y_0 \\\\\n", " y_1 \\\\\n", " \\vdots \\\\\n", " y_{n-1}\n", "\\end{matrix}\n", "\\right] \\, ,\n", "$$\n", "\n", "or $\\mathbf{e} = \\mathbf{M}\\mathbf{c} - \\mathbf{y}$. Here, $\\mathbf{M}$ represents the generic data matrix for the model, while $\\mathbf{c}$ are the model coefficients that are as yet undetermined.\n", "\n", "> **Note**: Generally $\\mathbf{M}$ is *not* square and $\\mathbf{e}$ is generally not zero. In other words, rarely can we hope to find $\\mathbf{c}$ from $\\mathbf{M}^{-1}\\mathbf{y}$! In the example above, we had **25 data points** and just **two unknowns** (the slope $a$ and intercept $b$). Too many equations means an **overdetermined** system!\n", "\n", "Recall that $||\\mathbf{e}||_2 = \\sqrt{\\mathbf{e}^T \\mathbf{e}}$, where $T$ represents the matrix transpose. Moreover, if $a$ and $b$ are found to minimize $\\sqrt{\\mathbf{e}^T \\mathbf{e}}$, then they also minimize $\\mathbf{e}^T \\mathbf{e}$. \n", "\n", "> **Note**: First, convince yourself $||\\mathbf{e}||_2 \\geq 0$, i.e., it is nonnegative. Then, convince yourself that any $x$ that minimizes a nonnegative function $f(x)$ also minimizes $f(x)^2$. \n", "\n", "Hence, the best values of $a$ and $b$ minimize the following:\n", "\n", "$$\n", "\\begin{split}\n", " \\mathbf{e}^T \\mathbf{e} \n", " &= (\\mathbf{M}\\mathbf{c} - \\mathbf{y})^T (\\mathbf{M}\\mathbf{c} - \\mathbf{y}) \\\\\n", " &= \\mathbf{c}^T \\mathbf{M}^T \\mathbf{Mc} - 2 \\mathbf{c}^T \\mathbf{M}^T \\mathbf{y} + \\mathbf{y}^T\\mathbf{y} \\, .\n", "\\end{split} \n", "$$\n", "\n", "Now, remember (e.g., from [Lecture 23](L22_UnconstrainedOptimization.ipynb)) that the minimum of a function $f(x)$ (nonlinear or otherwise) can be found when its *derivative* is zero. In this case, that function is $f(\\mathbf{c})$, and the \"derivative\" is the *gradient* $\\nabla ||\\mathbf{e}||$. Thus, differentiation leads to\n", "\n", "$$\n", "\\nabla \\mathbf{e}^T \\mathbf{e} = 2 \\mathbf{M}^T \\mathbf{M} \\mathbf{c} - 2\\mathbf{M}^T \\mathbf{y} = \\mathbf{0} \\, ,\n", "$$\n", "\n", "or\n", "\n", "$$\n", "\\mathbf{M}^T \\mathbf{M} \\mathbf{c} = \\mathbf{M}^T \\mathbf{y} \\, .\n", "$$\n", "\n", "This equation is often called the [normal equation](http://mathworld.wolfram.com/NormalEquation.html). The coefficients $\\mathbf{c}$ leading to the best fit are defined by\n", "\n", "$$\n", "\\mathbf{c} = (\\mathbf{M}^T \\mathbf{M})^{-1} \\mathbf{M}^T \\mathbf{y} \\, .\n", "$$\n", "\n", "Because $\\mathbf{M}^T \\mathbf{M}$ is square, the system has the same number of equations as it does unknowns. That alone does not guarantee a solution, but a solution is much more likely." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "