{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Unconstrained Optimization\n", "\n", "## Overview, Objectives, and Key Terms\n", "\n", "In [Lesson 21](L21_NonlinearEquations.ipynb), Newton's method solved equations by driving a residual to zero. Unconstrained optimization uses the same idea on a new residual: the derivative or gradient of an objective function.\n", "\n", "For one variable, an unconstrained optimum candidate satisfies\n", "\n", "$$\n", " f'(x)=0.\n", "$$\n", "\n", "For several variables, it satisfies\n", "\n", "$$\n", " \\nabla f(\\mathbf{x})=\\mathbf{0}.\n", "$$\n", "\n", "That makes the gradient from [Lesson 19](L19_NumericalDifferentiation.ipynb) the main bridge into optimization. This lesson focuses on unconstrained problems and `scipy.optimize.minimize`; constrained optimization is previewed only briefly because it is the topic of Lesson 23.\n", "\n", "### Objectives\n", "\n", "By the end of this lesson, you should be able to\n", "\n", "- Distinguish objective functions, design variables, local optima, and global optima.\n", "- Explain why unconstrained optimization can be viewed as root finding applied to derivatives or gradients.\n", "- Use gradient information to reason about a two-variable objective function.\n", "- Use `scipy.optimize.minimize` for one- and two-variable unconstrained problems.\n", "- Recognize constrained optimization as the next lesson's topic.\n", "\n", "### Key Terms\n", "\n", "- objective function\n", "- design variable\n", "- unconstrained optimization\n", "- local minimum and local maximum\n", "- global minimum and global maximum\n", "- stationary point\n", "- gradient vector\n", "- Hessian matrix\n", "- `scipy.optimize.minimize`\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Some Nomenclature\n", "\n", "Optimization problems often defined by minimizing an **objective function**\n", "\n", "$$\n", " \\min_{x} f(x) \\qquad \\text{or} \\qquad \\min_{\\mathbf{x}} f(\\mathbf{x}) \\, ,\n", "$$\n", "\n", "subject to the **inequality constraints**\n", "\n", "$$\n", " g(x) \\ge 0 \\qquad \\text{or} \\qquad \\mathbf{g(x)} \\ge \\mathbf{0} \\, ,\n", "$$\n", "\n", "and/or the **equality constraints**\n", "\n", "$$\n", " h(x) = 0 \\qquad \\text{or} \\qquad \\mathbf{h(x)} = \\mathbf{0} \\, .\n", "$$\n", "\n", "An optimization problem is *unconstrained* if it is subject to no inequality constraints and no equality constraints. Otherwise, the problem is *constrained*. In practice, unconstrained problems are often easier to solve using a variety of techniques for finding *extrema*. We'll start with unconstrained problems and then consider an important class of constrained problems known as *linear programs*." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Finding Extrema\n", "\n", "One of the great applications of differentiation is its ability\n", "to identify the *critical points* of a function $f(x)$, which\n", "include its *minima* and *maxima*, i.e., its *extrema*. Suppose\n", "$f(x)$ represents some quantity of interest, perhaps the cost \n", "of materials for some component or the signal-to-noise ratio of\n", "some sensor. If we want to minimize those costs or maximize those\n", "ratios, we need to use *optimization*. Although optimization problems\n", "and the techniques to solve them are quite diverse, we will stick to\n", "1-D problems in which the *objective function* to be minimized or \n", "maximized is continuous (as opposed to discrete).\n", "\n", "You've had calculus (and we've reviewed some its topics). Hence, you might recall that a function $f(x)$ exhibits an **extremum**, i.e., a **minimum** or **maximum**, at any point $x$ for which $f'(x)$ is zero and $f''(x)$ is nonzero. When $f'(x)$ *and* $f''(x)$ vanish, the point is often called a **saddle point**, though that term is more meaningful in multiple dimensions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "