Optimization Methods

TL;DR

• Techniques for locating maxima or minima of functions, and sometimes roots, used across multiple fields.
• Two principal categories: local optimization (search within a region) and global optimization (search over the entire domain).
• Common methods: gradient descent (local, uses the gradient to reach a minimum) and Newton–Raphson (described here as a global method used to find roots, using first and second derivatives).

Definition

Optimization methods, also known as maxima and minima methods, are techniques used to find the maximum or minimum value of a function.

Explanation

Optimization methods solve problems that involve maximizing or minimizing some quantity. They are applied in a wide range of fields, including mathematics, engineering, and economics.

There are two main types:

• Local optimization methods: used to find the maximum or minimum value of a function within a specific range or region.
• Global optimization methods: used to find the maximum or minimum value of a function over the entire domain of the function.

Two common methods described are:

• Gradient descent: a local optimization method for finding a function’s minimum. It starts at a point on the function and iteratively moves in the direction of steepest descent, using the gradient (a vector of partial derivatives) to determine that direction until a local minimum is reached.
• Newton–Raphson: described here as a global optimization method used to find a root of a function (the point where the function intersects the x-axis). It begins with an initial guess for the root and iteratively improves the guess using the derivative of the function; the method uses the second derivative to assess curvature and adjust the guess.

Examples

Gradient descent

The gradient descent method is a local optimization method that is used to find the minimum value of a function. It works by starting at a random point on the function and iteratively moving in the direction of the steepest descent until it reaches the local minimum. The algorithm uses the gradient of the function, which is a vector of partial derivatives, to determine the direction of the steepest descent.

Newton–Raphson

The Newton-Raphson method is a global optimization method that is used to find the root of a function, which is the point where the function intersects the x-axis. It works by starting at a guess for the root and iteratively improving the guess using the derivative of the function. The algorithm uses the second derivative of the function to determine the curvature of the function and to adjust the guess accordingly.

Use cases

• Applied across mathematics, engineering, and economics to solve problems involving maximization or minimization of quantities.

Related terms

• maxima and minima
• local optimization
• global optimization
• gradient descent
• Newton–Raphson
• gradient (vector of partial derivatives)
• derivative
• second derivative
• root