The standard Kalman filter is designed mainly for use in linear systems, however, versions of this estimation process have been developed for nonlinear systems, including the extended Kalman filter and the unscented Kalman filter. Since many real-world systems cannot be described by linear models, these nonlinear estimation techniques play a large role in numerous real-world applications.

While the standard Kalman filter is a powerful estimation tool, its algorithms begin to break down when the system being estimated is nonlinear. Fortunately, a version of the standard Kalman filter, known as the extended Kalman filter (EKF), has been extended to nonlinear systems and relies on linearization in estimating these nonlinear systems. Linearization operates on the principle that at a small section around a selected operating point, a nonlinear function can be approximated as a linear function. This linearized function can be derived from the nonlinear function using the first-order terms in a Taylor series expansion shown in Equation \ref{nfk:le}.

\begin{equation}\label{nfk:le} \boldsymbol{g}(x) \approx \boldsymbol{g}(a) + \frac{\partial\boldsymbol{g(x)}}{\partial\boldsymbol{x}}\Bigg|_{x = a}(x-a)\end{equation}

Using this method of linearization, an EKF will follow the same propagate and update process as the standard Kalman filter, but with a few modifications to the standard equations. During the propagate step, rather than using Equation 5 in Section 2.8, the state vector is instead estimated by evaluating the nonlinear system model equations at the most recent state estimate as shown in Equation \ref{nkf:sp}. Additionally, in the state covariance matrix propagation, the state transition matrix, $\Phi$, is replaced with a matrix $F$, which is a Jacobian matrix containing the first-order partial derivatives of the nonlinear system model equations.

\begin{equation} \label{nkf:sp} \boldsymbol{x}_{k+1} = f(\boldsymbol{x}_k,\boldsymbol{u}_k) \quad\quad\quad F = \frac{\partial\boldsymbol{f}}{\partial\boldsymbol{x}}\Bigg|_{\hat{\boldsymbol{x}}}\end{equation}

In the update step, the expected measurement vector is derived using the nonlinear measurement model equations, evaluated at the most recent state estimate as provided in Equation \ref{nkf:mu}. The measurement model matrix in each of the update equations is also replaced with the $H$ Jacobian matrix containing the first-order partial derivatives of the nonlinear measurement model equations.

\begin{equation}\label{nkf:mu} \boldsymbol{y}_{k} = h(\boldsymbol{x}_k) \quad\quad\quad\quad\quad\quad H = \frac{\partial\boldsymbol{h}}{\partial\boldsymbol{x}}\Bigg|_{\hat{\boldsymbol{x}}}\end{equation}

Though the EKF can be a powerful tool in estimating states in a nonlinear system, there are some limitations of its use. An EKF is designed in such a way to optimally update the state vector and state covariance matrix, assuming that the state covariance matrix is within the linear region of the linearization. However, if the uncertainties in the state covariance matrix grow to be larger than the size of this linear region, then the state covariance matrix can no longer accurately reflect the actual error in the system and divergence can occur. Typically, an EKF is best suited for applications with enough measurements to keep state uncertainties relatively low.

While the EKF works well for a majority of nonlinear systems, there are some cases where an EKF is not well suited, such as if the system is very nonlinear or poorly observable. In these particular systems, the unscented Kalman filter (UKF) can provide a more reliable estimation. Most navigation systems do not fall in this category, but the UKF is still seen in some systems.

The UKF estimates a nonlinear system by carefully selecting a number of points, known as sigma points, that adequately describe the state vector and associated uncertainty. These sigma points are then propagated through the nonlinear equations to estimate the next state vector and related uncertainty.

Though this estimation process is not as prone to divergence, a UKF does require quite a bit of computational efficiency to calculate the sigma points and propagate them through the nonlinear system. This is especially true for systems that have a large state vector, requiring large numbers of sigma points to be calculated and propagated.