Nonlinear Regression Models without Data Linearization

Autar Kaw, Jamie Trahan
University of South Florida
United States of America
kaw@eng.usf.edu

ClearAll;
Off[General::spell1]

Introduction

This worksheet illustrates finding the constants of a nonlinear regression model without data linearization. Three common nonlinear models are illustrated.

Exponential: y = ((ae)^bx)
Power: y= (ax^b)
Saturation: y = (((ax))/(b + x))

where a and b are the constants of the above regression models.

Given n data points ((Subscript[x, 1]),  (Subscript[y, 1])), ( (Subscript[x, 2]), (Subscript[y, 2])) , ((Subscript[x, 3]),  (Subscript[y, 3])),...., ( (Subscript[x, n]),  (Subscript[y, n])), you can best fit one of the above models to the data. In this worksheet, the constants a and b are calculated in the following steps:
	1) Find the sum of the squares of the residuals, Sr.
	2) Minimize Sr by differentiating with respect to a and b and setting the resulting two equations equal to zero.
	3) Solve the two nonlinear equations simultaneously for a and b.
	
Mathematica will then return the real solutions of a and b. To learn more about nonlinear regression models see the Nonlinear Regression model worksheet.

Section 1: Input Data

Below are the input parameters to begin the simulation. This is the only section that requires user input. Once the values are entered, Mathematica will return the constants of the nonlinear regression model that is specified by the user with the modeltype variable.
NOTE: Before evaluating the worksheet, the user must enter initial guesses of the constants of the model  a and b. For reasonable initial guesses, use the solution from the Nonlinear Model with data linearization worksheet. For convergence, use initial guesses a and b close to the values of a and b obtained by using data linearization. 

Input Parameters:

Number of data points, n

n = 5

Array of x values, X

X = {10, 16, 25, 40, 60}

Array of y values, Y

Y = {94, 118, 147, 180, 230}

For exponential model call model type to be "exponential"
For power model assign the model type variable as "power"
For saturation growth model, assign the model type variable to be "growth"

modeltype = "power"

Insert your initial guesses for a and b here. Reasonable initial guesses for a and b can be obtained from data linearization models.

Initial Guess values for a

Ainit = 30.199

Initial Guess values for b

Binit = 0.489

Clear[a];
Clear[b];
Clear[Sr];

Section 2: Finding the constants of the model

Assigning the proper regression model

If[modeltype == "exponential", 
        f[x_] = a*E^(b*x)];
If[modeltype == "power", 
        f[x_] = a*x^b];
If[modeltype == "growth", 
        f[x_] = (a*x)/(b + x)];

Calculating the sum of the square of the residuals

Sr = 0;
For[i = 1, i ≤ n, i++, 
        Sr = Sr + ((Y[[i]] - f[X[[i]]])^2)];

Sr

Differentiating the sum of the square of the residuals with respect to the constants of the model, a and b, to setup two simultaneous nonlinear equations and two unknowns.

eqn1 = Expand[D[Sr, a] == 0];
eqn2 = Expand[D[Sr, b] == 0];

Solving the two simultaneous nonlinear equations. We are using FindRoot since we are looking for real solutions.

soln=FindRoot[{eqn1, eqn2},{{a,Ainit},{b, Binit}},MaxIterations -> 1000];
a /. 
        soln[[1]];
a = %;
b /. 
        soln[[2]];
b = %;
Print["The constants of the ", 
      modeltype, " model are a = ", a, " and b = ", b]

Plotting the observed values and the predicted curve

predicted = f[x];
observed = 
        Table[{X[[i]], Y[[i]]}, {i, 1, 
            n}];
ttl = 
        Print[modeltype, " regression model, y vs x"];
points = 
        ListPlot[observed, PlotStyle -> PointSize[ .02], 
          DisplayFunction -> Identity];
lin = 
        Plot[predicted, {x, Min[X], Max[X]}, 
          DisplayFunction -> Identity];
Show[points, lin, DisplayFunction -> $DisplayFunction]

Conclusion

Mathematica helped us in illustrating three common nonlinear models of regression. Answer the following questions using the worksheet

1) Verify each of the models by using data that exactly follows the regression model.
2.) What is the difference in the solution between nonlinear models obtained via data that is linearized and data that is not linearized.

References

[1] Autar Kaw, Holistic Numerical Methods Institute, http://numericalmethods.eng.usf.edu/nbm, See
How does Nonlinear Regression Work?