CHAPTER 06.04: NONLINEAR REGRESSION: Polynomial Model: Derivation: Part 2 of 2
So what do I get as the first equation? So from . . . from the del Sr by del a0 equal to 0, this is what I get, I will get -2 summation, i is equal to 1 to n, yi, then plus 2 a0 summation, 1, i is equal to 1 to n, then plus 2 a1 summation, i is equal to 1 to n, xi, and the last one which I will get is 2 a2 summation, i is equal to 1 to n, xi squared equal to 0, that's what I'm going to get from the first summation which I have. Now, you'll have to . . . you'll have to refer to what I have done earlier to be able to look at that. So this . . . this one . . . this one is implying this, and this one implies what? This one implies -2 summation, i is equal to 1 to n, and I will get xi yi, plus 2 a0 summation, i is equal to 1 to n, xi, plus 2 summation . . . a1 summation, i is equal to 1 to n, xi squared, and plus 2 a2 summation, i is equal to 1 to n, xi cubed equal to 0, so that's what I'm going to get from the second equation. And the third equation, which was del Sr by del a2 equal to 0, that'll give -2 summation, i is equal to 1 to n, xi squared yi, plus 2 a0 summation, i is equal to 1 to n, xi squared, and then the next number will be 2 a1 summation, i is equal to 1 to n, xi cubed, and the last term in this last equation will be 2 times a2 summation, i is equal to 1 to n, xi raised to the power 4, and that will be equal to 0. So what you are basically getting is you're getting three equations. You're getting one equation right here, okay? That's your first equation right there, then this is your second equation, and this is your third equation which you are getting here, so you've got to set up these three equations. Now, if you look at these three equations which I have here, you've got to understand that when I'm going to set up these equations, what can I take to the right-hand side? I will be able to take this to the right-hand side. The reason why I'm able to take this to the right-hand side is is because that's the known quantity. So I want to take all the right-hand sides to the . . . all the knowns to the right-hand side, and then also, for example, this will be nothing n, because that's summation of 1, going from 1 to n, so I'm adding 1 n times, so I'm going to get n from there, and all of these will stay on the left-hand side, because I have a0, a1, and a2 as the unknowns. Now, if I look at the second equation, again, this quantity right here will go to the right-hand side . . . this quantity will go to the right-hand side, because this is a known quantity, all the xs and ys are known, again these quantities will stay on the left-hand side, because I have a0, a1, and a2 as my unknowns. Same thing here, when I look at this one, this quantity here will go to the right-hand side, because xis and yis are known to me, and that will go to the right-hand side, and I'll have these stay on the left-hand side, because I have a0, a1, and a2. Now, before I do that, I'll go back to equation number 1 here, and what I'm going to do is I'm going to get rid of this 2, and the reason why I'm able to get rid of this 2 is because I can divide both sides by 2, because the right-hand side is 0. Same thing here, I can get rid of this 2, because the right-hand side is 0, I can get rid of this 2, because the right-hand side is 0, so that's how I'm able to simplify it a little bit. So if I now write down the equations this is what I'm going to get, I'm going to . . . the first equation will turn out to be n times a0, plus a1 times summation, xi, i is equal to 1 to n, plus a2 summation, i is equal to 1 to n, xi squared is equal to summation, xi yi, because I took that to the right-hand side, because the quantity is knowns. Now, the second equation which I am going to get will be something like this, it'll be a0 summation, xi, i is equal to 1 to n, and a1 summation, i is equal to 1 to n, xi squared, then the third quantity will be i is equal to 1 to n, xi cubed, and the right-hand side which I took the . . . I told you that I took the known to the right-hand side, that will turn out to be something like this . . . sorry, this is just yi, okay, so this is nothing but yi, and this one will be xi times yi, that's from the second equation which I get. Now here, the equation number 3 will be a0 summation, i is equal to 1 to n, xi squared, so that's what I get as the coefficient of a0, the coefficient of a1 will turn out to be summation, i is equal to 1 to n, xi cubed, plus a . . . this should be a2, and then a2 summation, i is equal to 1 to n, xi 4 . . . xi raised to the power 4, summation, i is equal to 1 to n, xi squared yi. So I have rewritten my three equations and three unknowns now, my three unknowns being a0, a1, and a2, and they are just simply being multiplied by certain summations, and I took the known quantities to the right-hand side, like this one I took to the right-hand side, this one I took to the right-hand side, and this one here I took to the right-hand side. So this sets up three equations, three unknowns. The good news is that these three equations, three unknowns are simultaneous linear equations. So since they are simultaneous linear equations, I don't need to worry about having to solve simultaneous nonlinear equations like I have to for other nonlinear regression models, such as power model, or exponential model. So the good news here is that I am just going to set up these simultaneous equations, but they are all linear in nature. So let me write down these equations in the matrix form, and that will form the basis of this. So the matrix form of these equations which I just developed will give a clear picture of how we're going to set up these coefficient matrix and the right-hand side so that we know what the unknowns are. So we've got a0, a1, and a2 as the unknowns, and the coefficients are n, summation, xi, and then summation, xi squared, then we have summation, i is equal to 1 to n, xi, then summation, i is equal to 1 to n, xi squared here, and then summation, i is equal to 1 to n, xi cubed here, and then same thing here, I have summation, xi squared here, summation, xi cubed here, and summation, i is equal to 1 to n, xi to the power 4 here. So that's what I get in the coefficient matrix, and the right-hand side is nothing but summation of yi, i is equal to 1 to n, summation, xi yi, i is equal to 1 to n, and summation, i is equal to 1 to n, xi squared yi. So you have all the known quantities on the right-hand side, the unknowns are a0, a1, and a2, and then the coefficient are also known quantities, and now you can use any kind of techniques which you have learned in your simultaneous linear equations methods, such as Gauss elimination, Gauss-Seidel, LU decomposition method, to calculate a0, a1, and a2. Now, keep in mind that what happens is that when you start doing polynomial regression like this, and let's suppose you are doing a higher order polynomial regression, like a fourth- or a fifth-order polynomial, you can see that the order of the terms is going to be very different from one row to another, like, for example, here you just have the number of elements in here, and here you have xi raised to the power 4. So if your xi values of the x values are very small, then xi raised to the power 4 will be even smaller. If xi is a large number, then xi raised to the power 4 will be very large numbers. So what happens is that, whenever you're doing polynomial regression, it creeps in some ill conditioning in the system of equations, so you should be very careful when you are conducting polynomial regression, especially higher polynomial regression, like fourth- and fifth-order polynomials, that are you getting a coefficient matrix which is well conditioned or not, so you should be careful about that. So once you find a0, a1, and a2, then your regression model is nothing but a0, a1 x, plus a2 x squared, and that gives us . . . that gives us the regression model, and that's what we were trying to find in the first place. And that's the end of this derivation. |