Using the least squares method for an exponential function

[u/golubevich123]

Hey there!

I have an experimental data set with distance and time values and I need to make a regression model for it, preferably using the least squares method.

For the y=ax+b function it's really easy and I already did it. But what about a function y=a^x+b? My idea is that you could come to the linear least squares method through substitution and logarithmic operations, but I can't really get there and understand how...

Is it possible to use the least squares method to get a regression model of a y=a^x+b function? If yes, then how? 

[u/Shevek99]

For exponentials, the typical way is

y = A e^(Bx)

and then take logarithms, transforming it in

ln(y) = ln(A) + B x

that is a linear fitting.

The form

y = a^x + b

is a nonlinear fitting, that cannot be linearized and must be solved using approximate methods.

[u/golubevich123 → u/Shevek99]

Thank you! 

[u/Bounded_sequencE]

Do you really mean "y = a^x+b" ?

If yes, take logarithms on both sides to linearize, and do least squares on "ln(y)" instead. Most spread sheet programs (and gnuplot) natively support that as logarithmic regression, since it's so common.

If not, and it really is "y = a^x + b", you're out of luck: You are now dealing with non-linearizable least squares, and will likely need numerical tools to find approximate solutions.

[u/philolessphilosophy]

You can linearize the data with logarithms and then use linear least squares, but this is not necessarily going to give you the actual best fit to the data. You should just use a general least squares fitting algorithm. Many graphing calculators have these. It will be an approximate answer, but that's all you really need.

[u/golubevich123 → u/philolessphilosophy]

The problem is that I need to actually show how did I get a result, not just plugging the data in somewhere. 

[u/philolessphilosophy → u/golubevich123]

There isn't a formula like you're wanting. Could you just explain how non linear least squares fitting works? If not, would it be acceptable to write a program that implements a non linear least squares algorithm? This is about the best you can do.

The logarithm method would work, but again, it won't give you the actual optimal fit.

[u/golubevich123 → u/philolessphilosophy]

Yeah, I'll think about it. 

Another person suggested to use a function y=a*e^b*x and take logarithms from it to transform it to a linear fitting, then just substitute things - the only thing is that (I assume) the least squares method in this case will minimize the logarithmic error, right? 

[u/Shevek99 → u/golubevich123]

Yes. If you assume the same variance for all y_i data, then to transform

y = A e^(Bx)

into

ln(y) = ln(A) + B x = a + b x

now the variances of ln(y) are different. The function to minimize is now

𝜒² = sum ((ln(yi) - (a + b xi))² yi² )

and the result obeys the usual formulas

b = (S Sxy - Sx Sy)/(S Sxx - Sx²)

a = Sy/S - b Sx/S

where now S, Sx, Sy... are weighted sums

S = sum yi²

Sx = sum xi yi²

Sxx = sum xi² yi²

Sy = sum ln(yi) yi²

Sxy = sum xi ln(yi) yi²

[u/MezzoScettico → u/philolessphilosophy]

Edit: Sorry, this was to OP. I'm sure you know all this. I misread who I was responding to.

Least squares is about minimizing a function, E = sum(y(x_i) - y_i)^2, the squared differences between what your model predicts y(x_i) and the measured value of y. This squared error E can be thought of as a function of your unknown parameters you're trying to estimate.

So you're trying to minimize a function of several variables.

The general strategy to finding the minimum is to find a place where the gradient of E is a minimum. For linear least squares that leads to a set of linear equations that is relatively easy to solve.

For nonlinear least squares you need to go to optimization methods, general methods to find the minimum of a function of multiple variables. So nonlinear optimization is what you want to investigate. Either nonlinear least squares or a general nonlinear optimization method is built into many math libraries.

Steepest descent is the naive algorithm a lot of beginners go for. It's easy to understand and easy to implement. But it's slow and inefficient.

Newton's Method is more sophisticated but takes more math to implement.

Either way it's an iterative approach, starting with an initial guess and gradually improving it at each step. There could be thousands of steps.

And for those and other methods there are adaptive methods which adjust the step size on the fly.

[u/MezzoScettico → u/golubevich123]

What's your computer environment? I have a longer comment below on the general problem, which is nonlinear optimization, and you may very well have a math library that already gives you such a function.

I believe even Excel has nonlinear optimization built in, though that may be an add on. Glancing at my Excel menus, I think it's "Goal Seek" which appears under the "Data" tab.

[u/golubevich123 → u/MezzoScettico]

Thank you for your answer! y=a*e^b*x will work perfectly for me as well, so that's what I'll stick with.

It's not the computer environment problem, the problem is that I'm obligated to calculate things through formulas (obviously with a computer help for the calculation themselves), so I can't just plug in numbers in an Excel function for example. But with this exponential function, everything should be fine! 

[u/tstanisl]

No. LS requires all terms to be linear functions of a and b. Thus a*x^3 is fine while a^x is not. Try using non-linear LS, iterative method where non-linear terms are approximated using first-derivative.

[u/golubevich123 → u/tstanisl]

Could you please tell more about it or give a link to a study? 

[u/seanv507 → u/golubevich123]

Look up levenberg Marquardt method

or Newton s method (which also uses the second derivative)

Basically they are iterative methods, where you approximate the nearby region by a quadratic surface and move down the surface.

For only 2 variables I feel you could just do brute force search... (not optimal solution)

[u/tstanisl → u/seanv507]

There are methods like Interval Newton Method that can isolate, enumerate or cut-off all solutions providing proven global solution.

[u/FormulaDriven]

For a given choice of a, the sum of squares is minimised when

b = (1/n) * sum(yi - a^xi)

(so n datapoints (x1, y1), ... (xn, yn)).

So you could pick a candidate value for a, calculate b from the above then the sum of square errors is sum(a^xi + b - yi)²

Now try varying a, recalculating b, to see if you can reduce that error sum.