3.3.1 Error

We would like to have a way to measure how close a data point is to a particular function that may be chosen to model a data set. In order to do this, we have to agree on a way to measure the distance between a point and some curve. For simplicity, we will consider the case in which the curve is a straight line. Most people, when asked to measure the distance between a point and a line, will find the length of the line segment that is perpendicular to the line in question and intersects the point. But there are other possibilities. For example, we could also draw a horizontal or vertical line segment from the point and use one of those distances.

[NEED A PICTURE]

The definition of distance commonly used to define the error involved in data fitting is not the intuitive perpendicular distance, but the vertical distance. That is, identifying x as the independent variable and y as the dependent variable, we use as our measure of error the distance between the measured and computed values of y. There are two reasons for this choice. First, the aim of most data fitting is prediction of the value of a function at points where we do not have measured data. So a reasonable definition of the error at a data point is the difference between the measured value and the prediction made by the function we have defined. The second reason is that this definition of the error leads naturally to a model of our problem as a linear system. Remember that the limitation on the types of calculations that a computer can perform makes linear systems the basic building blocks of algorithms in computational science.

So suppose that the function f is used to fit a set of data points, and let (x_i, y_i) be one of these points. We will define the error, or residual, at the point (x_i, y_i) to be