3. Example Computations

a. Measurement Set

Let's compute the statistics of our measurement set: 45.66 45.66 45.68 45.65

(Keep in mind that this is an absurdly small measurement set hardly worthy of statistical analysis. We're using it only to demonstrate the random error analysis process).

(1) Compute MPV, Equation F-1

According to the rules of significant figures, the measurement sum has 5 sig fig.

In addition to isolating errors, repeating measurements can also increase accuracy. In this case, we’ve gone from 0.01 units to 0.001 units. Of course, the more measurements made, the stronger that additional accuracy. Four measurements, as in our example, are not really enough for a true statistical picture.

(2) Compute the residuals

Since this represents an intermediate computation, carry an additional digit. Residuals have the same units as the measurements.

(3) Square and sum the residuals

Keeping in mind the addition rule for significant figures, the measurement sum has 3 sig fig.

By the way, the term least squares comes from the fact the MPV is the number which results in the smallest (or least) sum of the squares of the residuals. Any other number will result in a larger sum.

(4) Compute the standard deviation, Equation F-2

The standard deviation has the same units as the measurements.

(5) Compute the Error of the MPV, Equation F-4

The error of the MPV has the same units as the measurements.

(6) Results

From (1), the MPV is good to 0.001

We should express the standard deviation and error of the mean to the same resolution level.

MPV = 45.662 ±0.013; E_MPV = ±0.006

So what’s all this mean? Based on our measurements:

• The most probable value of the measured quantity is 45.662;
• 68% of our measurements fall within ±0.013 of the MPV;
• The error in the MPV is expected to be ±0.006.

b. Example with Angles

One of the problems working with angles is the tendency to carry insufficient digits when converting from deg-min-sec to decimal degrees. For some reason, three decimal places seems to be the norm but this can cause substantial rounding error computing measurement statistics.

To demonstrate this, lets look at a simple example consisting of 4 angles: 168°42'30", 168°42'10", 168°42'15", and 168°42'25".

(1) Convert to and carry three decimal places.

The computations aren't too onerous (a byproduct of using only three decimal places). However, once the angles are converted to decimal degrees, it's difficult to interpret magnitudes of subsequent calculations. Residuals are in decimal degrees; what are those in terms of the original mixed units? It's hard to judge if the residuals (and MPV, SD, E_MPV) make sense for the measurement set.

(2) Convert to and carry seven decimal places

While the process is the same as the first, two things are readily apparent

• There are a lot more numbers to write down and uses in calculations
• The additional four decimal places have a significant impact the MPV, SD, and E_MPV values.

Conclusion? Carrying only three decimal places results in substantial intermediate rounding. As we learned in the Significant Figures chapter, carrying more digits than needed and rounding at the end means less biased results.

But seven decimal places are sooooo many to carry. Would six be enough? Five? Four? We discussed how to convert angles to decimal degrees to the correct number of sig fig in the Significant Figures chapter. Although we could determine how many decimal places are needed, we still have the same issue as before: it's difficult to mentally compare decimal degree values to deg-min-sec.

But there's a better, simpler way

(3) Work with smallest unit

An angle is a mixed units quantity and usually, only the smallest unit changes, the others don't (more on that in a bit - be patient). Recognize a pattern: in our angles, each has 168° and 42'; only the seconds vary. We can simplify computations considerably if we work with just the seconds. To do that, subtract 168°42' (both of which are exact) from each angle.

Lookie there - the same exact results as carrying seven decimal places! And the computations are soooo much easier (you could almost do them all in your head). The residuals are in the smallest unit, seconds, so are easy to compare to their respective angle. We don't have to carry a lot of decimal places or figure out significant figures for the converted angles.

OK, but what about an angle set where the minutes vary? Let's use: 89°36'58", 89°36'55, 89°37'05', 89°37'10"

To work with only the seconds portion, subtract 89°36' from each angle. For the last two angles, the results are 01'05" and 01'10", respectively; write them as seconds: 65" and 70".

Could we have instead subtracted 89°37' from each angle? Sure:

The results are the same as before but this time we must keep track of negative numbers. It less error prone to subtract 89°36' instead of 89°37'.