People are familiar with a standard linear regression (LR) method that makes assumptions about the X and Y variables in the analysis. Excel’s built-in regression methods make these same assumptions.

The X variable, also called the independent variable, is assumed to be known precisely, and all error in the regression is assumed to be in the measurement of the Y variable, the dependent variable.

This assumption is reasonable enough. But there are cases where errors in the values of X cannot be discounted. For example, X and Y may be comparisons of the output of two different machines for given inputs. When the outputs are compared, both machines are subject to errors in measurement, so the usual assumptions are invalid.

Deming Regression Example

Suppose Machines A and B had the following output. At each nominal output value, the results of two runs of each machine are tabulated. There is some variability in measurements of each machine’s output. The averages of each machine at each nominal output level are shown next to the individual measurements.

We may be tempted to plot each machine’s outputs as Y values against the nominal X values.

We may even compute the lines of best fit for the two machines. We can see that they are close, but not exactly the same. The problem with this analysis is that we are not directly regressing one machine against the other.

In the Deming Regression, outputs of the two machines are plotted against each other. Pairs of actual outputs are measured for each nominal output, and the differences between the two measurements from each machine will be used to estimate the variation of each machine’s outputs. The averages of each pair form the basis of the regression, corrected for the machine variation.

The table below shows three sets of calculations. The first row shows a standard linear regression of the measured data. The second row shows a standard linear regression performed on the averages of each pair of values. The third shows the Deming regression analysis. Details of the Deming regression calculations are not given here, but a motivated reader can check the links in the references at the end of this article.

The data are not too different, and in fact the correlation in the LR of the average values and of the Deming regression are identical. In the plot below, lines have been added to show the three sets of results in the table above, as well as the 45 degree line, where Machine B = Machine A. The lines are practically on top of each other.

In the example above, there is little difference in the output of the various LR models. However, if Machines A and B exhibit greater variation in their output, there can be noticeable differences.

Deming Regression Utility

I’ve created a small utility that facilitates Deming Regression analysis in Excel. Read about it and follow the download link in Deming Regression Utility.

References

Chapter 17:  Creating  Custom  Functions, Excel for Chemists: A Comprehensive Guide, 2nd Edition. by E. Joseph Billo, Copyright 2001 by John Wiley & Sons, Inc.

Incorrect Least-Squares Regression Coefficients in Method-Comparison Analysis by P. Joanne Cornbleet and Nathan Gochman, Clin. Chem. 25/3, 432-438 (1979).

Plotting Without Empty Cells

Here are some typical Excel charts, to remind us what they look like with fully populated data ranges. I’ve placed data labels on the charts to help show their behavior when empty cells (and other non-numeric fillers) are included in the source data.

Plotting With Empty Cells

The default behavior for most charts is to treat an empty cell by leaving a gap on the chart. Here are XY and Line charts that show how this looks. Note the lack of data labels where the gaps occur.

You can instead have Excel treat blanks as zeros (which in general is deceiving, since empty cells mean the absence of a value, not a value of zero). Note the data labels indicating a value of zero.

You can also tell Excel to interpolate over the missing data point. No data point means no data label.

Purists may say that the interpolation option may cause readers to think there is data along the line that spans the gap. Maybe so, but if the line or XY chart plots actual data points with markers, this is less of a risk.

Column charts offer only the gap and zero options for empty cells. Interpolating makes no sense in a column chart, which has discrete bars for each actual data value. The chart with a gap has no data label at the gap, the chart with zeros has a zero data label.

If we change the vertical axis so that the horizontal axis crosses below zero, we see that the first chart has no data point (bar) corresponding to the gap, but the second has a point (bar) with a value of zero.

Area charts seem to offer both the zero and interpolation options, but in both cases, the chart plunges to zero without a corresponding data label. If we mess with the vertical axis, the chart still plunges exactly to zero in both cases.

A more suitable appearance for an area chart would be one that leaves a real gap, with vertical edges, as below. To get this I had to make a two-axis chart, with a hidden series on the primary axis to provide the A-B-C category axis labels, and an area chart on the secondary axis, with two points at the second category (Y=4 and zero) and two at the fourth category (Y=zero and 8). A date scale on the secondary category axis aligns these points above each other to produce the vertical-sided gap. This only took me three minutes to construct, but an average user might never quite get it.

Pie charts don’t seem to offer any options for dealing with blanks, but the grayed-out default is the gap option. In the chart at the right, there is no point (wedge) and no corresponding data label where the empty cell would be plotted.

Apparently all charts except the Area chart provide a non-plotted point for an empty cell in the data range, though we can spoil that by selecting the option to treat a gap as a zero value.

Simulating Empty Cells

The main confusion comes along when someone uses formulas to fill the source data range.

The problem arises because  a formula that links to an empty cell

=A1

doesn’t display a blank, it displays the value zero.

Unfortunately Excel has no formulaic way to simulate a blank in a cell. There is no formulaic way, then, to get an actual gap in a chart. A function like BLANK() or NULL() would be nice, and we’ll keep asking. But we won’t hold our breath. And if we really need it, we can hack something in VBA.

Andy Pope shares a workaround in Broken Lines for formula linked data, but it requires one or more dummy series with a line color matching the chart background, which obscure the existing line where the gaps should go.

What people try to do to fake a blank is return “” in the formula.

=IF(A1=”",”",A1)

The result looks to you and me like a blank, but to Excel, “” is a text string, albeit a short one, and it is therefore assigned a value of zero. These charts use “” in a logical yet misguided attempt to simulate an empty cell. All charts plunge to zero, and all display a data label showing the value zero.

People have learned that the #N/A error in a worksheet cell is often not plotted in a chart. In a formula, the function NA() returns this error.

=IF(A1=”",NA(),A1)

What #N/A does is prevent the rendering of a marker in XY and line charts: notice the lack of markers and data labels. It’s not a gap, but it’s pretty much the next best thing.

This doesn’t work as well in other chart types. The column and area charts place a data label showing #N/A at zero. For these chart types it would be better to use “” or even zero, and apply a custom number format that suppresses the display of zeros (a format like “0;-0;;”).

A pie chart will behave like a column or area chart: there is no visible point (wedge) but there is a data label showing zero or #N/A where the wedge would be located.

Summary

The least deceptive way to display an empty cell in a chart, is to treat it as a blank. In line or XY charts, it is generally acceptable to interpolate across the blank with a line, if there are markers plotted for every actual data value. In the vast majority of cases, it is not appropriate to treat empty cells as zeros.

To simulate an empty cell in a line or XY chart, use NA() in your formula to produce the #N/A error in the cell. This produces an ugly #N/A error in the cell, but this error suppresses plotting of a point, and the line spans the position corresponding to the error’s slot in the data range. It’s not as good as a gap, but it’s perhaps the next best thing.

In an area or pie chart, the best approach is to use the null string “”, in conjunction with a number format that suppresses the display of zero values.

For an area chart you have to stand on your head to create a complicated chart that simulates a vertical edged gap and provides appropriate category axis labels.

Alternatively you could rely on VBA to clear cells which need to be cleared, but this routine would have to run every time the data range is changed, and it would have to reintroduce formulas into previously cleared cells as well as clear cells where blanks are needed. Or it could do all of the calculations in VBA, eliminating the worksheet formulas.

Excel Pivot Tables are a very powerful feature, and the AutoFilter is also very useful. The row fields of the Pivot Table can be manipulated like an AutoFilter, but the data fields cannot be sorted or filtered. And if you try to apply an AutoFilter to a pivot table, you find the menu command disabled.

But I learned a trick this week at the Excel Dashboard and Visualization Bootcamp. If you select a cell which borders on the pivot table, but is not part of the pivot table,…

…the AutoFilter menu items are now enabled.

Select AutoFilter, and the Pivot Table has been AutoFilter enabled.

These conferences are great.

AutoFilter by Selection

Access has a feature that lets you quickly apply a filter based on the selected item. Wouldn’t it be great to have this in Excel?

Well, this feature exists, and it’s built in. But it’s not present on the default menus or ribbons. You have to dig it out and install it yourself.

Excel 2003 and earlier

On Excel’s Tools menu, select Customize, click on the Commands tab, select the Data item in the Caegories list, click on the AutoFilter item, and drag it to a convenient place on a toolbar or menu.

I’ve put this just under the “regular” AutoFilter item on my Data menu/Filter submenu.

Excel 2007

Click on the down arrow to the right of the Quick Access Toolbar, and select More Commands from the popup menu.

In the oversized dialog that appears, select Commands not in the Ribbon from the right hand dropdown, select AutoFilter, click Add, and press OK.

The AutoFilter icon appears on the QAT.

What the “New” AutoFilter does

This Autofilter command shares a name with the “other” AutoFilter command, but it has a unique icon, and its functionality is enhanced. The command creates an AutoFilter using the range that includes the active cell, then it filters this range based on the active cell.

One of the answers to the generic question “Why do we need charts?” is to gain insight into a problem before we try to solve it.

Dick showed a couple of charts, lame little pie charts actually, which hand-waved a clever little trick he used to account for directions on either side of 360°. I’ll show how using more carefully crafted charts could have helped lead to the same result more quickly, by showing exactly what the relationships are. I will also show how such charts can easily show when a simplified approach is too simple, and leads to a very wrong answer.

When charted, the original data looks like the following.

By inspection, we can see that Directions 4 and 5 are pretty nearly straight lines as drawn, while the other three have points on either side of 360°. To turn these into linear fits, we need to add or subtract 360° so that the lines can cross 360° (or 0°).

Here is the data, corrected by adding 360° as appropriate. The calculated wind directions at 3000 ft are shown in blue.

Which leads to this chart. All five cases now follow nearly straight lines.

Dick’s Approach

The trick is knowing which data needs to be “enhanced”.Obviously you need to correct the ones which have values concentrated at the ends of the 0 to 360° range.

Dick’s approach was to examine the three directions in each case, and if the difference between the min and the max exceeded 180°, he added 180° and moduloed by 360°. This took data which was arranged around 360° and rearranged it around 180°. To these cases he then added 180° to the calculated result, and again moduloed by 360°.

Dick’s magic formula is

=IF(MAX(C4:C6)-MIN(C4:C6)>180,
    MOD(FORECAST($B7,MOD(C4:C6+180,360),$B4:$B6)+180,360),
    FORECAST($B7,C4:C6,$B4:$B6))
 

which is array-entered (by holding Ctrl+Shift while clicking Enter), so in the formula bar it is surrounded by curly brackets. B4:B6 are the altitudes 2000, 5000, and 10000 in the first column, B7 is the target altitude 3000, C4:C6 are the wind directions under the label Direction1, and this formula is entered in C7. As written, this formula can be copied and pasted into D7:G7 to provide all five calculations.

Instead of a difference of 180°, Dick could have used another cutoff value to decide which data needs correction. Alternatively, a check of the correlation or residuals of the three points could indicate whether the points are nearly aligned or far from linear. These latter approaches are more computationally intensive.

Dick’s calculated results for 3000 ft are shown by the squares in the chart above. They are all pretty close to the line segments connecting the 2000 ft and 5000 ft data.

Alternative Approach, Simple but Wrong

A comment by “T” suggested using some trig to convert all angles to fit into the ragne -180 to 180°, and computing the resulting fit. This seems like an easier approach, and if you only check the answers to the first three cases, they agree.

However, converting all angles to the arbitrary range -180 to 180° doesn’t check to see which angles should be changed. Dick’s fourth case has wind directiosn {190, 170, 200}, which become {-170, 170, -160}. Oops, now the three directions, which differed by at most 30°, now differ by 340°.

The incorrected directions are shown in the table and chart below.

Four of the five cases work out fine, but the fourth is obviously way off. Of course, without a simple chart, it’s not so obvious. The calculated values are shown as squares at 3000 ft. The black squares match perfectly with Dick’s solution, but the red square for the fourth data set is way off. If this solution had incorporated a check to see which sets of dta should be converted and which should be left alone, it may have made correct calculations for all cases.

There are a number of aspects involved in preparing your data, and I’ll cover some of them here. I will concentrate on best practices for chart source data, but the principles also apply for other purposes. Factors to keep in mind are the layout of the data, both in terms of blank rows and columns in the data and whether the data is oriented by row or by column. Aspects of chart data include knowing which ranges of data are used for X and Y values and for series names, and how to get Excel to use the right data for the right parts of the chart. Finally, should you format your chart data so it looks nice in that monthly report, or should you splurge and use multiple data ranges?

Contiguous Data Layout

The best way to arrange data for Excel charts is in a contiguous rectangular range. Contiguous is a two-dollar word that means don’t skip any rows or columns. For many of its features, if you select a single cell in a range of data, Excel will expand the selection until it reaches empty rows and columns, and use that as its first guess for the range you want used.

This screenshot shows a discontiguous range, B2:E15 with column D and rows 9-10 blank.

The Chart Wizard was started with cell B3 selected. Note that the highlighted range is only B2:C6. Excel 2007 handles ranges exactly the same way.

Sometimes you’re stuck with a discontiguous range. It’s usually better to convert it into a contiguous range, but if it’s for a one-off chart, that seems like too much trouble. You can select a discontiguous range and the chart wizard will accept it, or you can identify a discontiguous range in the data range selection box in the dialog, but there are rules that the range must adhere to.

The discontiguous range must be represented as a rectangular range that is subdivided by entire rows and entire columns, like the yellow range B2:E15 below split apart by the pink-shaded column D and rows 9-10. Each area of the range must be selected as a single section: for example, the area E2:E8 must be selected in one step, not as partial sections E2:E4 and E5:E8.

The Chart Wizard accepts this well-formed discontiguous range.

If the range does not meet the shape requirement above, the Chart Wizard will accept it, but will jump to the Series tab, and perhaps define the series mysteriously. The Data Range tab will show a message indicating that the range is too complex to be displayed.

One test for a well-formed discontiguous range  is whether Excel allows you to select and copy the range. You can select and copy the yellow highlighted range above. When you paste it, it will fill a contiguous range, as in G2:I13 below.

If you try to copy a range which is not well-formed, you get a misleading message that says you can’t do that with multiple selections. Actually you can do it with some multiple selections, just not the one you’ve selected.

Orientation of the Data

By definitions the data range can be laid out in two orientations: By Row and By Column. Some formulas accept 3D references that in effect have a By Sheet dimension, but for charts the data is strictly 2D.

In general it doesn’t really matter how your data is oriented. Out of habit, people probably use series data in columns more frequently. This habit probably results from several factors. First, in Excel 2003 and earlier, there are only 256 columns, which severely limits the number of points in your series. There are over 65,000 rows, which is more than twice the 32,000 points allowed in a series.

When data is extracted from a database, it generally comes in with records points) in rows and fields (series) in columns. This range may be a dataset resulting from a query of a database which returns values from fields Alpha, Beta, and Gamma for a certain set of other conditions.

When you apply an autofilter to the data,

or in Excel 2003 convert the data range to a List (called a Table in Excel 2007),

you gain the ability to filter out rows of data which meet various criteria in each column, or sort the data by columns.

Chart Source Data

In general, Excel tries to use data in the selected range for series names and category labels (X values). When using series in columns, this generally means the first row is reserved for series names and the first column for X values.

Excel uses the text labels in the first column for category labels in this line chart.

Excel uses the dates in the first column as X values in this line chart.

Excel doesn’t see anything special about the years in the first column below. They are numerical values, they are not text, nor are they formatted as dates. Therefore Excel considers them just another set of numbers and plots them as Y values. Of course, they lie far above the other values, which are clustered along the X axis. The X values are simply the counting numbers 1, 2, 3, etc.

That’s funny, in an XY chart with the same data, Excel uses the years as X values.

Excel’s behavior is not so hard to understand. In an XY chart, the first column is (almost) always used for X values. In a Line chart, if the first column is notably different from the other columns, it is also used for X values. This difference may be that the column contains text labels, or that it contains values formatted as dates. If the values are years, then they are not sufficiently different from the Y values for Excel to know that they should be treated differently.

Another way to show Excel that the first column and row should be treated differently is to leave the top left cell blank. Now the first column is different, because it has no header. The first row is also different for the same reason.

The line chart below was made with years in the first column, but the blank cell told Excel to use the years as X values.

If you select the chart area or plot area, Excel highlights the source data range. The Y values are highlighted with a blue border, the X values with a purple border, and the series names with a green border. I’ve highlighted the text in the cells to match the highlighting rectangles.

This blank top left range can be taken further. For example, in the range below, the top two row by two column range is blank. Excel uses the first two columns (under the blank cells) for a two-level set of category axis labels, and the first two rows (alongside the blank cells) for a set of two-cell series names. Excel highlights these ranges using the same color scheme.

Multipurpose Data vs. Multiple Copies of the Data

We’ve seen how the source data for a chart might be best laid out like this:

while a financial report may have a table with this formatting:

The source data for a pivot table may look similar to the chart’s data, with its blank cells filled in:

or it may use actual dates in one column rather than years and months in two:

When you have several uses for your data, and each requires a slightly or greatly different layout, what do you do? It will take hours to try to create a decent chart from the financial report’s fancy layout, with its centered labels and blank rows and columns. You’ll never make a proper pivot table from the chart source data or from the financial table.

Back in the day, computer storage came at a premium, and you conserved every byte you could. Since data was stored on magnetic tapes and punch cards, it wasn’t only memory but also reading and writing of the data that was limiting. Nowadays we have terabyte drives for under $1 per GB, which can transfer data at rates of several GB per minute. It is illogical to try to improve storage efficiency by conserving a few bytes here and there. This paradigm has left the building.

The  solution is easy. Compile a master table of data, then make several copies of the data that link back to this table. The links keep the copies in synch with the original, while each copy of the data can be rearranged to suit a specific purpose.

Create a worksheet with the source data for a pivot table. Place the source data for a chart on another sheet, and if you have multiple charts that require unique layouts, go nuts! Create as many data sheets as you have charts. Link your data into one table that’s optimized for on-screen viewing, and another table that’s formatted just perfectly for that  report you print out for the boss every week. Once created and linked, these tables practically maintain themselves.

The NIST Engineering Statistics Handbook has a good description of the LOESS technique, including a worked example. A commenter named Nick used the NIST chapter as a starting point for his implementation of a LOESS function for Excel, and he posted in in a comment on JunkCharts. Nick’s approach was to create a UDF in VBA. The UDF accepts as inputs the X and Y data ranges, the number of points to use in the moving regression, and the X value for which to calculate Y. Nick’s UDF used Dectionary objects to hold intermediate values, and it outputs the Y value for the input X value.

I’ve expanded on Nick’s starting point, and produced the function presented later in this article. I’ve discarded the Dictionary objects in favor of VB arrays. It accepts the input X and Y data and the output X values either as ranges or as vertical arrays, and it outputs the calculated LOESS Y values as a vertical array. This means it can be called from within other procedures using arrays, or as a UDF from a worksheet, as an array formula. The original data must be sorted by X in either ascending or descending order (Nick’s Dictionaries do not require sorted input data, and I have an idea to remove the requirement from my function).

To use the function as a UDF, select the multicell output Y range, and enter this formula:

=loess(C2:C22,D2:D22,F2:F21,7)
 

where C2:C22 and D2:D22 are the input X and Y ranges, F2:F21 is the output X range, and 7 is the number of points in the moving regression (see screenshot below).

Enter this as an array formula by holding Ctrl and Shift while pressing Enter, and the selection fills with the calculated Y values. Note the curly braces around the formula in the formula bar, which indicates the formula is an array formula.

This chart shows the original NIST data points and the smoothed LOESS curve.

In Local regression, Wikipedia has a decent description of LOESS, with some pros and cons of this approach compared to other smoothing methods.

Example Uses of LOESS

This chart compares LOESS smoothing of website statistics with a simple 7-day moving average. The LOESS captures the major trends in the data, but is less severely affected by week to week fluctuations such as those occuring around Thanksgiving and over the year-end and New Year holidays.

Using LOESS to analyze the body mass indexes (BMI) of Playboy playmates gives more insights than linear regression over the whole data set or over portions of the data. See the discussion in Wired Relates Playboy Playmate BMI and Average BMI, 1954-2008 on the FlowingData blog.

The LOESS Function

Public Function LOESS(X As Variant, Y As Variant, xDomain As Variant, nPts As Long) As Double()
  Dim i As Long
  Dim iMin As Long
  Dim iMax As Long
  Dim iPoint As Long
  Dim iMx As Long
  Dim mx As Variant
  Dim maxDist As Double
  Dim SumWts As Double, SumWtX As Double, SumWtX2 As Double, SumWtY As Double, SumWtXY As Double
  Dim Denom As Double, WLRSlope As Double, WLRIntercept As Double
  Dim xNow As Double
  Dim distance() As Double
  Dim weight() As Double
  Dim yLoess() As Double

  If TypeName(X) = "Range" Then
    X = X.Value
  End If

  If TypeName(Y) = "Range" Then
    Y = Y.Value
  End If

  If TypeName(xDomain) = "Range" Then
    xDomain = xDomain.Value
  End If

  ReDim yLoess(LBound(xDomain, 1) To UBound(xDomain, 1), 1 To 1)

  For iPoint = LBound(xDomain, 1) To UBound(xDomain, 1)

    iMin = LBound(X, 1)
    iMax = UBound(X, 1)

    xNow = xDomain(iPoint, 1)

    ReDim distance(iMin To iMax)
    ReDim weight(iMin To iMax)

    For i = iMin To iMax
      ' populate x, y, distance
      distance(i) = Abs(X(i, 1) - xNow)
    Next

    Do
      ' find the nPts points closest to xNow
      If iMax + 1 - iMin <= nPts Then Exit Do
      If distance(iMin) > distance(iMax) Then
        ' remove first point
        iMin = iMin + 1
      ElseIf distance(iMin) < distance(iMax) Then
        ' remove last point
        iMax = iMax - 1
      Else
        ' remove both points?
        iMin = iMin + 1
        iMax = iMax - 1
      End If
    Loop

    ' Find max distance
    maxDist = -1
    For i = iMin To iMax
      If distance(i) > maxDist Then maxDist = distance(i)
    Next

    ' calculate weights using scaled distances
    For i = iMin To iMax
      weight(i) = (1 - (distance(i) / maxDist) ^ 3) ^ 3
    Next

    ' do the sums of squares
    SumWts = 0
    SumWtX = 0
    SumWtX2 = 0
    SumWtY = 0
    SumWtXY = 0
    For i = iMin To iMax
      SumWts = SumWts + weight(i)
      SumWtX = SumWtX + X(i, 1) * weight(i)
      SumWtX2 = SumWtX2 + (X(i, 1) ^ 2) * weight(i)
      SumWtY = SumWtY + Y(i, 1) * weight(i)
      SumWtXY = SumWtXY + X(i, 1) * Y(i, 1) * weight(i)
    Next
    Denom = SumWts * SumWtX2 - SumWtX ^ 2

    ' calculate the regression coefficients, and finally the loess value
    WLRSlope = (SumWts * SumWtXY - SumWtX * SumWtY) / Denom
    WLRIntercept = (SumWtX2 * SumWtY - SumWtX * SumWtXY) / Denom
    yLoess(iPoint, 1) = WLRSlope * xNow + WLRIntercept

  Next

  LOESS = yLoess

End Function
 

The LOESS Utility

The flexibility of this LOESS function will make it easy to encapsulate into an add-in that uses a dialog to facilitate user selection of data and parameters. A working version uses the following dialog:

Update 24 June 2009

A LOESS utility for Excel has finally been made ready for public consumption. It is described in LOESS Utility for Excel, where there is a link to download the utility. It’s still in preliminary form, but runs pretty much trouble free. Users are encouraged to comment on it to drive further development.

Update 8 October 2009

The LOESS utility for Excel has been updated, and the interface made more flexible. It is described in LOESS Utility – Awesome Update, where there is a link to download the new utility.

References

Cleveland, W.S. (1979), “Robust Locally Weighted Regression and Smoothing Scatterplots,” Journal of the American Statistical Association, Vol. 74, pp. 829-836.

Cleveland, W.S. and Devlin, S.J. (1988), “Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting,” Journal of the American Statistical Association, Vol. 83, pp. 596-610.

NIST Engineering Statistics Handbook, 4.1.4.4. LOESS (aka LOWESS)

NIST Engineering Statistics Handbook, Example of LOESS Computations

Wikipedia, Local regression

Scenario 1

You’ve made a gorgeous chart of the data in Sheet1. You copied the chart from Sheet1 to Sheet2 so you could plot Sheet2’s data in the same splendor. And you hit a snag: the chart on Sheet2 refers back to Sheet1’s data. There are two ways to correct this:

• Create a copy of Sheet1 including the chart, so the chart on the copied sheet refers to the data on the copied sheet. Then copy Sheet2’s data and paste it over the copied sheet’s data.
• Edit the series formulas of the copied chart in Sheet2, changing all instances of one sheet name to the other. This becomes tedious if there are multiple series in the copied chart, or if you’ve copied multiple charts.

Scenario 2

You’ve charted data in rows 1 to 100 of your worksheet. Then you’ve updated the data so it reaches down to row 150. Your wonderful chart only shows data down to row 100. There are three ways to correct this:

• Before updating the data, convert the data range to a List (Excel 2003) or a Table (Excel 2007). Then update the data. The list/table will expand to include all of the data, and all formulas that refer to all rows in the list/table will update accordingly. This includes the chart’s SERIES formula.
• Create dynamic named ranges for the X and Y data ranges in the chart, and apply them to the chart series. This is described in Dynamic Charts in this blog and in a number of Dynamic and Interactive Chart examples described in this web site. However, this is an involved procedure that you never have time for.
• Edit the series formulas of the chart, changing all instances of one row number to another. This becomes tedious if there are multiple series  or multiple charts to correct.

Scenario 3

Your charts show the data for product Alpha beautifully, but you want to show the data for product Beta instead. The data is listed in another column. You can:

• Select the series, then drag the highlighted rectangles on the worksheet to reflect the new data range. This becomes a laborious process for multiple charts and series.
• Edit the series formulas of the chart, changing all instances of one column to another. This becomes tedious if there are multiple series  or multiple charts to correct.

The Solution

These suggested means for editing your charts all leave out one approach. In Change Series Formulas I showed how to programmatically change the series formulas in your charts, and I provided a utility that does the task for you. I’ve just recently updated the utility to account for glitches in Excel, and to streamline using the utility.

The Change Series Formula Utility

The new utility is located in PTS_ChangeSeriesFormula.zip. You can install it following the protocol in Installing an Excel Add-In.

Upon installation, the utility creates a toolbar (In Excel 2007, the toolbar buttons are buried on the Add-Ins tab of the ribbon).

Not too fancy or complicated. Suppose I have the following scenario, a chart showing data for Alpha in column B, and I want to show Beta from column C instead.

If it’s only one series in one chart, I can drag the colored highlight rectangles in the sheet, or edit the series formula in the formula bar. But if there’s more, why waste time? With the chart selected (I like to select the series, so the formula bar is visible and I don’t have to remember what to change), click on the Change Series Formula button on the toolbar. Enter the old and new text in the dialog. In the old text entry box, I like to explicitly use punctuation like the dollar signs, to make sure I’m changing a column designation and not some other text in the formula. In the new text box, it doesn’t matter, because Excel capitalizes and adds dollar signs automatically.

Click the Change Active Chart Only button (the other button is inactive because there are no other charts on this worksheet).

Magically, the series formula, the colored highlights, and the chart itself now reference the new data.

Here is how I would extend a chart’s data from row 6 to row 12. In this case there are multiple charts on the worksheet, and you have a choice of changing the active chart or changing all charts on the worksheet.

This dialog shows how to change references from Sheet1 to Sheet2. In addition, the buttons reflect that multiple charts have been selected using Shift+Click. The options are to change all selected charts or all charts on the worksheet.

It’s a pretty smart utility, and it has saved me tons of time. Download it, install it, and use it. Give me feedback, make suggestions, and report errors in a comment below.

In the earlier two posts, I found that the drainage time was related to hole diameter through a power law relationship with an exponent of -2, and to height with an exponent of 1/2.

Often when modeling a physical process you have theoretical relationships that describe the process, and you just need to find the appropriate factors in the formula. Then you can manually plug and chug to find these factors, or you can use an analytical tool to find them.

In Goal Seek – Optimization Approach to a Simple Physics Problem I showed how to use the built-in Goal Seek optimization tool to determine the coefficient to the power law function involving initial water height and hole diameter. Excel has another tool, an optional add-in called SOLVER, which is a more powerful optimization tool.

I have set up my data as shown below. The initial data is in C9:F12. In C3:F6 I have calculated values for a formula of the form

t = K * ha * db
 

where t is time to drain the bucket, h is the water height, d is hole diameter, K is the unknown coefficient, a is the exponent -2, and b is the exponent 0.5. I put a provisional factor K in cell A1, based on the preliminary regression work. Cell C3 has this formula, which is copied into the whole range C3:F6:

=$A$1*C$2^0.5/$B3^2
 

Finally, C15:F18 tabulates the squared differences between the calculations and the actual data. The formula in C15 is simply

=(C3-C9)^2
 

Cell A16 contains the sum of all of these differences (sum of squares). This is more routinely used than the simple sum of differences used in the Goal Seek approach, but the final answers are close.

I used SOLVER to minimize the sum of the squared differences in A16 by changing the factor in cell A1. Unlike Goal Seek, SOLVER actually does find a minimum.

From the Tools menu, choose SOLVER (in Excel 2007, on the Data ribbon tab, click on Solver in the Analysis group). If the command for SOLVER is unavailable, you will have to install it. See Installing an Excel Add-In for instructions; the SOLVER add-in is either present in the list on the Add-Ins dialog, or you can click the Browse button and look in the SOLVER folder.

The following dialog allows you to indicate which cell to change in order to set another cell to a particular value. We are not using the powerful capabilities of SOLVER to include constraints in the model.

Click OK, and SOLVER does its magic, and shows a dialog which either shows a solution or explains why no solution was found. SOLVER did find a solution to this problem.

Here is the complete SOLVER solution.

These charts show the SOLVER solution (markers and dotted lines) in comparison to the original data (solid lines).

In the earlier two posts, I found that the drainage time was related to hole diameter through a power law relationship with an exponent of -2, and to height with an exponent of 1/2.

Often when modeling a physical process you have theoretical or empirical relationships that describe the process, and you just need to find the appropriate factors in the formula to fit your specific set of conditions. Then you can manually plug and chug to find these factors, or you can use an analytical tool to find them.

Excel has a built-in tool called Goal Seek, which I will use to find the coefficient to the power law function involving initial water height and hole diameter.

I have set up my data as shown. The initial data is in C9:F12. In C3:F6 I have calculated values for a formula of the form

t = K * ha * db
 

where t is time to drain the bucket, h is the water height, d is hole diameter, K is the unknown coefficient, a is the exponent 0.5, and b is the exponent -2. I put a provisional factor K in cell A1, based on the preliminary regression work. Cell C3 has this formula, which is copied into the whole range C3:F6:

=$A$1*C$2^0.5/$B3^2
 

Finally, C15:F18 tabulates the differences between the calculations and the actual data. The formula in C15 is simply

=C3-C9
 

Cell A16 contains the sum of all of these differences.

I used Goal Seek to minimize the sum of the differences in A16 by changing the factor in cell A1. Goal Seek doesn’t actually find a minimum, but if you enter zero, it will get as close as it can.

From the Tools menu, choose Goal Seek (in Excel 2007, on the Data ribbon tab, click on What-If Analysis in the Data Tools group, and select Goal Seek). The following dialog allows you to indicate which cell to change in order to set another cell to a particular value.

Click OK, and Goal Seek does its magic, and shows a dialog which either shows a solution or explains why no solution was found. Goal Seek did find a solution to this problem.

Here is the complete Goal Seek solution.

These charts show the Goal Seek solution (markers and dotted lines) in comparison to the original data (solid lines).

The first step is to plot the data using XY charts. The charts below show time to drain vs height for various hole sizes and time to drain vs hole size for various diameters.

The regression approach can be followed in Excel using the LINEST worksheet function. The help topic for LINEST is pretty good, so I would direct you there first if you have questions.

LINEST Fit 1

I set up the following table based on the initial data. Knowing that we have height^0.5 and diameter^-2 dependencies on time to drain the bucket, I calculated the transformations in columns C and D. The LINEST regression output is in C22:E26.

To compute the regression output, select a range 5 rows high and N+1 columns wide, where N is the number of X variables used. We have two X variables, 1/dia² and sqrt(ht), so we need a range three columns wide. With the range selected, enter the LINEST formula, and use CTRL+SHIFT+ENTER to accept the formula, not just ENTER, because the output is an array. The specific formula used here is:

=LINEST(E2:E17,C2:D17,,TRUE)
 

where E2:E17 contains the Y values, C2:D17 contains two columns of X values, the blank third argument tells Excel to calculate the constant normally, and TRUE tells Excel to provide the extra rows of  statistical parameters, even though I will pretty much ignore them.

Note that when the LINEST formula contains no syntax error and is properly entered using CTRL+SHIFT+ENTER, Excel wraps it in curly braces to indicate that it is an array formula. If you try typing these curly braces yourself, you will only cause an error.

{=LINEST(E2:E17,C2:D17,,TRUE)}
 

E22 contains the constant, D22 contains the coefficient for 1/dia², and C22 contains the coefficient for sqrt(ht). The predicted value in F2 uses this formula:

=E$22+D$22*C2+C$22*D2
 

This formula is filled down to F17. Row 19 shows the prediction from the regression formula for intermediate values of diameter and height. The time of 15.4 is much different than the graphical interpolation of 8.5 found in Graphical Approach to a Simple Physics Problem.

The predicted values don’t match the actual values too closely, but let’s plot them. The following charts use markers and dotted lines for predictions and solid lines for actual values.

Not very good fits at all, and there are some negative values (are we filling the bucket up in these cases?).

The log-log plots below don’t even have the appropriate shapes.

These plots illustrate out problem. The regression calculated a single coefficient (slope) for each of the X variables, and the terms are added together. The consequence of the single slope and added terms is that all curves have the same slope, and the predicted lines pass through the averages of the actual lines. This is the wrong constraint for this problem.

LINEST Fit 2

After the previous fiasco I brushed off myself and my bruised ego, and started again. This time I ignored the powers I’d imposed on diameter and height, and simply took logs of all variables. The LINEST formula is:

=LINEST(F2:F17,D2:E17,,TRUE)
 

I could actually have skipped the logarithmic transformations in columns D:F and used this LINEST formula:

=LINEST(LOG(C2:C17),LOG(A2:B17),,TRUE)
 

Remember to select the formula’s output range, type the formula, and array-enter it by holding CTRL+SHIFT while pressing ENTER. Excel puts curly braces around the formula if it’s syntactically correct.

The predictions in column G use this formula:

=F$22+E$22*D2+D$22*E2
 

These predictions are the logarithms of the times to drain, so they must be converted to their antilogs using:

=10^G2
 

These predictions are pretty close to the actual values. The prediction for the arbitrary starting values in row 19, 8.5, agrees with the graphical interpolation from Graphical Approach to a Simple Physics Problem. Let’s go to the charts, which use markers and dotted lines for predictions and solid lines for actual values.

The fitting coefficients for log(height) and log(diameter) are constant, so we get parallel lines in the log-log charts.

The corresponding straight lines for time vs. swrt(ht) and 1/diam² are no longer parallel, but instead they converge on the origin.

When the regression is done properly, its results agree closely with those from a simple graphical interpolation. In the next post on this topic, we will look at optimization approaches to determining these relationships.