Most commonly you have a worksheet with a bunch of data and a corresponding chart, and you have another sheet of data you want to add a chart to. Copying and pasting the chart onto the new sheet requires you to change links in the chart, usually series by series. This is tedious and error-prone. But the article above describes an easier way:

1. Make a copy of the worksheet with the old data and chart;
2. Copy the new data;
3. Paste the new data over the old data on the copied worksheet.

The chart on the new worksheet updates as soon as the new data is pasted into place. Works in Excel 2003 and earlier, and in Excel 2007 if you’ve installed the latest service packs.

What if the worksheet contains a pivot table and its sister pivot chart? Well, knowing the above protocol, you’d think you could copy the worksheet, change the data, and the chart would be fine. And in Excel 2003 this set of steps works great:

1. Make a copy of the worksheet with the old pivot table and pivot chart;
2. Change the pivot table’s data source to the new range;
3. Refresh the pivot table.

The new pivot chart (on the copied sheet) retains its link to the pivot table on its parent worksheet, so it updates as soon as the pivot table is refreshed.

But in Excel 2007, these steps  don’t work the same way. When you copy the worksheet with the pivot table and chart, not only does the new pivot table link to the same old data, the new chart also links to the old pivot table. You can easily enough change the data source of the new pivot table to the appropriate range; in fact, this is easier to do in Excel 2007 than in earlier versions. But the new chart cannot be linked to the new pivot table. It is permanently linked to the old pivot table.

My colleague Bill Manville wrote to me about this problem, citing an old forum post in which I doubted this could ever be solved. I’m glad to say that Bill has proved me wrong. He sent me a new protocol that makes this work.

1. Make a copy of the worksheet with the old pivot table and pivot chart in a different workbook;
2. Move the copied worksheet back into the original workbook;
3. Change the new chart’s source data to the new pivot table;
4. Change the pivot table’s data source to the new range;
5. Refresh the pivot table.

The difference is that the worksheet is copied into a new workbook (or another existing workbook) rather than within the original workbook. When this happens, the pivot table still links to the original data, but the chart becomes unlinked from the pivot table. In fact, the chart changes back from a pivot chart to a regular chart. It is unlinked from any data range, and the series formulas have been converted to written-out arrays.

Since the chart has been unlinked in the process, it now can be relinked. Thanks, Bill, for saving us lots of time and effort.

Bill tells me that the familiar Excel 2003 behavior has been restored to Excel 2010.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Experimental Design

There are two ways to find the minimum and maximum of a set of values:

• Loop through the values, and keep track of the largest and smallest you’ve encountered
• Use the worksheet functions MIN() and MAX() (i.e., WorksheetFunction.Min and .Max)

There are three ways to obtain the values for each series:

• Use the .Values property of the series, which returns a 1-D array
• Parse the series formula to identify the range, then get the .Value of this range object, which returns a 2-D array
• Parse the series formula to identify the range, and analyze the range itself

This suggests a 2×3 matrix, but I eliminated looping cell-by-cell through the range to find the largest and smallest values, since experience shows that looping through the cells of a range in this way is much slower than using the .Value property of the range to generate an array, and working with this array.

The five-block test matrix becomes:

• Looping Over Series Values
• Worksheet Function on Series Values
• Looping Over Range Values
• Worksheet Function on Range Values
• Worksheet Function on Range Object

I wrote VBA procedures to perform each of these approaches. Then I wrote a master procedure that called each of these procedures 1000 times, 50 times each in 20 blocks, so I’d have the mean time for each block. The overall mean and standard deviation of the block means can be used to compare the different approaches.

I used a simple XY chart with 1000 points. Both X and Y values were taken from the standard normal distribution. The procedures below could be adapted to check the Y values of any chart and the X values of an XY chart or a line/column/bar/area chart with a date scale X axis. Here the X values don’t matter since I was merely sampling Y.

I ran the tests in Excel 2003 SP3 on a reasonably modern laptop, running Windows 7 Ultimate with 4GB of memory on a 2.20 GHz dual-core AMD processor.

VBA Procedures

Looping Over Series Values

Sub Max_Min_Loop_Values()
  Dim Cht As Chart
  Dim Srs As Series
  Dim i As Long
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  A = Srs.Values
  MaxVal = A(LBound(A, 1))
  MinVal = A(LBound(A, 1))
  For i = LBound(A, 1) + 1 To UBound(A, 1)
    If A(i) > MaxVal Then MaxVal = A(i)
    If A(i) < MinVal Then MinVal = A(i)
  Next i
End Sub

Assuming there is one chart on the active worksheet, and one series in the chart, we set the chart and the series simply. We assign the Y values of the series to an array, and initially set the minimum and maximum values to the first value in the array. Then we loop from the second to the last values in the array, and if a given value is larger than the maximum value so far, we set the new maximum to the current value, and likewise for the minimum. I didn’t output the minimum and maximum values, all I wanted was the time to run the core of the procedure.

Worksheet Function on Series Values

Sub Max_Min_Wkfn_Values()
  Dim Cht As Chart
  Dim Srs As Series
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double  

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  A = Srs.Values
  With WorksheetFunction
    MaxVal = .Max(A)
    MinVal = .Min(A)
  End With
End Sub

We set the chart and the series simply as above. We assign the Y values of the series to an array, and use the worksheet functions Min() and Max() to determine the minimum and maximum values.

Looping Over Range Values

Sub Max_Min_Loop_RangeArray()
  Dim Cht As Chart
  Dim Srs As Series
  Dim i As Long
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double
  Dim sSeriesFmla As String 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  sSeriesFmla = Split(Srs.Formula, ",")(2)
  A = Range(sSeriesFmla).Value
  MaxVal = A(LBound(A, 1), 1)
  MinVal = A(LBound(A, 1), 1)
  For i = LBound(A, 1) + 1 To UBound(A, 1)
    If A(i, 1) > MaxVal Then MaxVal = A(i, 1)
    If A(i, 1) < MinVal Then MinVal = A(i, 1)
  Next i
End Sub

We set the chart and the series simply as above. We parse the formula of the series to find the range, use the .Value property of the range to define an array, and initially set the minimum and maximum values to the first value in the array. Then we loop from the second to the last values in the array, and if a given value is larger than the maximum value so far, we set the new maximum to the current value, and likewise for the minimum.

Worksheet Function on Range Values

Sub Max_Min_Wkfn_RangeArray()
  Dim Cht As Chart
  Dim Srs As Series
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double
  Dim sSeriesFmla As String 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  sSeriesFmla = Split(Srs.Formula, ",")(2)
  A = Range(sSeriesFmla).Value
  With WorksheetFunction
    MaxVal = .Max(A)
    MinVal = .Min(A)
  End With
End Sub

We set the chart and the series simply as above. We parse the formula of the series to find the range, use the .Value property of the range to define an array, and use the worksheet functions Min() and Max() to determine the minimum and maximum values.

Worksheet Function on Range Object

Sub Max_Min_Wkfn_Range()
  Dim Cht As Chart
  Dim Srs As Series
  Dim rng As Range
  Dim MinVal As Double
  Dim MaxVal As Double
  Dim sSeriesFmla As String 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  sSeriesFmla = Split(Srs.Formula, ",")(2)
  Set rng = Range(sSeriesFmla)
  With WorksheetFunction
    MaxVal = .Max(rng)
    MinVal = .Min(rng)
  End With
End Sub

We set the chart and the series simply as above. We parse the formula of the series to find the range, and use the Worksheet Functions Min() and Max() to determine the minimum and maximum values of the range object directly.

Timing Procedure

Sub TestTimes()
  Dim Looper As Long
  Dim Blocker As Long
  Dim tStart As Double
  Dim tTotal As Double
  Const LooperMax As Long = 50
  Const BlockerMax As Long = 20  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Loop_Values
    Next
    tTotal = Timer - tStart
    WriteToLog "Loop - Values: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Wkfn_Values
    Next
    tTotal = Timer - tStart
    WriteToLog "Func - Values: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Loop_RangeArray
    Next
    tTotal = Timer - tStart
    WriteToLog "Loop - RngArr: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Wkfn_RangeArray
    Next
    tTotal = Timer - tStart
    WriteToLog "Func - RngArr: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Wkfn_Range
    Next
    tTotal = Timer - tStart
    WriteToLog "Func - Range:  " & tTotal & " sec for " & LooperMax & " iterations"
  Next 

End Sub

This procedure calls each of the Min_Max procedures LooperMax (50) times, records the total time for 50 iterations, and repeats this BlockerMax (20) times for each procedure, before moving on to the next. The times are recorded in a text file using the WriteToLog procedure below.

Logging Procedure

Public Sub WriteToLog(sLogEntry As String)
  ' write information to a log file
  Dim iFile As Integer
  Dim sFileName As String
  Const sLogFile As String = "TimeLogger"   

  sFileName = ThisWorkbook.Path & "\" & sLogFile & Format$(Now, "YYMMDD") & ".txt"
  iFile = FreeFile
  Open sFileName For Append As iFile
  Print #iFile, Now; " "; sLogEntry
  Close iFile
End Sub

This is a very handy procedure I use all the time for logging debug information or for saving settings. It uses good old VB I/O protocols, which are very fast, to populate a text file.

Test Results

The results of these experiments are shown in the following table. The approaches are sorted from slowest to fastest. Processing an array extracted from the source data range takes about 1.2 milliseconds, whether looping or using the worksheet functions. Processing the series values array takes about 0.7 milliseconds by either technique. Measuring the range directly takes less than 0.5 milliseconds.

50-iteration means of the 20 blocks for each approach are plotted in this chart, with means sorted from fastest to slowest to give a pseudo-distribution of procedure times.

This chart is stepped, because there seems to be a discrete quantum of time, that is, a resolution of about 0.078 milliseconds in the measurement of time using VBA’s Timer function on this microprocessor.

What These Results Mean

One school of thought is that looping an array would be slow compared to a single function call, while another is that any call from VBA to Excel (i.e., WorksheetFunction) would incur a performance hit. An unexpected result is that on this machine, it makes essentially no difference whether I use a loop or worksheet functions to find the minimum and maximum values in a series of 1000 points. I guess that’s why we occasionally bother to run these time trials.

We see that using the array of series values is faster than using the array of values from the source data range, by not quite a factor of two. Thinking of the VBA-Excel interface, we have to cross it once to get the series values, or twice to get the series formula and again to get the range values. This makes sense, and I have to admit, I would never have thought of getting the values from the source data range, since they are so easy to extract using series .Values.

The fastest method of all, which was proposed by Daniel Ferry (Excel Hero), is to parse the series formula to find the range, and directly measure the min and max of this range. Apparently the performance hit to cross the VBA-Excel barrier is offset by the much faster performance of worksheet functions operating on Excel objects directly within Excel.

Although the direct range object measurement approach is fastest, I will stick to my series values approach. The main reason is that not all series values come from a single range in a worksheet, or even from a worksheet range at all: some series use multiple area ranges, while others have values hard-coded as a literal array directly in the series formula.

This is a “normal” series formula (based on the first ten points of experimental data from our chart above):

=SERIES(Sheet1!$B$1,Sheet1!$A$2:$A$11,Sheet1!$B$2:$B$11,1)

In this formula, the series name is the argument in green, X values are in purple, Y values are in blue, and plot order is in red. These are the same colors Excel uses to highlight the data ranges of a selected series in the underlying worksheet (illustrated below).

This formula is for a series that uses multiple area ranges for X and Y values, the areas for X and Y enclosed in parentheses:

=SERIES(Sheet1!$B$1,(Sheet1!$A$2:$A$501,Sheet1!$A$502:$A$1001),
(Sheet1!$B$2:$B$501,Sheet1!$B$502:$B$1001),1)

Multiple area formulas are not too uncommon. Less common is a series whose formula has hard-coded arrays as series values (first ten points of experimental data converted to values using F9 key):

=SERIES("Y",{-0.172902780162854;0.0788276881579948;0.752718316150293;-1.15534740638389;
-0.651271029427549;1.61522977181724;0.218894823127643;-1.38264191435207;
0.107605524088065;0.281018951052876},
{-0.640448122430912;-0.355817827507841;-0.318480200209528;-0.0116013833433011;
-1.15185971606638;0.183605522997797;-0.376897171002227;-0.488810695788606;
-2.14375416629929;0.154502474308393},1)

There is a practical limit of around 250 characters (you’d guess 255, but it’s slightly less) each for the X values and Y values of a series formula. In the formula above, the ten X values consume 183 characters, the ten Y values 188. You can squeeze in more values by reducing the resolution of the values (e.g., -0.173 in place of -0.172902780162854), but that’s just nibbling around the margins. A practical limit is around 17 or 18 values before the formula crashes.

The arrays (within {curly braces} above) are delimited by semicolons in this case, because the original data came from a column, but data in rows would be delimited by commas.

These complications in the series formula lead to errors when using simple approaches to parse such formulas. John Walkenbach has developed A Class Module To Manipulate A Chart Series which can deal with such difficult situations.

Since accessing values using the .Values property is so easy, and since this approach is only about 50% slower than the risky approach of parsing the formula and measuring the range directly, that’s the approach I will stick with.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

People are familiar with a standard linear regression (LR) method that makes assumptions about the X and Y variables in the analysis. Excel’s built-in regression methods make these same assumptions.

The X variable, also called the independent variable, is assumed to be known precisely, and all error in the regression is assumed to be in the measurement of the Y variable, the dependent variable.

This assumption is reasonable enough. But there are cases where errors in the values of X cannot be discounted. For example, X and Y may be comparisons of the output of two different machines for given inputs. When the outputs are compared, both machines are subject to errors in measurement, so the usual assumptions are invalid.

Deming Regression Example

Suppose Machines A and B had the following output. At each nominal output value, the results of two runs of each machine are tabulated. There is some variability in measurements of each machine’s output. The averages of each machine at each nominal output level are shown next to the individual measurements.

We may be tempted to plot each machine’s outputs as Y values against the nominal X values.

We may even compute the lines of best fit for the two machines. We can see that they are close, but not exactly the same. The problem with this analysis is that we are not directly regressing one machine against the other.

In the Deming Regression, outputs of the two machines are plotted against each other. Pairs of actual outputs are measured for each nominal output, and the differences between the two measurements from each machine will be used to estimate the variation of each machine’s outputs. The averages of each pair form the basis of the regression, corrected for the machine variation.

The table below shows three sets of calculations. The first row shows a standard linear regression of the measured data. The second row shows a standard linear regression performed on the averages of each pair of values. The third shows the Deming regression analysis. Details of the Deming regression calculations are not given here, but a motivated reader can check the links in the references at the end of this article.

The data are not too different, and in fact the correlation in the LR of the average values and of the Deming regression are identical. In the plot below, lines have been added to show the three sets of results in the table above, as well as the 45 degree line, where Machine B = Machine A. The lines are practically on top of each other.

In the example above, there is little difference in the output of the various LR models. However, if Machines A and B exhibit greater variation in their output, there can be noticeable differences.

Deming Regression Utility

I’ve created a small utility that facilitates Deming Regression analysis in Excel. Read about it and follow the download link in Deming Regression Utility.

References

Chapter 17:  Creating  Custom  Functions, Excel for Chemists: A Comprehensive Guide, 2nd Edition. by E. Joseph Billo, Copyright 2001 by John Wiley & Sons, Inc.

Incorrect Least-Squares Regression Coefficients in Method-Comparison Analysis by P. Joanne Cornbleet and Nathan Gochman, Clin. Chem. 25/3, 432-438 (1979).

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Plotting Without Empty Cells

Here are some typical Excel charts, to remind us what they look like with fully populated data ranges. I’ve placed data labels on the charts to help show their behavior when empty cells (and other non-numeric fillers) are included in the source data.

Plotting With Empty Cells

The default behavior for most charts is to treat an empty cell by leaving a gap on the chart. Here are XY and Line charts that show how this looks. Note the lack of data labels where the gaps occur.

You can instead have Excel treat blanks as zeros (which in general is deceiving, since empty cells mean the absence of a value, not a value of zero). Note the data labels indicating a value of zero.

You can also tell Excel to interpolate over the missing data point. No data point means no data label.

Purists may say that the interpolation option may cause readers to think there is data along the line that spans the gap. Maybe so, but if the line or XY chart plots actual data points with markers, this is less of a risk.

Column charts offer only the gap and zero options for empty cells. Interpolating makes no sense in a column chart, which has discrete bars for each actual data value. The chart with a gap has no data label at the gap, the chart with zeros has a zero data label.

If we change the vertical axis so that the horizontal axis crosses below zero, we see that the first chart has no data point (bar) corresponding to the gap, but the second has a point (bar) with a value of zero.

Area charts seem to offer both the zero and interpolation options, but in both cases, the chart plunges to zero without a corresponding data label. If we mess with the vertical axis, the chart still plunges exactly to zero in both cases.

A more suitable appearance for an area chart would be one that leaves a real gap, with vertical edges, as below. To get this I had to make a two-axis chart, with a hidden series on the primary axis to provide the A-B-C category axis labels, and an area chart on the secondary axis, with two points at the second category (Y=4 and zero) and two at the fourth category (Y=zero and 8). A date scale on the secondary category axis aligns these points above each other to produce the vertical-sided gap. This only took me three minutes to construct, but an average user might never quite get it.

The protocol to produce an area chart with a gap that has vertical sides is given in Area Chart With Gap » Peltier Tech Blog

Pie charts don’t seem to offer any options for dealing with blanks, but the grayed-out default is the gap option. In the chart at the right, there is no point (wedge) and no corresponding data label where the empty cell would be plotted.

Apparently all charts except the Area chart provide a non-plotted point for an empty cell in the data range, though we can spoil that by selecting the option to treat a gap as a zero value.

Simulating Empty Cells

The main confusion comes along when someone uses formulas to fill the source data range.

The problem arises because  a formula that links to an empty cell

=A1

doesn’t display a blank, it displays the value zero.

Unfortunately Excel has no formulaic way to simulate a blank in a cell. There is no formulaic way, then, to get an actual gap in a chart. A function like BLANK() or NULL() would be nice, and we’ll keep asking. But we won’t hold our breath. And if we really need it, we can hack something in VBA.

Andy Pope shares a workaround in Broken Lines for formula linked data, but it requires one or more dummy series with a line color matching the chart background, which obscure the existing line where the gaps should go.

What people try to do to fake a blank is return “” in the formula.

=IF(A1=”",”",A1)

The result looks to you and me like a blank, but to Excel, “” is a text string, albeit a short one, and it is therefore assigned a value of zero. These charts use “” in a logical yet misguided attempt to simulate an empty cell. All charts plunge to zero, and all display a data label showing the value zero.

People have learned that the #N/A error in a worksheet cell is often not plotted in a chart. In a formula, the function NA() returns this error.

=IF(A1=”",NA(),A1)

What #N/A does is prevent the rendering of a marker in XY and line charts: notice the lack of markers and data labels. It’s not a gap, but it’s pretty much the next best thing.

This doesn’t work as well in other chart types. The column and area charts place a data label showing #N/A at zero. For these chart types it would be better to use “” or even zero, and apply a custom number format that suppresses the display of zeros (a format like “0;-0;;”).

A pie chart will behave like a column or area chart: there is no visible point (wedge) but there is a data label showing zero or #N/A where the wedge would be located.

Summary

The least deceptive way to display an empty cell in a chart, is to treat it as a blank. In line or XY charts, it is generally acceptable to interpolate across the blank with a line, if there are markers plotted for every actual data value. In the vast majority of cases, it is not appropriate to treat empty cells as zeros.

To simulate an empty cell in a line or XY chart, use NA() in your formula to produce the #N/A error in the cell. This produces an ugly #N/A error in the cell, but this error suppresses plotting of a point, and the line spans the position corresponding to the error’s slot in the data range. It’s not as good as a gap, but it’s perhaps the next best thing.

In an area or pie chart, the best approach is to use the null string “”, in conjunction with a number format that suppresses the display of zero values.

For an area chart you have to stand on your head to create a complicated chart that simulates a vertical edged gap and provides appropriate category axis labels.

Alternatively you could rely on VBA to clear cells which need to be cleared, but this routine would have to run every time the data range is changed, and it would have to reintroduce formulas into previously cleared cells as well as clear cells where blanks are needed. Or it could do all of the calculations in VBA, eliminating the worksheet formulas.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Excel Pivot Tables are a very powerful feature, and the AutoFilter is also very useful. The row fields of the Pivot Table can be manipulated like an AutoFilter, but the data fields cannot be sorted or filtered. And if you try to apply an AutoFilter to a pivot table, you find the menu command disabled.

But I learned a trick this week at the Excel Dashboard and Visualization Bootcamp. If you select a cell which borders on the pivot table, but is not part of the pivot table,…

…the AutoFilter menu items are now enabled.

Select AutoFilter, and the Pivot Table has been AutoFilter enabled.

These conferences are great.

AutoFilter by Selection

Access has a feature that lets you quickly apply a filter based on the selected item. Wouldn’t it be great to have this in Excel?

Well, this feature exists, and it’s built in. But it’s not present on the default menus or ribbons. You have to dig it out and install it yourself.

Excel 2003 and earlier

On Excel’s Tools menu, select Customize, click on the Commands tab, select the Data item in the Caegories list, click on the AutoFilter item, and drag it to a convenient place on a toolbar or menu.

I’ve put this just under the “regular” AutoFilter item on my Data menu/Filter submenu.

Excel 2007

Click on the down arrow to the right of the Quick Access Toolbar, and select More Commands from the popup menu.

In the oversized dialog that appears, select Commands not in the Ribbon from the right hand dropdown, select AutoFilter, click Add, and press OK.

The AutoFilter icon appears on the QAT.

What the “New” AutoFilter does

This Autofilter command shares a name with the “other” AutoFilter command, but it has a unique icon, and its functionality is enhanced. The command creates an AutoFilter using the range that includes the active cell, then it filters this range based on the active cell.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

One of the answers to the generic question “Why do we need charts?” is to gain insight into a problem before we try to solve it.

Dick showed a couple of charts, lame little pie charts actually, which hand-waved a clever little trick he used to account for directions on either side of 360°. I’ll show how using more carefully crafted charts could have helped lead to the same result more quickly, by showing exactly what the relationships are. I will also show how such charts can easily show when a simplified approach is too simple, and leads to a very wrong answer.

When charted, the original data looks like the following.

By inspection, we can see that Directions 4 and 5 are pretty nearly straight lines as drawn, while the other three have points on either side of 360°. To turn these into linear fits, we need to add or subtract 360° so that the lines can cross 360° (or 0°).

Here is the data, corrected by adding 360° as appropriate. The calculated wind directions at 3000 ft are shown in blue.

Which leads to this chart. All five cases now follow nearly straight lines.

Dick’s Approach

The trick is knowing which data needs to be “enhanced”.Obviously you need to correct the ones which have values concentrated at the ends of the 0 to 360° range.

Dick’s approach was to examine the three directions in each case, and if the difference between the min and the max exceeded 180°, he added 180° and moduloed by 360°. This took data which was arranged around 360° and rearranged it around 180°. To these cases he then added 180° to the calculated result, and again moduloed by 360°.

Dick’s magic formula is

=IF(MAX(C4:C6)-MIN(C4:C6)>180,
    MOD(FORECAST($B7,MOD(C4:C6+180,360),$B4:$B6)+180,360),
    FORECAST($B7,C4:C6,$B4:$B6))
 

which is array-entered (by holding Ctrl+Shift while clicking Enter), so in the formula bar it is surrounded by curly brackets. B4:B6 are the altitudes 2000, 5000, and 10000 in the first column, B7 is the target altitude 3000, C4:C6 are the wind directions under the label Direction1, and this formula is entered in C7. As written, this formula can be copied and pasted into D7:G7 to provide all five calculations.

Instead of a difference of 180°, Dick could have used another cutoff value to decide which data needs correction. Alternatively, a check of the correlation or residuals of the three points could indicate whether the points are nearly aligned or far from linear. These latter approaches are more computationally intensive.

Dick’s calculated results for 3000 ft are shown by the squares in the chart above. They are all pretty close to the line segments connecting the 2000 ft and 5000 ft data.

Alternative Approach, Simple but Wrong

A comment by “T” suggested using some trig to convert all angles to fit into the ragne -180 to 180°, and computing the resulting fit. This seems like an easier approach, and if you only check the answers to the first three cases, they agree.

However, converting all angles to the arbitrary range -180 to 180° doesn’t check to see which angles should be changed. Dick’s fourth case has wind directiosn {190, 170, 200}, which become {-170, 170, -160}. Oops, now the three directions, which differed by at most 30°, now differ by 340°.

The incorrected directions are shown in the table and chart below.

Four of the five cases work out fine, but the fourth is obviously way off. Of course, without a simple chart, it’s not so obvious. The calculated values are shown as squares at 3000 ft. The black squares match perfectly with Dick’s solution, but the red square for the fourth data set is way off. If this solution had incorporated a check to see which sets of dta should be converted and which should be left alone, it may have made correct calculations for all cases.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

There are a number of aspects involved in preparing your data, and I’ll cover some of them here. I will concentrate on best practices for chart source data, but the principles also apply for other purposes. Factors to keep in mind are the layout of the data, both in terms of blank rows and columns in the data and whether the data is oriented by row or by column. Aspects of chart data include knowing which ranges of data are used for X and Y values and for series names, and how to get Excel to use the right data for the right parts of the chart. Finally, should you format your chart data so it looks nice in that monthly report, or should you splurge and use multiple data ranges?

Contiguous Data Layout

The best way to arrange data for Excel charts is in a contiguous rectangular range. Contiguous is a two-dollar word that means don’t skip any rows or columns. For many of its features, if you select a single cell in a range of data, Excel will expand the selection until it reaches empty rows and columns, and use that as its first guess for the range you want used.

This screenshot shows a discontiguous range, B2:E15 with column D and rows 9-10 blank.

The Chart Wizard was started with cell B3 selected. Note that the highlighted range is only B2:C6. Excel 2007 handles ranges exactly the same way.

Sometimes you’re stuck with a discontiguous range. It’s usually better to convert it into a contiguous range, but if it’s for a one-off chart, that seems like too much trouble. You can select a discontiguous range and the chart wizard will accept it, or you can identify a discontiguous range in the data range selection box in the dialog, but there are rules that the range must adhere to.

The discontiguous range must be represented as a rectangular range that is subdivided by entire rows and entire columns, like the yellow range B2:E15 below split apart by the pink-shaded column D and rows 9-10. Each area of the range must be selected as a single section: for example, the area E2:E8 must be selected in one step, not as partial sections E2:E4 and E5:E8.

The Chart Wizard accepts this well-formed discontiguous range.

If the range does not meet the shape requirement above, the Chart Wizard will accept it, but will jump to the Series tab, and perhaps define the series mysteriously. The Data Range tab will show a message indicating that the range is too complex to be displayed.

One test for a well-formed discontiguous range  is whether Excel allows you to select and copy the range. You can select and copy the yellow highlighted range above. When you paste it, it will fill a contiguous range, as in G2:I13 below.

If you try to copy a range which is not well-formed, you get a misleading message that says you can’t do that with multiple selections. Actually you can do it with some multiple selections, just not the one you’ve selected.

Orientation of the Data

By definitions the data range can be laid out in two orientations: By Row and By Column. Some formulas accept 3D references that in effect have a By Sheet dimension, but for charts the data is strictly 2D.

In general it doesn’t really matter how your data is oriented. Out of habit, people probably use series data in columns more frequently. This habit probably results from several factors. First, in Excel 2003 and earlier, there are only 256 columns, which severely limits the number of points in your series. There are over 65,000 rows, which is more than twice the 32,000 points allowed in a series.

When data is extracted from a database, it generally comes in with records points) in rows and fields (series) in columns. This range may be a dataset resulting from a query of a database which returns values from fields Alpha, Beta, and Gamma for a certain set of other conditions.

When you apply an autofilter to the data,

or in Excel 2003 convert the data range to a List (called a Table in Excel 2007),

you gain the ability to filter out rows of data which meet various criteria in each column, or sort the data by columns.

Chart Source Data

In general, Excel tries to use data in the selected range for series names and category labels (X values). When using series in columns, this generally means the first row is reserved for series names and the first column for X values.

Excel uses the text labels in the first column for category labels in this line chart.

Excel uses the dates in the first column as X values in this line chart.

Excel doesn’t see anything special about the years in the first column below. They are numerical values, they are not text, nor are they formatted as dates. Therefore Excel considers them just another set of numbers and plots them as Y values. Of course, they lie far above the other values, which are clustered along the X axis. The X values are simply the counting numbers 1, 2, 3, etc.

That’s funny, in an XY chart with the same data, Excel uses the years as X values.

Excel’s behavior is not so hard to understand. In an XY chart, the first column is (almost) always used for X values. In a Line chart, if the first column is notably different from the other columns, it is also used for X values. This difference may be that the column contains text labels, or that it contains values formatted as dates. If the values are years, then they are not sufficiently different from the Y values for Excel to know that they should be treated differently.

Another way to show Excel that the first column and row should be treated differently is to leave the top left cell blank. Now the first column is different, because it has no header. The first row is also different for the same reason.

The line chart below was made with years in the first column, but the blank cell told Excel to use the years as X values.

If you select the chart area or plot area, Excel highlights the source data range. The Y values are highlighted with a blue border, the X values with a purple border, and the series names with a green border. I’ve highlighted the text in the cells to match the highlighting rectangles.

This blank top left range can be taken further. For example, in the range below, the top two row by two column range is blank. Excel uses the first two columns (under the blank cells) for a two-level set of category axis labels, and the first two rows (alongside the blank cells) for a set of two-cell series names. Excel highlights these ranges using the same color scheme.

Multipurpose Data vs. Multiple Copies of the Data

We’ve seen how the source data for a chart might be best laid out like this:

while a financial report may have a table with this formatting:

The source data for a pivot table may look similar to the chart’s data, with its blank cells filled in:

or it may use actual dates in one column rather than years and months in two:

When you have several uses for your data, and each requires a slightly or greatly different layout, what do you do? It will take hours to try to create a decent chart from the financial report’s fancy layout, with its centered labels and blank rows and columns. You’ll never make a proper pivot table from the chart source data or from the financial table.

Back in the day, computer storage came at a premium, and you conserved every byte you could. Since data was stored on magnetic tapes and punch cards, it wasn’t only memory but also reading and writing of the data that was limiting. Nowadays we have terabyte drives for under $1 per GB, which can transfer data at rates of several GB per minute. It is illogical to try to improve storage efficiency by conserving a few bytes here and there. This paradigm has left the building.

The  solution is easy. Compile a master table of data, then make several copies of the data that link back to this table. The links keep the copies in synch with the original, while each copy of the data can be rearranged to suit a specific purpose.

Create a worksheet with the source data for a pivot table. Place the source data for a chart on another sheet, and if you have multiple charts that require unique layouts, go nuts! Create as many data sheets as you have charts. Link your data into one table that’s optimized for on-screen viewing, and another table that’s formatted just perfectly for that  report you print out for the boss every week. Once created and linked, these tables practically maintain themselves.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

The NIST Engineering Statistics Handbook has a good description of the LOESS technique, including a worked example. A commenter named Nick used the NIST chapter as a starting point for his implementation of a LOESS function for Excel, and he posted in in a comment on JunkCharts. Nick’s approach was to create a UDF in VBA. The UDF accepts as inputs the X and Y data ranges, the number of points to use in the moving regression, and the X value for which to calculate Y. Nick’s UDF used Dectionary objects to hold intermediate values, and it outputs the Y value for the input X value.

I’ve expanded on Nick’s starting point, and produced the function presented later in this article. I’ve discarded the Dictionary objects in favor of VB arrays. It accepts the input X and Y data and the output X values either as ranges or as vertical arrays, and it outputs the calculated LOESS Y values as a vertical array. This means it can be called from within other procedures using arrays, or as a UDF from a worksheet, as an array formula. The original data must be sorted by X in either ascending or descending order (Nick’s Dictionaries do not require sorted input data, and I have an idea to remove the requirement from my function).

To use the function as a UDF, select the multicell output Y range, and enter this formula:

=loess(C2:C22,D2:D22,F2:F21,7)
 

where C2:C22 and D2:D22 are the input X and Y ranges, F2:F21 is the output X range, and 7 is the number of points in the moving regression (see screenshot below).

Enter this as an array formula by holding Ctrl and Shift while pressing Enter, and the selection fills with the calculated Y values. Note the curly braces around the formula in the formula bar, which indicates the formula is an array formula.

This chart shows the original NIST data points and the smoothed LOESS curve.

In Local regression, Wikipedia has a decent description of LOESS, with some pros and cons of this approach compared to other smoothing methods.

Example Uses of LOESS

This chart compares LOESS smoothing of website statistics with a simple 7-day moving average. The LOESS captures the major trends in the data, but is less severely affected by week to week fluctuations such as those occuring around Thanksgiving and over the year-end and New Year holidays.

Using LOESS to analyze the body mass indexes (BMI) of Playboy playmates gives more insights than linear regression over the whole data set or over portions of the data. See the discussion in Wired Relates Playboy Playmate BMI and Average BMI, 1954-2008 on the FlowingData blog.

The LOESS Function

Public Function LOESS(X As Variant, Y As Variant, xDomain As Variant, nPts As Long) As Double()
  Dim i As Long
  Dim iMin As Long
  Dim iMax As Long
  Dim iPoint As Long
  Dim iMx As Long
  Dim mx As Variant
  Dim maxDist As Double
  Dim SumWts As Double, SumWtX As Double, SumWtX2 As Double, SumWtY As Double, SumWtXY As Double
  Dim Denom As Double, WLRSlope As Double, WLRIntercept As Double
  Dim xNow As Double
  Dim distance() As Double
  Dim weight() As Double
  Dim yLoess() As Double

  If TypeName(X) = "Range" Then
    X = X.Value
  End If

  If TypeName(Y) = "Range" Then
    Y = Y.Value
  End If

  If TypeName(xDomain) = "Range" Then
    xDomain = xDomain.Value
  End If

  ReDim yLoess(LBound(xDomain, 1) To UBound(xDomain, 1), 1 To 1)

  For iPoint = LBound(xDomain, 1) To UBound(xDomain, 1)

    iMin = LBound(X, 1)
    iMax = UBound(X, 1)

    xNow = xDomain(iPoint, 1)

    ReDim distance(iMin To iMax)
    ReDim weight(iMin To iMax)

    For i = iMin To iMax
      ' populate x, y, distance
      distance(i) = Abs(X(i, 1) - xNow)
    Next

    Do
      ' find the nPts points closest to xNow
      If iMax + 1 - iMin <= nPts Then Exit Do
      If distance(iMin) > distance(iMax) Then
        ' remove first point
        iMin = iMin + 1
      ElseIf distance(iMin) < distance(iMax) Then
        ' remove last point
        iMax = iMax - 1
      Else
        ' remove both points?
        iMin = iMin + 1
        iMax = iMax - 1
      End If
    Loop

    ' Find max distance
    maxDist = -1
    For i = iMin To iMax
      If distance(i) > maxDist Then maxDist = distance(i)
    Next

    ' calculate weights using scaled distances
    For i = iMin To iMax
      weight(i) = (1 - (distance(i) / maxDist) ^ 3) ^ 3
    Next

    ' do the sums of squares
    SumWts = 0
    SumWtX = 0
    SumWtX2 = 0
    SumWtY = 0
    SumWtXY = 0
    For i = iMin To iMax
      SumWts = SumWts + weight(i)
      SumWtX = SumWtX + X(i, 1) * weight(i)
      SumWtX2 = SumWtX2 + (X(i, 1) ^ 2) * weight(i)
      SumWtY = SumWtY + Y(i, 1) * weight(i)
      SumWtXY = SumWtXY + X(i, 1) * Y(i, 1) * weight(i)
    Next
    Denom = SumWts * SumWtX2 - SumWtX ^ 2

    ' calculate the regression coefficients, and finally the loess value
    WLRSlope = (SumWts * SumWtXY - SumWtX * SumWtY) / Denom
    WLRIntercept = (SumWtX2 * SumWtY - SumWtX * SumWtXY) / Denom
    yLoess(iPoint, 1) = WLRSlope * xNow + WLRIntercept

  Next

  LOESS = yLoess

End Function
 

The LOESS Utility

The flexibility of this LOESS function will make it easy to encapsulate into an add-in that uses a dialog to facilitate user selection of data and parameters. A working version uses the following dialog:

Update 24 June 2009

A LOESS utility for Excel has finally been made ready for public consumption. It is described in LOESS Utility for Excel, where there is a link to download the utility. It’s still in preliminary form, but runs pretty much trouble free. Users are encouraged to comment on it to drive further development.

Update 8 October 2009

The LOESS utility for Excel has been updated, and the interface made more flexible. It is described in LOESS Utility – Awesome Update, where there is a link to download the new utility.

References

Cleveland, W.S. (1979), “Robust Locally Weighted Regression and Smoothing Scatterplots,” Journal of the American Statistical Association, Vol. 74, pp. 829-836.

Cleveland, W.S. and Devlin, S.J. (1988), “Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting,” Journal of the American Statistical Association, Vol. 83, pp. 596-610.

NIST Engineering Statistics Handbook, 4.1.4.4. LOESS (aka LOWESS)

NIST Engineering Statistics Handbook, Example of LOESS Computations

Wikipedia, Local regression

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Scenario 1

You’ve made a gorgeous chart of the data in Sheet1. You copied the chart from Sheet1 to Sheet2 so you could plot Sheet2’s data in the same splendor. And you hit a snag: the chart on Sheet2 refers back to Sheet1’s data. There are two ways to correct this:

• Create a copy of Sheet1 including the chart, so the chart on the copied sheet refers to the data on the copied sheet. Then copy Sheet2’s data and paste it over the copied sheet’s data.
• Edit the series formulas of the copied chart in Sheet2, changing all instances of one sheet name to the other. This becomes tedious if there are multiple series in the copied chart, or if you’ve copied multiple charts.

Scenario 2

You’ve charted data in rows 1 to 100 of your worksheet. Then you’ve updated the data so it reaches down to row 150. Your wonderful chart only shows data down to row 100. There are three ways to correct this:

• Before updating the data, convert the data range to a List (Excel 2003) or a Table (Excel 2007). Then update the data. The list/table will expand to include all of the data, and all formulas that refer to all rows in the list/table will update accordingly. This includes the chart’s SERIES formula.
• Create dynamic named ranges for the X and Y data ranges in the chart, and apply them to the chart series. This is described in Dynamic Charts in this blog and in a number of Dynamic and Interactive Chart examples described in this web site. However, this is an involved procedure that you never have time for.
• Edit the series formulas of the chart, changing all instances of one row number to another. This becomes tedious if there are multiple series  or multiple charts to correct.

Scenario 3

Your charts show the data for product Alpha beautifully, but you want to show the data for product Beta instead. The data is listed in another column. You can:

• Select the series, then drag the highlighted rectangles on the worksheet to reflect the new data range. This becomes a laborious process for multiple charts and series.
• Edit the series formulas of the chart, changing all instances of one column to another. This becomes tedious if there are multiple series  or multiple charts to correct.

The Solution

These suggested means for editing your charts all leave out one approach. In Change Series Formulas I showed how to programmatically change the series formulas in your charts, and I provided a utility that does the task for you. I’ve just recently updated the utility to account for glitches in Excel, and to streamline using the utility.

The Change Series Formula Utility

The new utility is located in PTS_ChangeSeriesFormula.zip. You can install it following the protocol in Installing an Excel Add-In.

Upon installation, the utility creates a toolbar (In Excel 2007, the toolbar buttons are buried on the Add-Ins tab of the ribbon).

Not too fancy or complicated. Suppose I have the following scenario, a chart showing data for Alpha in column B, and I want to show Beta from column C instead.

If it’s only one series in one chart, I can drag the colored highlight rectangles in the sheet, or edit the series formula in the formula bar. But if there’s more, why waste time? With the chart selected (I like to select the series, so the formula bar is visible and I don’t have to remember what to change), click on the Change Series Formula button on the toolbar. Enter the old and new text in the dialog. In the old text entry box, I like to explicitly use punctuation like the dollar signs, to make sure I’m changing a column designation and not some other text in the formula. In the new text box, it doesn’t matter, because Excel capitalizes and adds dollar signs automatically.

Click the Change Active Chart Only button (the other button is inactive because there are no other charts on this worksheet).

Magically, the series formula, the colored highlights, and the chart itself now reference the new data.

Here is how I would extend a chart’s data from row 6 to row 12. In this case there are multiple charts on the worksheet, and you have a choice of changing the active chart or changing all charts on the worksheet.

This dialog shows how to change references from Sheet1 to Sheet2. In addition, the buttons reflect that multiple charts have been selected using Shift+Click. The options are to change all selected charts or all charts on the worksheet.

It’s a pretty smart utility, and it has saved me tons of time. Download it, install it, and use it. Give me feedback, make suggestions, and report errors in a comment below.

Peltier Technical Services, Inc., Copyright © 2010.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

In the earlier two posts, I found that the drainage time was related to hole diameter through a power law relationship with an exponent of -2, and to height with an exponent of 1/2.

Often when modeling a physical process you have theoretical relationships that describe the process, and you just need to find the appropriate factors in the formula. Then you can manually plug and chug to find these factors, or you can use an analytical tool to find them.

In Goal Seek – Optimization Approach to a Simple Physics Problem I showed how to use the built-in Goal Seek optimization tool to determine the coefficient to the power law function involving initial water height and hole diameter. Excel has another tool, an optional add-in called SOLVER, which is a more powerful optimization tool.

I have set up my data as shown below. The initial data is in C9:F12. In C3:F6 I have calculated values for a formula of the form

t = K * ha * db
 

where t is time to drain the bucket, h is the water height, d is hole diameter, K is the unknown coefficient, a is the exponent -2, and b is the exponent 0.5. I put a provisional factor K in cell A1, based on the preliminary regression work. Cell C3 has this formula, which is copied into the whole range C3:F6:

=$A$1*C$2^0.5/$B3^2
 

Finally, C15:F18 tabulates the squared differences between the calculations and the actual data. The formula in C15 is simply

=(C3-C9)^2
 

Cell A16 contains the sum of all of these differences (sum of squares). This is more routinely used than the simple sum of differences used in the Goal Seek approach, but the final answers are close.

I used SOLVER to minimize the sum of the squared differences in A16 by changing the factor in cell A1. Unlike Goal Seek, SOLVER actually does find a minimum.

From the Tools menu, choose SOLVER (in Excel 2007, on the Data ribbon tab, click on Solver in the Analysis group). If the command for SOLVER is unavailable, you will have to install it. See Installing an Excel Add-In for instructions; the SOLVER add-in is either present in the list on the Add-Ins dialog, or you can click the Browse button and look in the SOLVER folder.

The following dialog allows you to indicate which cell to change in order to set another cell to a particular value. We are not using the powerful capabilities of SOLVER to include constraints in the model.

Click OK, and SOLVER does its magic, and shows a dialog which either shows a solution or explains why no solution was found. SOLVER did find a solution to this problem.

Here is the complete SOLVER solution.

These charts show the SOLVER solution (markers and dotted lines) in comparison to the original data (solid lines).