Here is a pivot table with daily values. I worked this out in Excel 2003, because that’s what was open at the time, but the technique is much the same in all versions. I want to condense this into monthly values, and show only data from 2010 and 2011.

I navigate to the Group and Show Detail > Group command, and choose the appropriate parameters in the Grouping dialog.

I change the range of dates in the pivot table by unchecking the two Auto boxes at the top and entering the start and finish dates I want. Then I select the time units I want the data grouped by. Pick Months and Years, or you’ll only get 12 monthly values summed over all the years within selected the date range.

The pivot table is condensed into the desired time periods. The only problem is that Excel includes two additional pivot items in each date-related pivot field. The pivot fields Years and Date have pivot items <1/1/10 and >12/31/11 to account for data from outside our selected date range.

We can click on the field header dropdowns and see the pivot items in each. The Years and Date pivot item lists are shown below.

The pivot items have checkboxes, so we can manually uncheck the extraneous items, every time the pivot table is refreshed. Bo-o-o-oring.

I’ve actually been doing this manually for years in some of my accounting workbooks. Like the plumber’s faucet that always drips, the programmer’s workbook gets updated by hand. But a reader emailed me and asked how to get rid of those extra date items. I thought for about 30 seconds, and coded for about 3 minutes, and came up with this routine that cleans up any fields in the Row and Column areas of all pivot tables on the active sheet that have items beginning with “<” or “>”.

Sub Remove_GT_LT_PivotItems()
  Dim pt As PivotTable
  Dim pf As PivotField
  Dim pi As PivotItem

  For Each pt In ActiveSheet.PivotTables

    For Each pf In pt.RowFields
      For Each pi In pf.PivotItems
        If Left$(pi.Caption, 1) = "<" Or Left$(pi.Caption, 1) = ">" Then
          pi.Visible = False
        End If
      Next
    Next

    For Each pf In pt.ColumnFields
      For Each pi In pf.PivotItems
        If Left$(pi.Caption, 1) = "<" Or Left$(pi.Caption, 1) = ">" Then
          pi.Visible = False
        End If
      Next
    Next

  Next

End Sub

Here is the finished pivot table.

Peltier Technical Services, Inc., Copyright © 2011.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

I proposed a panel chart to show the stacked chart data more clearly. Joe Mako commented that the line chart I showed prior to building a panel chart could be cleaned up by taking the rolling percentages of the cumulative total of the original data. I agreed that it makes the chart look cleaner, but I think it forces us to sacrifice too much detail in the data.

I’ll illustrate my reasoning in this post.

Individual Values

These are the original values I used in my previous post.

Here are the stacked column chart and line chart of this data. They are rather cluttered, and not easy to interpret.

Here is the panel chart. Splitting the data into separate panels according to category makes it much easier to see trends within an individual category and to compare values across categories.

Percentages of Annual Total

This table shows the data as a percentage of the annual total. For example, the “alpha” value for 2005 is the alpha value divided by all values, or 16/(16+14+13+10+8), or 26.2%.

The stacked chart is now flat across the top, because the total in each year is 100%. The details within the stacked column chart and the line chart are no easier to see than the charts showing the individual values. In fact, it’s hard to see the differences between these and the earlier charts.

The panel chart of the percentage data isn’t much different than that for the individual values, and it is much easier to read than the stacked and line charts.

Rolling Percentages of Cumulative Total

Here are the calculations proposed by Joe Mako. The values for the first year are the same. The values for the each subsequent year are the values for a given category for all years so far divided by the values for all categories for all years so far. So alpha for 2006 would be (16+18)/((16+14+13+10+8)+(18+17+12+9+8)), or 27.2%.

Essentially each year’s rolling value is a weighted average of the current value with the previous years’ values, which results in a smoothing of the data. The data is smoothed so much that it’s not easy to see much difference at all within any particular category in the stacked chart below. In the line chart, we can make out trends, but they are not nearly as pronounced as in the original data or in the annual percentages.

The panel chart also shows this smoothed data.

What’s the Big Deal?

So why don’t I like this smoothing of the data? Smoothing can be helpful when you’re trying to find patterns in noisy data. In fact, I’ve written about LOESS smoothing in Excel, and I’ve released a utility to perform LOESS smoothing on worksheet data.

But smoothing can also obliterate details in data, and it can give the wrong impression of trends in the data.

In the latest stacked column chart above, the variation in the data from year to year has been almost smoothed out of existence. The latest line chart contains more white space, because points have been moved away to provide this space.

The line charts below compare how the unsmoothed data compares to the rolling average data. In the chart on the left, the first and last years in the analysis are connected, without showing the intervening years. The solid symbols and lines show the annual percentages, while the open symbols and dashed lines show the rolling averages. The lines start at the same points in 2005, and both sets of data move in the same direction. But for all categories, the rolling averages change much less than the annual averages. The rolling averages make it seem that alpha and beta have become close in value, while the annual averages show that they have diverged to a great degree. The rolling averages show that gamma, delta, and epsilon have moved very close to each other, while the annual averages have changes so much that the categories have crossed each other and reversed their order. The line chart on the right is another view of the two sets of data.

As is often the case, the panel chart shows the data most clearly. The blue solid diamond markers show the annual percentages, with their considerable variability, while the red open squares show very gradual changes. The smoothed data in this case tends toward the overall time average of the data.

Data Smoothing

Data soothing can be a useful tool for teasing patterns out of noisy data. It’s important that the consumer of the data understands the unsmoothed and smoothed data, and how the smoothed data may have been distorted by the smoothing technique used.

Peltier Technical Services, Inc., Copyright © 2011.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Interpolation requires some simple algebra. The diagram below shows two points (blue diamonds connected by a blue line) with coordinates (X1, Y1) and (X2, Y2). We need to find the value of Y corresponding to a given X, represented by the red square at (X, Y).

The smaller triangle with hypotenuse (X1, Y1)-(X, Y) is “similar” to the larger triangle with hypotenuse (X1, Y1)-(X2, Y2), so the sides of the triangles are proportionally sized, leading to the first equation below the sketch. We rearrange this to solve for Y, in the second equation.

We’ll set up our interpolation in the example below. Our data is in A5:B18, and the known values are plotted as blue diamonds connected by blue lines in the chart.

The analysis has two parts: first we need to determine which pair of points to interpolate between, second we need to do the interpolation. We will judge the validity of our interpolation by plotting the calculated point on the same chart.

Solving for Y

I’ve put the calculations above the data table. The yellow shaded cell, A2, holds the known X value, and a formula in cell B2 holds the calculated Y value. Cell A3 indicates which pair of points to interpolate between. The formulas are:

A3: =MATCH(A2,A6:A18)

B2: =INDEX(B6:B18,A3) + (A2-INDEX(A6:A18,A3)) * (INDEX(B6:B18,A3+1)-INDEX(B6:B18,A3)) / (INDEX(A6:A18,A3+1)-INDEX(A6:A18,A3))

We want the Gauge value (Y) when the Flow value (X) equals 3, so this is entered into the yellow shaded cell A2. The formula in A3 tells us that our computed point is between the 7th and 8th data point, and the formula in B2 calculate Y=0.444, and the calculated point (A2, B2) is the red square that lies along the plotted data points. Looks good.

Te determine the Gauge value for a Flow value of 12, we enter this into A2. The red square moves along the blue line past the 11th point, where Gauge=0.548.

Solving for X

With a minor rearrangement, we can instead solve for Flow, given a value for Gauge. The known Gauge value is entered into B2, shaded yellow. B3 indicates the pair of points to interpolate between, and A2 provides the value for Flow. The formulas are:

B3: =MATCH(B2,B6:B18)

A2: =INDEX(A6:A18,B3) + (B2-INDEX(B6:B18,B3)) * (INDEX(A6:A18,B3+1)-INDEX(A6:A18,B3)) / (INDEX(B6:B18,B3+1)-INDEX(B6:B18,B3))

For a Gauge (Y) of 0.53, we compute a Flow (X) of 9.790. The red square shows where the calculate point lies along our plotted data.

Changing the Gauge value to 0.35 moves the red square way to the left, to a Flow value of 0.400.

Peltier Technical Services, Inc., Copyright © 2011.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Another problem, which is stated in the Trendline Fitting Errors post is that the reader had calculated a 6th order polynomial fit to his data. This is fine, I guess, if you only need the data for interpolation, the fit is close, and due to curvature the fit deviates from a straight line between points.

Despite the problems with using a sixth order fit, I’ve decided to work out the calculations and compute the area under the curve, both under the measured data and unser the calculated data based on the fit.

The Experimental Data

The measured data is shown in the table below. This represents the force needed to move a surface closer to a particular molecule starting at a distance of 1.22 (which is asymptotically close enough to infinity that we assume a force of zero). The units of force and distance are not stated, but it doesn’t matter for this exercise. As the surface approaches the molecule, the force increases, until it is close enough to start to distort the molecule, at which point the force tops out and even decreases at the closest measurement.

Here’s a plot of the data.

When I first saw the data, I mentioned to the reader that there appeared to be a bump in the curve near the bottom. I’ve zoomed in on this region below. The curve coming from the upper left doesn’t match up with the curve coming from the lower right, resulting in a zig-zag at about X=0.95.

To me this means there may be some slippage in the mechanism that moves the surface, or some other experimental inconsistency. I would think that the actual curve would look like the dotted line below. If it were my experiment I would repeat the measurements, but the exercise below was performed on the unmodified data.

The Trendline

I added a 6th order polynomial trendline to the data. Then, since trendlines are bold black lines that blast your eyes and obscure the data being fitted, I formatted the fitted curve as a thin red line. For most of the range of data the trendline fits pretty closely to the measured data, and the R² value of 0.9996 is very close to a perfect 1.0.

Near the top of the data, the trendline is not very accurate, overshooting the maximum of the measured data, then overcompensating for the decline in the last point.

The fit at the opposite end also isn’t very good. The red line careens back and forth like an out-of-control bobsled trying to hold a line. Note how the trendline is thrown off by the discontinuity I remarked about earlier. The trendline actually reaches a local minimum at about X=1.15, then a local maximum at about X=1.20, before continuing downward, but not quite to zero.

These deviations of the trendline from the measurements, particularly the serpentine behavior at the lower end of the data, illustrates the problem with high order polynomial fits. There’s really not a physical basis for choosing such a fit; it’s simply convenient and gives a high R². Of course, a high R² is not the only reason to select a particular mathematical model, and does not by itself mean the model is a good one. You have to decide whether you think the selected model or the actual measurements know more about your data.

We must take care using the trendline equation in the earlier chart. Its coefficients have only five digits, and with so many coefficients multiplied by so many powers of X, errors can accumulate. Here are the points calculated with these imprecise coefficients. Not very good agreement: the calculated points deviate somewhat more than the trendline at the top, and they go off in another direction entirely at the bottom.

We can improve the precision of the coefficients by formatting the trendline formula in the chart to use scientific notation with 14 digits after the decimal point. It’s ugly, but it’s precise.

Calculating Coefficients

Now, we could then retype all of these coefficients into worksheet cells, but that would take a long time and leave us cross-eyed. The better approach is to let Excel make the calculations using its LINEST worksheet function. You need a range with five rows and one plus the order of the poly fit columns. For a sixth order fit, we need seven columns. Select the range with the active cell in the top left of the range, type in the formula below, then hold down CTRL and SHIFT while pressing ENTER. CTRL+SHIFT+ENTER produces an array formula, which is a topic that could cover dozens of blog posts.

=LINEST(B2:B19,A2:A19^{1,2,3,4,5,6},,TRUE)

B2:B19 contains the Y values, A2:A19 contains the X values. We denote a six order poly fit with the ^{1,2,3,4,5,6} notation, which tells Excel to apply each of the exponents in brackets to X in its calculations. The third argument is left blank, because we don’t want to force the fit to go through the origin, and the last is TRUE because we do want to fill the entire selected range with calculations. (The first row of the output contains our coefficients. The second through fifth rows of the output contain information about the coefficients and the model, so we could have simply selected the first row of the range and not bothered with the rest.)

After pressing CTRL+SHIFT+ENTER we get the following table. I’ve inserted headers at the top to remind me which coefficient is which.

Calculating Points

Now we can calculate values at the X values of interest. In this case, I’ve used the same X values for which we have measured data. The formula in cell C2 is shown below, and it is filled down to C19.

=$N$2+$M$2*A2+$L$2*A2^2+$K$2*A2^3+$J$2*A2^4+$I$2*A2^5+$H$2*A2^6

These points exactly fit the trendline, and are pretty close to the measured data.

I’ve removed the curved trendline and connected the calculated points with straight lines. With this formatting it may be easier to see the deviation of the calculated points from the measured data.

Residuals

We can easily calculate the deviation between measured and calculated points. Column D simply shows the difference between columns C and B (calculated minus measured).

And here is a plot of the residuals, which is the fancy word statisticians use for this deviation. It’s particularly high at the left end of the data (the top, where the trendline overshot the maximum, then overcompensated on the rebound.

Calculating Areas

The original question the reader had was “What’s the area under the curve?” I think the whole polynomial overfit was really a distraction.

To calculate the area under a curve, we can cut the area into slices, figure out the area of each slice, then add them up to get the total area. We already have data points at certain intervals, so let’s slice the curve at each point. Here is the sliced up area under the measured curve.

Here’s the sliced up area under the calculated curve. Not too different.

The slices are trapezoids, and we know the area of a trapezoid: average height times thickness. In columns E and F I’ve calculated the areas under the measured and calculated curves. The formula in cell E2, filled down to E18, is

=(A3-A2)*(B2+B3)/2

where A3-A2 is the thickness and (B2+B3)/2 is the average height. The formula in cell F2, filled down to F18, is

=(A3-A2)*(C2+C3)/2

The total areas are summed in row 21.

The two computed areas are unusually close, differing by less than 0.03%. In this case, there was no benefit to using a trendline to calculate this area.

You could make the case that trapezoids don’t accurately capture the area under a curve if the data shows lots of curvature. If we had taken measurements more frequently, our points would lie under or over the straight top segments of our trapezoids. If we believe our trendline, we could calculate values at closer intervals, as shown below. We might then say that the computed area was more accurate.

The area calculated for the thinner slices was 1.404070, which is about 0.2% less than the areas computed using thicker trapezoids. This difference is probably from the range between X=0.80 and X=0.97, where curvature in the trendline moved it below the straight line segments of the measured data. Is this a better value? It’s not substantially different from the calculation based on only the unmodified measurements, and I’m sure there are greater sources of error in the experimental setup.

Peltier Technical Services, Inc., Copyright © 2011.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Most commonly you have a worksheet with a bunch of data and a corresponding chart, and you have another sheet of data you want to add a chart to. Copying and pasting the chart onto the new sheet requires you to change links in the chart, usually series by series. This is tedious and error-prone. But the article above describes an easier way:

1. Make a copy of the worksheet with the old data and chart;
2. Copy the new data;
3. Paste the new data over the old data on the copied worksheet.

The chart on the new worksheet updates as soon as the new data is pasted into place. Works in Excel 2003 and earlier, and in Excel 2007 if you’ve installed the latest service packs.

What if the worksheet contains a pivot table and its sister pivot chart? Well, knowing the above protocol, you’d think you could copy the worksheet, change the data, and the chart would be fine. And in Excel 2003 this set of steps works great:

1. Make a copy of the worksheet with the old pivot table and pivot chart;
2. Change the pivot table’s data source to the new range;
3. Refresh the pivot table.

The new pivot chart (on the copied sheet) retains its link to the pivot table on its parent worksheet, so it updates as soon as the pivot table is refreshed.

But in Excel 2007, these steps  don’t work the same way. When you copy the worksheet with the pivot table and chart, not only does the new pivot table link to the same old data, the new chart also links to the old pivot table. You can easily enough change the data source of the new pivot table to the appropriate range; in fact, this is easier to do in Excel 2007 than in earlier versions. But the new chart cannot be linked to the new pivot table. It is permanently linked to the old pivot table.

My colleague Bill Manville wrote to me about this problem, citing an old forum post in which I doubted this could ever be solved. I’m glad to say that Bill has proved me wrong. He sent me a new protocol that makes this work.

1. Make a copy of the worksheet with the old pivot table and pivot chart in a different workbook;
2. Move the copied worksheet back into the original workbook;
3. Change the new chart’s source data to the new pivot table;
4. Change the pivot table’s data source to the new range;
5. Refresh the pivot table.

The difference is that the worksheet is copied into a new workbook (or another existing workbook) rather than within the original workbook. When this happens, the pivot table still links to the original data, but the chart becomes unlinked from the pivot table. In fact, the chart changes back from a pivot chart to a regular chart. It is unlinked from any data range, and the series formulas have been converted to written-out arrays.

Since the chart has been unlinked in the process, it now can be relinked. Thanks, Bill, for saving us lots of time and effort.

Bill tells me that the familiar Excel 2003 behavior has been restored to Excel 2010.

Peltier Technical Services, Inc., Copyright © 2011.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
 

Experimental Design

There are two ways to find the minimum and maximum of a set of values:

• Loop through the values, and keep track of the largest and smallest you’ve encountered
• Use the worksheet functions MIN() and MAX() (i.e., WorksheetFunction.Min and .Max)

There are three ways to obtain the values for each series:

• Use the .Values property of the series, which returns a 1-D array
• Parse the series formula to identify the range, then get the .Value of this range object, which returns a 2-D array
• Parse the series formula to identify the range, and analyze the range itself

This suggests a 2×3 matrix, but I eliminated looping cell-by-cell through the range to find the largest and smallest values, since experience shows that looping through the cells of a range in this way is much slower than using the .Value property of the range to generate an array, and working with this array.

The five-block test matrix becomes:

• Looping Over Series Values
• Worksheet Function on Series Values
• Looping Over Range Values
• Worksheet Function on Range Values
• Worksheet Function on Range Object

I wrote VBA procedures to perform each of these approaches. Then I wrote a master procedure that called each of these procedures 1000 times, 50 times each in 20 blocks, so I’d have the mean time for each block. The overall mean and standard deviation of the block means can be used to compare the different approaches.

I used a simple XY chart with 1000 points. Both X and Y values were taken from the standard normal distribution. The procedures below could be adapted to check the Y values of any chart and the X values of an XY chart or a line/column/bar/area chart with a date scale X axis. Here the X values don’t matter since I was merely sampling Y.

I ran the tests in Excel 2003 SP3 on a reasonably modern laptop, running Windows 7 Ultimate with 4GB of memory on a 2.20 GHz dual-core AMD processor.

VBA Procedures

Looping Over Series Values

Sub Max_Min_Loop_Values()
  Dim Cht As Chart
  Dim Srs As Series
  Dim i As Long
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  A = Srs.Values
  MaxVal = A(LBound(A, 1))
  MinVal = A(LBound(A, 1))
  For i = LBound(A, 1) + 1 To UBound(A, 1)
    If A(i) > MaxVal Then MaxVal = A(i)
    If A(i) < MinVal Then MinVal = A(i)
  Next i
End Sub

Assuming there is one chart on the active worksheet, and one series in the chart, we set the chart and the series simply. We assign the Y values of the series to an array, and initially set the minimum and maximum values to the first value in the array. Then we loop from the second to the last values in the array, and if a given value is larger than the maximum value so far, we set the new maximum to the current value, and likewise for the minimum. I didn’t output the minimum and maximum values, all I wanted was the time to run the core of the procedure.

Worksheet Function on Series Values

Sub Max_Min_Wkfn_Values()
  Dim Cht As Chart
  Dim Srs As Series
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double  

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  A = Srs.Values
  With WorksheetFunction
    MaxVal = .Max(A)
    MinVal = .Min(A)
  End With
End Sub

We set the chart and the series simply as above. We assign the Y values of the series to an array, and use the worksheet functions Min() and Max() to determine the minimum and maximum values.

Looping Over Range Values

Sub Max_Min_Loop_RangeArray()
  Dim Cht As Chart
  Dim Srs As Series
  Dim i As Long
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double
  Dim sSeriesFmla As String 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  sSeriesFmla = Split(Srs.Formula, ",")(2)
  A = Range(sSeriesFmla).Value
  MaxVal = A(LBound(A, 1), 1)
  MinVal = A(LBound(A, 1), 1)
  For i = LBound(A, 1) + 1 To UBound(A, 1)
    If A(i, 1) > MaxVal Then MaxVal = A(i, 1)
    If A(i, 1) < MinVal Then MinVal = A(i, 1)
  Next i
End Sub

We set the chart and the series simply as above. We parse the formula of the series to find the range, use the .Value property of the range to define an array, and initially set the minimum and maximum values to the first value in the array. Then we loop from the second to the last values in the array, and if a given value is larger than the maximum value so far, we set the new maximum to the current value, and likewise for the minimum.

Worksheet Function on Range Values

Sub Max_Min_Wkfn_RangeArray()
  Dim Cht As Chart
  Dim Srs As Series
  Dim A As Variant
  Dim MinVal As Double
  Dim MaxVal As Double
  Dim sSeriesFmla As String 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  sSeriesFmla = Split(Srs.Formula, ",")(2)
  A = Range(sSeriesFmla).Value
  With WorksheetFunction
    MaxVal = .Max(A)
    MinVal = .Min(A)
  End With
End Sub

We set the chart and the series simply as above. We parse the formula of the series to find the range, use the .Value property of the range to define an array, and use the worksheet functions Min() and Max() to determine the minimum and maximum values.

Worksheet Function on Range Object

Sub Max_Min_Wkfn_Range()
  Dim Cht As Chart
  Dim Srs As Series
  Dim rng As Range
  Dim MinVal As Double
  Dim MaxVal As Double
  Dim sSeriesFmla As String 

  Set Cht = ActiveSheet.ChartObjects(1).Chart
  Set Srs = Cht.SeriesCollection(1)
  sSeriesFmla = Split(Srs.Formula, ",")(2)
  Set rng = Range(sSeriesFmla)
  With WorksheetFunction
    MaxVal = .Max(rng)
    MinVal = .Min(rng)
  End With
End Sub

We set the chart and the series simply as above. We parse the formula of the series to find the range, and use the Worksheet Functions Min() and Max() to determine the minimum and maximum values of the range object directly.

Timing Procedure

Sub TestTimes()
  Dim Looper As Long
  Dim Blocker As Long
  Dim tStart As Double
  Dim tTotal As Double
  Const LooperMax As Long = 50
  Const BlockerMax As Long = 20  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Loop_Values
    Next
    tTotal = Timer - tStart
    WriteToLog "Loop - Values: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Wkfn_Values
    Next
    tTotal = Timer - tStart
    WriteToLog "Func - Values: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Loop_RangeArray
    Next
    tTotal = Timer - tStart
    WriteToLog "Loop - RngArr: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Wkfn_RangeArray
    Next
    tTotal = Timer - tStart
    WriteToLog "Func - RngArr: " & tTotal & " sec for " & LooperMax & " iterations"
  Next  

  For Blocker = 1 To BlockerMax
    tStart = Timer
    For Looper = 1 To LooperMax
      Max_Min_Wkfn_Range
    Next
    tTotal = Timer - tStart
    WriteToLog "Func - Range:  " & tTotal & " sec for " & LooperMax & " iterations"
  Next 

End Sub

This procedure calls each of the Min_Max procedures LooperMax (50) times, records the total time for 50 iterations, and repeats this BlockerMax (20) times for each procedure, before moving on to the next. The times are recorded in a text file using the WriteToLog procedure below.

Logging Procedure

Public Sub WriteToLog(sLogEntry As String)
  ' write information to a log file
  Dim iFile As Integer
  Dim sFileName As String
  Const sLogFile As String = "TimeLogger"   

  sFileName = ThisWorkbook.Path & "\" & sLogFile & Format$(Now, "YYMMDD") & ".txt"
  iFile = FreeFile
  Open sFileName For Append As iFile
  Print #iFile, Now; " "; sLogEntry
  Close iFile
End Sub

This is a very handy procedure I use all the time for logging debug information or for saving settings. It uses good old VB I/O protocols, which are very fast, to populate a text file.

Test Results

The results of these experiments are shown in the following table. The approaches are sorted from slowest to fastest. Processing an array extracted from the source data range takes about 1.2 milliseconds, whether looping or using the worksheet functions. Processing the series values array takes about 0.7 milliseconds by either technique. Measuring the range directly takes less than 0.5 milliseconds.

50-iteration means of the 20 blocks for each approach are plotted in this chart, with means sorted from fastest to slowest to give a pseudo-distribution of procedure times.

This chart is stepped, because there seems to be a discrete quantum of time, that is, a resolution of about 0.078 milliseconds in the measurement of time using VBA’s Timer function on this microprocessor.

What These Results Mean

One school of thought is that looping an array would be slow compared to a single function call, while another is that any call from VBA to Excel (i.e., WorksheetFunction) would incur a performance hit. An unexpected result is that on this machine, it makes essentially no difference whether I use a loop or worksheet functions to find the minimum and maximum values in a series of 1000 points. I guess that’s why we occasionally bother to run these time trials.

We see that using the array of series values is faster than using the array of values from the source data range, by not quite a factor of two. Thinking of the VBA-Excel interface, we have to cross it once to get the series values, or twice to get the series formula and again to get the range values. This makes sense, and I have to admit, I would never have thought of getting the values from the source data range, since they are so easy to extract using series .Values.

The fastest method of all, which was proposed by Daniel Ferry (Excel Hero), is to parse the series formula to find the range, and directly measure the min and max of this range. Apparently the performance hit to cross the VBA-Excel barrier is offset by the much faster performance of worksheet functions operating on Excel objects directly within Excel.

Although the direct range object measurement approach is fastest, I will stick to my series values approach. The main reason is that not all series values come from a single range in a worksheet, or even from a worksheet range at all: some series use multiple area ranges, while others have values hard-coded as a literal array directly in the series formula.

This is a “normal” series formula (based on the first ten points of experimental data from our chart above):

=SERIES(Sheet1!$B$1,Sheet1!$A$2:$A$11,Sheet1!$B$2:$B$11,1)

In this formula, the series name is the argument in green, X values are in purple, Y values are in blue, and plot order is in red. These are the same colors Excel uses to highlight the data ranges of a selected series in the underlying worksheet (illustrated below).

This formula is for a series that uses multiple area ranges for X and Y values, the areas for X and Y enclosed in parentheses:

=SERIES(Sheet1!$B$1,(Sheet1!$A$2:$A$501,Sheet1!$A$502:$A$1001),
(Sheet1!$B$2:$B$501,Sheet1!$B$502:$B$1001),1)

Multiple area formulas are not too uncommon. Less common is a series whose formula has hard-coded arrays as series values (first ten points of experimental data converted to values using F9 key):

=SERIES("Y",{-0.172902780162854;0.0788276881579948;0.752718316150293;-1.15534740638389;
-0.651271029427549;1.61522977181724;0.218894823127643;-1.38264191435207;
0.107605524088065;0.281018951052876},
{-0.640448122430912;-0.355817827507841;-0.318480200209528;-0.0116013833433011;
-1.15185971606638;0.183605522997797;-0.376897171002227;-0.488810695788606;
-2.14375416629929;0.154502474308393},1)

There is a practical limit of around 250 characters (you’d guess 255, but it’s slightly less) each for the X values and Y values of a series formula. In the formula above, the ten X values consume 183 characters, the ten Y values 188. You can squeeze in more values by reducing the resolution of the values (e.g., -0.173 in place of -0.172902780162854), but that’s just nibbling around the margins. A practical limit is around 17 or 18 values before the formula crashes.

The arrays (within {curly braces} above) are delimited by semicolons in this case, because the original data came from a column, but data in rows would be delimited by commas.

These complications in the series formula lead to errors when using simple approaches to parse such formulas. John Walkenbach has developed A Class Module To Manipulate A Chart Series which can deal with such difficult situations.

Since accessing values using the .Values property is so easy, and since this approach is only about 50% slower than the risky approach of parsing the formula and measuring the range directly, that’s the approach I will stick with.

Peltier Technical Services, Inc., Copyright © 2011.
 
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.