LOESS Utility – Awesome Update
by Jon Peltier
Thursday, October 8th, 2009
Peltier Technical Services, Inc., Copyright © 2010.
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
I talked about LOESS smoothing in LOESS Smoothing in Excel, where I showed my improved VBA function for calculating smoothed data. I introduced an improved LOESS Smoothing utility in LOESS Utility for Excel. Since I use the utility frequently, I encountered many things about it that I wanted to change. Since I am the developer of the utility, I have actually been able to make these changes. And after a few months of use and a few hour-long sessions aimed at making it do what I want it to do, I’ve developed this utility into something that is, in the local vernacular, “wicked awesome”. If you’ve never seen “Good Will Hunting”, that means “way cool”.
What makes this utility so great, you ask? The calculations were already perfectly adequate, and I didn’t change them at all. But I’ll show the new dialog, so you can see what has changed.

First, the previous interface for selecting input and output ranges was frustratingly restrictive. So I made the range selection function much more flexible. The colorful buttons on the left indicate the built-in options. In the most elementary case, all four entries (X input, Y input, X output, and Y output) are separate ranges, which can even be located on separate worksheets.

The options make range selection easier if some columns happen to be adjacent to each other, or if the same X values are to be used for both input and output. The range selection (Ref Edit) boxes and related labels update according to which option has been selected.
When the X input range has been selected, the program determines Npts, the number of points that should be used for the selected value of alpha. When the user changes alpha or Npts, the other parameter updates.
In the following case, one column serves as X input and X output (A7:A135), the adjacent column holds the Y input data (B7:B135), and the program will dump the Y output into a separate column (D7:D135).

Even this wasn’t smooth enough for me, because I got tired of scrolling up and down to select long ranges. So I made another improvement. I made the program smart enough to know that if only one row had been selected, it should use the range of data below the selected row, stopping at the first empty cell.

Couldn’t be easier.
The last change I made was to have the utility store the last LOESS parameters in each worksheet. This way, if I run a particular analysis on each sheet in a workbook, I only have to activate that sheet, run the program, and the dialog is pre-populated with the settings I need. What a time-saver!
PTS LOESS Utility Demo
Here is a demo of the new utility. This analysis only took a few minutes. I started with some Global Temperature Anomaly data from NASA.

I ran the utility to generate a smoothed temperature anomaly curve.

Then ran the utility a few more times to see the effect of alpha (number of points in the moving linear regression) on the shape of the smoothed curve. The values of 42, 25, and 12 points correspond to alpha values of 0.33, 0.2, and 0.1.

The fewer points (i.e., smaller alpha) in an analysis, the more that the smoothed curve follows local variations in the data.
Download and Install the Utility
Click this PTS LOESS Smoothing Utility link to download the utility. The download is a simple Excel add-in. Save it to any convenient directory, then install it following the instructions in Installing an Excel Add-In or Installing an Add-In in Excel 2007.
Related Posts:
- LOESS Utility for Excel
- LOESS Utility – What the Buttons Mean
- LOESS Smoothing in Excel
- Deming Regression Utility
- Deming Regression
- Plotting Measured Data
- Announcing the Box and Whisker Chart Utility
- Label Last Point – Updated Add-In
- How to Edit Series Formulas
- Build an Excel Add-In 1 – Basic Routine
Posted: Thursday, October 8th, 2009 under Utilities.
Comments: 33
Comments
Comment from Bob
Time: Thursday, October 8, 2009, 12:20 pm
Way to go Jon.
Can’t wait to play with this.
Cheers,
Bob
Comment from Jerry Betz
Time: Thursday, October 8, 2009, 1:52 pm
Jon,
Ayuh, wicked gorgeous. Many thanks.
Jerry
Comment from Kurt
Time: Thursday, October 8, 2009, 3:00 pm
Hi Jon,
tried your LOESS-Tool, it installs a Menu
“PTS Charts” in my Excel 2003. clicking this
menu I have *two* indentical menu-entries
“LOESS”
Do you have any hint, what mistake I made?
thanks
Kurt
Comment from Jon Peltier
Time: Thursday, October 8, 2009, 4:18 pm
Kurt -
I don’t know why that happened. Before you quit Excel next time, go to View menu > Toolbars > Customize, then you can click on the PTS Charts menu and drag it off the menu bar to delete it. When you restart Excel, there should then be only one LOESS item.
Comment from Bob
Time: Thursday, October 8, 2009, 5:24 pm
Hi Jon,
The same thing happened to me in Office 2007 SP2 and all the patches.
Works fine. I also have the error bar utility you cooked up for Excel 2007. Happy camper.
Thanks for your hard work.
Cheers,
Bob
Comment from Jon Peltier
Time: Thursday, October 8, 2009, 8:03 pm
I’ll have to check that when I get a chance. At least the underlying utility works. At least for N > 4, that is, because one early tester found problems with very small N.
Comment from Kurt
Time: Friday, October 9, 2009, 12:27 pm
Jon,
the sugested procedure did not work for me.anyway it is just a “cosmetic” thing.
thanks for making this tool available for free
Kurt
Comment from Jon Peltier
Time: Friday, October 9, 2009, 2:23 pm
Kurt – I’ll dig into the code, see what might be messing up the interface.
Comment from Stephen Hughes
Time: Monday, October 12, 2009, 8:40 am
Is there a minimum version of Excel needed to run this? I’ve got 97 but get a ‘Compile Error in Hidden Module: M_LOESS’ error when I try to run it.
Any suggestions would be appreciated as I need a program to smooth out experimental data.
Comment from Jon Peltier
Time: Monday, October 12, 2009, 10:26 am
Stephen -
There was a major upgrade of VBA between Excel 97 and Excel 2000, and Macs have never gotten this upgrade. I would expect all of my code to work fine in Excel 2000 and later, but I would expect problems in Excel 97 and Mac Excel.
I don’t know if I even still have Excel 97 disks. If I do, maybe I’ll install it in a VM and see where the code falls down.
Comment from Stephen Hughes
Time: Monday, October 12, 2009, 10:33 am
Hi Jon,
Anything you can manage would be appreciated! Meanwhile I’ll try (again) to persuade my boss of the benefits of having a modern computer and up-to-date software!
Comment from Jon Peltier
Time: Monday, October 12, 2009, 10:43 am
The bare minimum is Excel 2000, but I’m sure that would be nearly impossible to find. The best version so far is Excel 2003, and even that isn’t easy to get your hands on.
Comment from Omair Noor
Time: Thursday, December 17, 2009, 8:14 pm
Hi Jon,
I have approximately 2400 data points per column. I tried to use your LOESS utility function, but i get the following error:
Run-time error ‘6′:
Overflow.
Any idea how should i be going about resolving this problem.
Thanking in anticipation.
Comment from Jon Peltier
Time: Thursday, December 17, 2009, 9:23 pm
Hi Omair -
I just did some fooling around, and had no problem with a data set with 5000 points. But I used some vanilla data, and yours may have some characteristic that makes it behave differently.
Is it possible to send me a sample? jonpeltier at-sign gmail period com.
Comment from tom
Time: Friday, January 29, 2010, 6:24 pm
I also experienced the runtime error.
Does it have to do with very small alpha values? I chose 20 points out of 2000.
bug #2:
The utility crashes from the menu if Excel is opened without any workbook.
bug #3:
Sometimes the dialog disappears when I am entering ranges in the text boxes.
idea:
Could you add something like a auto-fit for columns in the tool. I rather would like
to add full columns. The utility could start the range below the last text row and
would end the range before the first empty cell.
Comment from Jon Peltier
Time: Saturday, January 30, 2010, 10:36 am
Tom -
I don’t know what may have caused your first error. When it was reported last time, the problem was resolved by selecting just the data, not the entire column. I have added a slightly different dange auto-detect feature. Select the top cell in a range of data (the first data cell, not the header), and the utility will detect how far down the column the range extends.
I also don’t know about the second crash. Similar problems arise if not all upgrades and service packs have been installed.
How frequently does the dialog vanish? I see it once in a while, perhaps once every few dozen times I run the utility. I’ve just assumed it was related to the inherent instability of the RefEdit used to select the data ranges.
Comment from Nick
Time: Sunday, January 31, 2010, 8:22 pm
Hi Jon!
Nice little utility you have here! I’m utterly hopeless at math and I have a report involving normalizing microarray data, which requires me to apply LOESS to some data. I have trouble following your example of plotting LOESS for anomaly data for NASA. There are seven choices for entering the data and I don’t know which to choose! If you give a step by step instruction of how you used the add in, it would be great.
Please help the mathematically crippled biologist!
Comment from Jon Peltier
Time: Sunday, January 31, 2010, 10:41 pm
Nick – I’m preparing some documentation for a new article. Stay tuned.
Comment from K Hayhoe
Time: Thursday, April 15, 2010, 12:50 pm
any chance of a Mac versions for this and all your other great add-ins, pleeease?
Comment from Jon Peltier
Time: Thursday, April 15, 2010, 10:54 pm
Katharine -
The problem is that the version of Visual Basic in Mac Excel is a generation older than that in Windows Excel. There are some incompatibilities in the code because of this, and I have no means to test the older code.
Comment from david
Time: Wednesday, June 9, 2010, 11:37 am
Hi John,
I’m also attempting to run your loess app on some microarray data. I’m not certain if it’s working properly or not. My data consists of ~385k values (I can break into subsets of 2 30k and 2 150k).
I’m using the “3 adjacent column” button. When I hit it, the task manager says Excel is not responding. I’ve experienced this before with large data sets (this one is a smaller one for me), analysis and Excel. So I walk away — after a 1/2 hour it’s still not responding — with either the 30k, 150k or the 380k.
Should I wait longer??
Thanks
Excel 2007
Windows XP, x64 SP2
Dell Precision T5500, 2.27ghz, 24g RAM
Comment from Jon Peltier
Time: Wednesday, June 9, 2010, 5:44 pm
David -
That’s a lot of points. Is LOESS the right way to smooth your data set?
To get a handle on how long the analysis should take (on my computer) I started with some dummy data, and ran a series of LOESS analyses on it. I tested 10 to 10,000 points, with alpha=0.5. Times to calculate the LOESS fit are shown in this table. Each time shown is the average of six runs, except the time for 10,000 points is the average of three runs.

The slope of the right side of the curve is close to 2, which jibes with my faint recollection that the time to compute a regression is proportional to the number of points squared, and the LOESS fit is basically a set of regressions. If I extrapolate to 30k, 150k, and 380k points, I predict times of 31 minutes, 13 hours, and 86 hours.

Comment from DD
Time: Wednesday, June 23, 2010, 1:27 pm
Can’t get this to work at all in Excel 2007.
Steps to reproduce:
1. Go to Add-Ins and select “PTS Charts => LOESS”
2. Choose the button in the lower right (four columns)
3. Select 270 values for X
4. Select 270 next to it for Y
5. Do the same for output (other columns)
6. Type “0.25″ in the alpha box (Number of points updates to 67)
7. Press OK
Result is X output empty and Y output containing the same value repeated 270 times.
I can pick any data, and this is always the result.
Comment from Jon Peltier
Time: Wednesday, June 23, 2010, 2:24 pm
DD -
Does the X output range contain values? It should either contain values (e.g., if you want to specify different points than are in the X input range, or you should use an option that uses the X input range for input and output.
Comment from david
Time: Tuesday, June 29, 2010, 4:46 pm
Hi John,
Thanks for your excellent reply, and please accept my apology for my delayed response.
Yes, it’s many data points, but it’s actually a small data set for us (thus the use of Excel). I’ve some data sets that contain ~3 million data points, and yes a standard first pass is to do LOESS.
It would be nice to have a LOESS function in Excel — it’s more familiar and comfortable to some folks around here — rather than R. That’s why we were excited to come upon your solution. We do have some other ~20-30K datasets (and the above that I mentioned), but trusting Excel and XP to 30 minutes (or longer) of stable computing does seem to be living to close to edge for me!
Thanks for your excellent and informative blog.
David
Comment from Jon Peltier
Time: Tuesday, June 29, 2010, 7:56 pm
David -
Even though there are 385k input data points, you could output LOESS values for many fewer points, even just a thousand. This should cut down on most of the calculations.
Comment from DanF
Time: Saturday, July 10, 2010, 5:42 pm
I particularly like the remember last use feature.
However, after a half day of LOESSing all over the place, I can’t get the utility to work any longer. When I click the x input range, I get the normal selection box. When I try to put a y range in or click any of the input buttons or text boxes, it goes poof and the LOESS window closes. When I close Excel (thinking that a good old fashioned close it-open it sequence is the cure-all for just about anything), I get the password requests. I read your REF CTL commentary, but I didn’t see what I might have done to cause LOESS to go on strike.
Comment from Jon Peltier
Time: Sunday, July 11, 2010, 8:17 am
Dan -
I have not yet changed the LOESS utility to use textbox/inputbox instead of the flaky RefEdit. When it works, the RefEdit is nice, but when it doesn’t work, it is horrible.
Comment from Thomas Rand-Nash
Time: Thursday, July 22, 2010, 11:40 am
First and foremost, THANK YOU!! This is coming in incredibly handy for smoothing out an unruly data horde! My question has to do with an “PTS_LOESS Password” box which opened after I quit excel. Is there a password I should know?
Thanks
Comment from Jon Peltier
Time: Thursday, July 22, 2010, 1:13 pm
Thomas -
There’s no password you should know. Make sure you’ve installed all service packs and updates for the version of Office you’re running. If that doesn’t help, there’s some Registry diving that might help.
Comment from Nate
Time: Tuesday, August 3, 2010, 12:52 pm
Jon, I’m looking forward to trying this tool, but after installing it, I get the following error message when opening Excel 2007: “object library invalid or contains references to object definitions that could not be found”
Any ideas on what might cause that?
Comment from Nate
Time: Wednesday, August 4, 2010, 7:48 am
Jon, regarding my comment on the missing object library: I determined that this was being caused by a corrupted installation and not your add-in. I reinstalled E2007 and have it working now.
Comment from Jon Peltier
Time: Wednesday, August 4, 2010, 10:35 am
Nate -
Thanks for following up and reporting the fix. The error is more than likely due to the RefEdit issues I’ve discussed (Using RefEdit Controls in Excel Dialogs) and finally programmed away (Alternative to Excel’s Flaky RefEdit Control) in my commercial add-ins, but not the simple ones available with my blog posts.



















Write a comment
I welcome comments from my readers. If you have an opinion on this post, if you have a question or if there is anything to add, I want to hear from you. Whether you agree or disagree, please join the discussion.
If you want to include an image in your comment, post it on your own site or on one of the many free image sharing sites, and include a link in your comment. I'll download your image and insert the necessary html to display the image inline.
Read the PTS Blog Comment Policy.