Peltier Tech Blog

Excel Charting Utilities | Custom Solutions | Training | Charts and Tutorials | PTS Blog

 

Main menu:

 
Peltier Tech
Chart Utilities

PTS Waterfall Chart Utility
PTS Cluster-Stack Column Chart Utility
PTS Box and Whisker Chart Utility
PTS Marimekko Chart Utility
PTS Dot Plot Utility
PTS Cascade Chart Utility

Subscribe

Site search

Subscribe

Site search


Recent Posts

Recently Commented

Popular Posts

Archive


 

Categories


 

Buy me a coffee

If this topic or the whole blog has been helpful, support further development by treating me to a cup of coffee.


Privacy Policy

Creative Commons License
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

LOESS Utility – Awesome Update

by Jon Peltier
Thursday, October 8th, 2009
Peltier Technical Services, Inc., Copyright © 2010.
Licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

I talked about LOESS smoothing in LOESS Smoothing in Excel, where I showed my improved VBA function for calculating smoothed data. I introduced an improved LOESS Smoothing utility in LOESS Utility for Excel. Since I use the utility frequently, I encountered many things about it that I wanted to change. Since I am the developer of the utility, I have actually been able to make these changes. And after a few months of use and a few hour-long sessions aimed at making it do what I want it to do, I’ve developed this utility into something that is, in the local vernacular, “wicked awesome”. If you’ve never seen “Good Will Hunting”, that means “way cool”.

What makes this utility so great, you ask? The calculations were already perfectly adequate, and I didn’t change them at all. But I’ll show the new dialog, so you can see what has changed.

New and Improved LOESS Utility Dialog

First, the previous interface for selecting input and output ranges was frustratingly restrictive. So I made the range selection function much more flexible. The colorful buttons on the left indicate the built-in options. In the most elementary case, all four entries (X input, Y input, X output, and Y output) are separate ranges, which can even be located on separate worksheets.

New and Improved LOESS Utility Dialog

The options make range selection easier if some columns happen to be adjacent to each other, or if the same X values are to be used for both input and output. The range selection (Ref Edit) boxes and related labels update according to which option has been selected.

When the X input range has been selected, the program determines Npts, the number of points that should be used for the selected value of alpha. When the user changes alpha or Npts, the other parameter updates.

In the following case, one column serves as X input and X output (A7:A135), the adjacent column holds the Y input data (B7:B135), and the program will dump the Y output into a separate column (D7:D135).

New and Improved LOESS Utility Dialog

Even this wasn’t smooth enough for me, because I got tired of scrolling up and down to select long ranges. So I made another improvement. I made the program smart enough to know that if only one row had been selected, it should use the range of data below the selected row, stopping at the first empty cell.

New and Improved LOESS Utility Dialog

Couldn’t be easier.

The last change I made was to have the utility store the last LOESS parameters in each worksheet. This way, if I run a particular analysis on each sheet in a workbook, I only have to activate that sheet, run the program, and the dialog is pre-populated with the settings I need. What a time-saver!

PTS LOESS Utility Demo

Here is a demo of the new utility. This analysis only took a few minutes. I started with some Global Temperature Anomaly data from NASA.

Measured and LOESS-Smoothed Temperature Anomaly Data

I ran the utility to generate a smoothed temperature anomaly curve.

Measured and LOESS-Smoothed Temperature Anomaly Data

Then ran the utility a few more times to see the effect of alpha (number of points in the moving linear regression) on the shape of the smoothed curve. The values of 42, 25, and 12 points correspond to alpha values of 0.33, 0.2, and 0.1.

Measured and LOESS-Smoothed Temperature Anomaly Data

The fewer points (i.e., smaller alpha) in an analysis, the more that the smoothed curve follows local variations in the data.

Download and Install the Utility

Click this PTS LOESS Smoothing Utility link to download the utility. The download is a simple Excel add-in. Save it to any convenient directory, then install it following the instructions in Installing an Excel Add-In or Installing an Add-In in Excel 2007.

Related Posts:

Bookmark and share this entry:
  • Twitter
  • Digg
  • Facebook
  • LinkedIn
  • del.icio.us
  • Technorati
  • StumbleUpon
  • Google Bookmarks
  • Reddit
  • MySpace
  • Slashdot
  • Yahoo! Buzz
  • Tumblr

Learn how to create Excel dashboards.

Comments


Comment from Bob
Time: Thursday, October 8, 2009, 12:20 pm

Way to go Jon.

Can’t wait to play with this.

Cheers,

Bob


Comment from Jerry Betz
Time: Thursday, October 8, 2009, 1:52 pm

Jon,

Ayuh, wicked gorgeous. Many thanks.

Jerry


Comment from Kurt
Time: Thursday, October 8, 2009, 3:00 pm

Hi Jon,

tried your LOESS-Tool, it installs a Menu
“PTS Charts” in my Excel 2003. clicking this
menu I have *two* indentical menu-entries
“LOESS”

Do you have any hint, what mistake I made?

thanks

Kurt


Comment from Jon Peltier
Time: Thursday, October 8, 2009, 4:18 pm

Kurt -

I don’t know why that happened. Before you quit Excel next time, go to View menu > Toolbars > Customize, then you can click on the PTS Charts menu and drag it off the menu bar to delete it. When you restart Excel, there should then be only one LOESS item.


Comment from Bob
Time: Thursday, October 8, 2009, 5:24 pm

Hi Jon,

The same thing happened to me in Office 2007 SP2 and all the patches.

Works fine. I also have the error bar utility you cooked up for Excel 2007. Happy camper.

Thanks for your hard work.

Cheers,

Bob


Comment from Jon Peltier
Time: Thursday, October 8, 2009, 8:03 pm

I’ll have to check that when I get a chance. At least the underlying utility works. At least for N > 4, that is, because one early tester found problems with very small N.


Comment from Kurt
Time: Friday, October 9, 2009, 12:27 pm

Jon,

the sugested procedure did not work for me.anyway it is just a “cosmetic” thing.
thanks for making this tool available for free

Kurt


Comment from Jon Peltier
Time: Friday, October 9, 2009, 2:23 pm

Kurt – I’ll dig into the code, see what might be messing up the interface.


Comment from Stephen Hughes
Time: Monday, October 12, 2009, 8:40 am

Is there a minimum version of Excel needed to run this? I’ve got 97 but get a ‘Compile Error in Hidden Module: M_LOESS’ error when I try to run it.

Any suggestions would be appreciated as I need a program to smooth out experimental data.


Comment from Jon Peltier
Time: Monday, October 12, 2009, 10:26 am

Stephen -

There was a major upgrade of VBA between Excel 97 and Excel 2000, and Macs have never gotten this upgrade. I would expect all of my code to work fine in Excel 2000 and later, but I would expect problems in Excel 97 and Mac Excel.

I don’t know if I even still have Excel 97 disks. If I do, maybe I’ll install it in a VM and see where the code falls down.


Comment from Stephen Hughes
Time: Monday, October 12, 2009, 10:33 am

Hi Jon,

Anything you can manage would be appreciated! Meanwhile I’ll try (again) to persuade my boss of the benefits of having a modern computer and up-to-date software!


Comment from Jon Peltier
Time: Monday, October 12, 2009, 10:43 am

The bare minimum is Excel 2000, but I’m sure that would be nearly impossible to find. The best version so far is Excel 2003, and even that isn’t easy to get your hands on.


Comment from Omair Noor
Time: Thursday, December 17, 2009, 8:14 pm

Hi Jon,

I have approximately 2400 data points per column. I tried to use your LOESS utility function, but i get the following error:

Run-time error ‘6′:

Overflow.

Any idea how should i be going about resolving this problem.

Thanking in anticipation.


Comment from Jon Peltier
Time: Thursday, December 17, 2009, 9:23 pm

Hi Omair -

I just did some fooling around, and had no problem with a data set with 5000 points. But I used some vanilla data, and yours may have some characteristic that makes it behave differently.

Is it possible to send me a sample? jonpeltier at-sign gmail period com.


Comment from tom
Time: Friday, January 29, 2010, 6:24 pm

I also experienced the runtime error.
Does it have to do with very small alpha values? I chose 20 points out of 2000.

bug #2:
The utility crashes from the menu if Excel is opened without any workbook.

bug #3:
Sometimes the dialog disappears when I am entering ranges in the text boxes.

idea:
Could you add something like a auto-fit for columns in the tool. I rather would like
to add full columns. The utility could start the range below the last text row and
would end the range before the first empty cell.


Comment from Jon Peltier
Time: Saturday, January 30, 2010, 10:36 am

Tom -

I don’t know what may have caused your first error. When it was reported last time, the problem was resolved by selecting just the data, not the entire column. I have added a slightly different dange auto-detect feature. Select the top cell in a range of data (the first data cell, not the header), and the utility will detect how far down the column the range extends.

I also don’t know about the second crash. Similar problems arise if not all upgrades and service packs have been installed.

How frequently does the dialog vanish? I see it once in a while, perhaps once every few dozen times I run the utility. I’ve just assumed it was related to the inherent instability of the RefEdit used to select the data ranges.


Comment from Nick
Time: Sunday, January 31, 2010, 8:22 pm

Hi Jon!
Nice little utility you have here! I’m utterly hopeless at math and I have a report involving normalizing microarray data, which requires me to apply LOESS to some data. I have trouble following your example of plotting LOESS for anomaly data for NASA. There are seven choices for entering the data and I don’t know which to choose! If you give a step by step instruction of how you used the add in, it would be great.
Please help the mathematically crippled biologist!


Comment from Jon Peltier
Time: Sunday, January 31, 2010, 10:41 pm

Nick – I’m preparing some documentation for a new article. Stay tuned.

Write a comment

I welcome comments from my readers. If you have an opinion on this post, if you have a question or if there is anything to add, I want to hear from you. Whether you agree or disagree, please join the discussion.

If you want to include an image in your comment, post it on your own site or on one of the many free image sharing sites, and include a link in your comment. I'll download your image and insert the necessary html to display the image inline.

Read the PTS Blog Comment Policy.





Subscribe without commenting

PTS Waterfall Chart Utility PTS Cluster-Stack Column Chart Utility PTS Box and Whisker Chart Utility PTS Marimekko Chart Utility PTS Dot Plot Utility PTS Cascade Chart Utility

Create Excel dashboards quickly with Plug-N-Play reports.