Oh You False Empty Cell

I was comparing two worksheets by creating formulas off to the side of one sheet. These returned True if the corresponding cells in the two sheets matched, False if they didn’t. I converted the Trues and Falses to values, and replaced the Trues with nothing. Then I threw some conditional formatting on the data in one of the sheets. The formatting referred to the area with empty cells and Falses. The condition was for cells offset from Falses to be shaded.

I expected a smattering of color. What I got was a solid swath of light orange. Kind of like this, but on three separate sheets:

I assumed that I’d screwed up the conditional formatting formula, a not unreasonable premise. However, after a couple of minutes I was sure I hadn’t, and it was then I realized that empty cells equate to False. I’m sure everybody else already knows this, so I figured I’d better dig a little deeper in case it comes up at a party or something.

Next I checked if cells with =”" equate to False. They don’t. Neither do ones with 0, which I thought they might, since False multiplied by 1 equals 0. Here’s the visual summary, this time with a lambent blue for False.

In VBA the results are similar, except that 0 = vbEmpty. I have no idea what that means. I’m pretty sure Dick had a post on this, called “testing for empty cells,” but it seems to have gone down in the great DDOE server crash of 2011.

yoursumbuddy - Doug Glancy's Excel Site

Data Normalizer – the SQL

In Data Normalizer I showed you how I normalize worksheet data using arrays and For/Next loops. I’ve been doing a fair amount of SQL in VBA lately, and thought I’d rewrite the code using that approach.

A little searching revealed the T-SQL/SQL Server “Unpivot” command, which normalizes your data and sets the new field names all in one swell foop. It’s not available in Access/Jet SQL though, so can’t be used on Excel. Instead, the preferred method is to use a series of Selects that pick one normalizing column at a time (along with the repeating columns) and Unions them together.

I tried ADO first, using the method of SaveCopyAs’ing the workbook-to-be-normalized in order to avoid the ADO memory leak. ADO was way slower than DAO, something like four times slower with 3000 records of 16 columns. So I went with DAO, which still takes about twice as long as the array method. Turning the ADO to DAO was easy, especially with this concise sample from XL-Dennis.

As Jeff Weir pointed out in a comment, this does require a reference (Tools>References) to the Microsoft DAO 3.5 Object Library. I’ve been switching some code over to late binding, but there doesn’t seem to be much enthusiasm for this with DAO. I’m not sure if that’s because it’s so pervasive and well-established, or for some other reason.

The core logic of the routine is pretty simple. In pseudo-English:

Without further ado (heh heh) here’s the routine.

If you break after strSql is created, it looks like this:

Call it like this:

One pitfall of this SQL version is the mixed data-type issue with the Excel ISAM driver. In this example it converts all the home run counts from numbers to text because of the blanks in the data.

All in all, I think the array approach is better than SQL for this use. The core skill of creating SQL in VBA is a valuable one though, and one I’m glad to be developing.

I updated the Data Normalizer .xls to include this code.

yoursumbuddy - Doug Glancy's Excel Site

Create Pivot Table Named Ranges

I need to calculate percentiles from subsets of data in a pivot table. In order to refer to pivot table fields, it sure would be nice if they had dynamic named ranges. So I wrote some code to create pivot table named ranges.

Programming pivot tables is fun. The extensive object model is a VBA wonderland with treats around every turn. There are great web sites out there with excellent pivot table coding samples – Contextures leaps to mind. In terms of identifying PivotFields, DataFields and other pivot table ranges, Jon Peltier wrote a superb post in 2009 that’s still generating discussion.

My code is pretty simple. It cycles through the data fields, and any other visible fields, in the specified pivot table and adds a named range for each one to the pivot table’s worksheet:

The PivotField.Orientation property has five enumerated constants that tell you what type of field it is – xlDataField, xlRowField, etc. The For/Next loop skips over the ones that come up xlHidden and processes the rest. Strangely, even though there’s a xlDataField type, and even though I can refer to pvt.PivotFields(“Sum of Home Runs”), the data fields don’t actually show up when cycling through the PivotFields. Instead, to get those fields the code first cycles through the pivot table’s DataFields collection.

When calling the AddNamedRange routine for a DataField, the codes passes its SourceName, not the Name. So in this example, the new name will include “Home Runs,” not “Sum of Home Runs.” You may want to pass the Name instead.

This next routine does what it says and clears out the previous range names associated with the pivot table. It’s not fool-proof. For example, if the pivot table name was changed, it won’t find the range names. I should probably use the pivot table’s Tag property to store names that won’t get changed:

The routine below adds the worksheet-level names to the pivot table’s sheet. It calls a function that replaces spaces and other characters that aren’t allowed in range names (code at the end of the post). It also adds a “_” at the beginning of the name to hopefully avoid illegal names like “A1″:

To automate this stuff, put the code in a regular module and call RefreshPivotNamedRanges from a PivotTableUpdate event. The names will be regenerated each time the pivot table is refreshed, either manually or when you drag a field, or however.

So now my Percentile array formula can find the value for the selected year and percentile:

Here’s the code to get the legal range names. You’ll need to set a reference to Microsoft VBScript Regular Expressions (at least if you’re an early binder):

This has undergone a massive .5 days of testing, so I can guarantee there’s glitches. But if you’d like to give it a spin, here you go. It’s an Excel 2007/10 file as earlier versions don’t support the “Repeat All Item Labels” pivot setting that I rely on for the Percentile array formula. Other than that, it works just as well in Excel 2003.

yoursumbuddy - Doug Glancy's Excel Site

Prompt to Save Addin

I’m pretty good about saving my work, and probably hit Ctrl-S a couple hundred times a day. And of course, as long as things don’t crash, Excel makes it hard to lose your work. One exception is addins, which don’t trigger a save prompt when you close them after making changes, at least when the IsAddin property is True. So I have a routine in my most-used addins that reminds me to save them. But I don’t have it in all of them. The other day this bit me, and I lost 10 minutes of work on an xlam. I decided to generalize my prompt to save addins and put it in an application-level event in my main utility addin. (These decisions come easy; what’s more fun than building a new tool?) This way I’m prompted to save any time I close an addin that I’ve changed.

If you’ve never used application-level events, Chip Pearson’s site has some good information. Okay, here’s how you can add this code to your favorite utility addin (personal.xls will do nicely).

Create a Class called “clsApplication” and paste this code into it:

Create a global variable to hold the class instance. At the top of a regular code module (before any procedures) put this line:

(I like to put all my global variables like the one above in a single module, called modGlobals.)

In the ThisWorkbook WorkbookOpen event for your utility addin, put this code:

One problem is that when an addin is saved with more than one instance of Excel open, it gets saved to a new location (maybe the folder of ActiveWorkbook?). So I added code to the BeforeClose event to cancel the save if that’s true. Here’s the function that does the checking:

One last thing to do is add code to my global error handler that re-instantiates the cApplication.Class and its App property if they’ve gotten lost, which can easily happen during debugging.

yoursumbuddy - Doug Glancy's Excel Site

Solving the NPR Sunday Puzzle – #3

Every time I hear Will Shortz say “name a country of the world” my heart beats a little faster, cause I know there’s probably an array-formula post in it. This week was no exception:

“Name an article of clothing that contains three consecutive letters of the alphabet consecutively in the word. For example, “canopy” contains the consecutive letters N-O-P. This article of clothing is often worn in a country whose name also contains three consecutive letters of the alphabet together. What is the clothing article, and what is the country?”

This spreadsheet is live. Double-clicking in a cell in the 2nd column shows the formula. You can then drag the lower-right fill handle to see the whole gnarly thing.

Countries are a lot more list-friendly, so I solved that part first. There are two types of solutions above: an array formula in the 2nd column, and a conditional formatting solution in the 3rd to 40-somethingth column. The 2nd solution is the way I’d do it for serious work, but the array formula is more fun:

This formula says to compare every letter in the country name to the next two letters. If the numeric code for a letter is one less than the letter after it, and two less than the letter after that, return those three letters. The reason the 2nd column is mostly blank is because surprisingly few country names have three consecutive letters. (Or my formula sucks.)

Here’s how it works. Let’s assume the formula refers to Albania, the 3rd country in the list:

LEN returns the length of the country’s name.

INDIRECT(“1:”& LEN($A4)-2) evaluates to (1:5).

ROW(INDIRECT(“1:”& LEN($A4)-2)) evaluates to {1;2;3;4;5}. This tells the array formula which characters in “Albania” to evaluate against their respective two next characters. Using ROW this way is very handy in array formulas.

CODE(MID(LOWER($A4),ROW(INDIRECT(“1:”& LEN($A4)-2)),1)) resolves to {97;108;98;97;110}. In other words, the numeric values of the first five letters of Albania are 97, 108, etc. CODE is the function that returns the values. LOWER converts each letter to lower case, otherwise, for example, a lowercase “b” wouldn’t be seen as following an uppercase “A”. The MID piece tells the array formula to parse each of the first 5 letters in the word.

Okay, now we’re getting close. This bit:
CODE(MID(LOWER($A4),ROW(INDIRECT(“1:”& LEN($A4)-2)),1))+1=
CODE(MID(LOWER($A4),ROW(INDIRECT(“2:”& LEN($A4)-1)),1)) resolves to
{98;109;99;98;111}={108;98;97;110;105}, which in turn resolves to {FALSE;FALSE;FALSE;FALSE;FALSE}. In other words, one added to the code for the first five letters in “Albania” doesn’t ever equal the code for their respective next letters. So there’s not even two consecutive letters. What a country! The next two lines in the formula repeat the comparison, this time with the 2nd through 7th letters.

The 2nd line, which starts with MID, and the 2nd to last line say that if the whole mess is true, return the three letters that start at that position.

The IFERROR function at the beginning and the last line simply say that if the formula errors (doesn’t find three consecutive letters) put a blank in the cell. IFERROR is available in Excel 2007 and 2010. I like it very much as they keep formulas like this from being twice as ridiculously long.

Much of the evaluation above was done using the F9 key. Selecting part of a formula and then pressing F9 evaluates that piece of the formula, a very useful feature.

There are only two countries that meet the criteria, Afghanistan and Tuvalu. I figured it was probably the former and soon enough realized the article of clothing was “hijab.”

If you’ve made it this far, thanks! Oh yeah, the multiple cell and conditional formatting way works similarly to the array formula, only by breaking it up it’s a lot easier.

If anybody has a better array formula, please share.

yoursumbuddy - Doug Glancy's Excel Site

Copy Table Data While Not Breaking References

I’ve mentioned before I’m a big fan of tables in Excel 2010. One way I use them is in models, where each table represents a different scenario. The models let users create new scenarios, based either on an existing one or a blank template. Each version is stored in a table in its own worksheet. These tables always have the same fields (columns) but the values in the variable fields are different. The number of rows can differ from table to table.

I often want to copy, in VBA, the contents from a “source” to a “target” table. If I just copy the whole thing, the target table will be overwritten and renamed – something like “tblSource1″ (adding a “1″ to the source table name). That breaks any formulas referring to “tblTarget.” They’ll show a #REF error because they can’t find “tblTarget.” So I need code that copies the table data from tblSource to tblTarget without completely replacing tblTarget.

I’ve been writing code on a case-by-case basis, but thought that I’d generalize it a bit more. In addition to keeping the table’s identity intact, it should copy the source’s totals row if there is one, and turn off the target’s total row if there isn’t. The number of rows should increase or decrease to match the source. And, although I’ve only ever copied tables as values, I want the option to copy formulas.

I thought about dealing with a different number of columns but, at least in my uses so far, that shouldn’t happen. If I ever do try to accommodate models with changing numbers of fields, I think I’d do some testing before ever calling this code, and adjust the headers in another procedure.

So here’s what I came up with:

One thing I learned is that there are two Resizes in a table (listobject). The first type, the Range property, was familiar, e.g.,

… which yields a range object whose address is A1:A20.

The second is the Listobject.Resize method, which allows you to modify a table’s range, e.g.,

which will change loTarget’s range to A1:F20.

Both of these types of Resizes are used in the code above, in the same line, happily enough.

yoursumbuddy - Doug Glancy's Excel Site

Data Normalizer

Sometimes I get data like this…

that needs to be like this…

The goal here is to roll up all the home runs into one, much longer, column. The data will then be pivot-worthy.

Generally, I need to keep one or more leftmost column headers, in this case “League” and “Year.” I need a new column to describe the rolled-up category (“Team”) and one for the data itself (“Home Runs”). I’ve written code a couple of times to handle specific cases and thought I’d try to generalize it. Here’s the result:

You’d call it like this:

It runs pretty fast. The sample sheet above – 109 years of data by 16 teams – completes instantly. 3,000 rows completes in a couple of seconds.

If I also run the routine on some American League data and put all the new rows in one sheet (with the same column headers) I can generate a pivot table that looks like this, which I couldn’t have done with the original data:

You can download a zip file with a .xls workbook that contains the data and code. Just click on the “normalize” button.

yoursumbuddy - Doug Glancy's Excel Site

Anything

I think about this blog a lot. I’m always pondering the next post, a challenge given the multitude of great ones that have gone before. I’ve tinkered with WordPress quite a bit, which is fun. Using it inspired me to create this gravatar …

… which I like to call the “smiling sigma.”

And I check my stats (such as they are) a lot. One of my favorites is the list of search terms bringing people to this site.

Search Terms

The two most common have to do with pivot tables and conditional formatting. After that there are a few concerning the NPR Sunday Puzzle, and I’m proud to note that when I google “NPR Sunday Puzzle,” this site is right up there in the results.

Here’s some of my favorites:

“word puzzle first letter hero villain world capital blog” – definitely an NPR Sunday Puzzle solver

“toolbar lengkap excel 2003” – I understand that “lengkap” is Indonesian for “complete”

“goofy code” – I rank very high in the results on this, hopefully only because there’s a post here with that name

My Second Most Favorite Search Term

“sum buddy i use to no” – Now there’s a person after my own heart! What were they looking for? And how do you “use somebody to no” anyways? It sounds handy.

Sadly, I’ll never no.

My Most Favorite Search Term

“Anything”

Yup, according to AwStats, somebody googled “anything” and ended up here. I’m not completely convinced this happened, but it’s there on my screen, so I’m going with it.

yoursumbuddy - Doug Glancy's Excel Site

A Flexible VBA Chooser Form

Fairly often in VBA code I need to offer the user a list and have them make a choice, like picking which open workbook to do something to. I created a function and a userform to handle these situations. (Around the house, I call the form “ChooserForm” but it’s given name is “frmChooser.”) The function takes an array of choices and a caption as its arguments. The function loads frmChooser and passes it the string array and the caption. When the user makes a choice and clicks OK the function returns the choice to the calling routine.

Let’s look at how it works, starting from the inside out (by which I mean with the userform):

The frmChooser UserForm

The form has three custom properties. The first, Let ChoiceList, assigns the array of choices to the form’s module-level variable, mChoiceList(). On form activation the combobox cboChooser’s list is filled with the mChoiceList array.

The second property, Get ChoiceValue, is the currently selected value of the combobox. The function will “get” this, after the OK button is clicked, to determine the user’s choice.

The third property, Get ClosedWithOk tells the calling function whether the user hit the OK button. If it’s True then the function will do its processing. If it’s false, then the user hit the Cancel button or the “X,” and we’ll skip the processing.

The Function code

The function creates an instance of frmChooser, called “ufChooser,” passes the Caption and ChoiceList properties and shows the form. After the .Show command, processing passes into the form and the code shown in the previous section. Processing returns to the function when the form is hidden, by either the OK or Cancel button’s click event. The function then checks the form’s ClosedWithOK property. If it’s true the function returns the form’s ChoiceValue property – the value selected in the combobox – to the calling routine.

You may have noticed the question “why is this necessary?” I can’t just pass strChoices() straight into the frmChooser instance. It causes a runtime “internal error.” Instead I have to declare a second string array strChoicesToPass() and copy the first array to it. If anybody can explain why, please share! (I think I could pass a variant straight through, but I don’t.)

The general form of this function’s code, and that of the userform, is from the venerable Professional Excel Development.

Using the Function

Now that we’ve got the function and the form, let’s choose something! I’ve got some code below that lists all the visible fields in a pivot table. When one is picked, the data range for the field is highlighted, along with the source column in the table, and the fields source name is displayed:

This type of code can be useful when the PivotField names have been changed drastically from their underlying SourceNames, especially if the SourceNames are cryptic, similar, and there’s lots of them. In the picture below the SourceNames in the table were “Field 1″, “Field 2″, etc., but were changed to meaningful names like “Continent” in the pivot table.

Here’s the sample workbook for your downloading pleasure.

yoursumbuddy - Doug Glancy's Excel Site

The Best Min Function, I Think

A few weeks ago I wrote a Min array function to determine the minimum of a subset of items in a table. It was not the best Min function, in fact it didn’t work. I’d based it on the Max function I’d written moments earlier – a rookie mistake – and it resulted in zero when it shouldn’t have. I fixed it, but the fix was ugly. Then I realized I could use an If statement in an array formula, which helped a lot. Then I read this informative and lively post.

I’ll summarize what I learned, and propose my own Best Min Function, using the interactive workbook below.

The Workbook

The table on the left contains record high and low Fahrenheit temperatures from various cities around the world, listed by continent. (Column B contains the countries. You can unhide it.)

The formulas are in the table in the top right. They all look for the lowest of the record lows by continent.

Below the formulas are a couple of cells which trigger a couple of pitfalls of the Min function. Below that are text versions of the formulas.

The Formulas

The first formula, “multiplication,” is my original failure. The conditions (record type and continent) and the record lows are all multiplied:

As described in the post linked above, this only works for Min if at least one value is equal to or less than zero. Multiplying arrays of conditions like this will always yield some zeros – whenever all the conditions aren’t met – and so the minimum can never be higher than zero.

This list low temps is illustrates the problem well. For some continents the record low is negative, but for others it’s positive. You can see that this formula fails for continents where the record low is positive. For example Africa should show 30 degrees, but it shows zero.

My fix was to force all the numbers in the table to be negative, by subtracting the maximum record temperature from each number, find the minimum, and then add the maximum back. This eliminates the zero problem…

… but it’s ugly. And anyways, both multiplication versions fail if there is any stray text in your list. I noticed this because Florence had “NA’s” for both high and low records. The multiplication chokes on the text value, just as it would on 2 * “NA”, and the formula fails. You can see this by changing the “Has Text” in the “Potential Pitfalls” section to True. Both formulas yield #VALUE.

So, as faithful readers of DDOE already knew – at least if their memories are better than mine – If statements are the way to go. The “nested If” and the “Elias one-if” versions above work because they return an array filled either with temperatures that meet all the criteria, or Falses. The Min function ignores the Falses, and so returns the correct minimum.

Now for the second pitfall: Empty cells in your list are a problem because an IF statement cannot return a null. It converts blank cells to zeroes which, you guessed it, become the result of the Min function if there’s no lower number.

You can see this by changing the “Has Blanks” in the “Potential Pitfalls” section to True and following the instructions. You’ll see that the “nested if” and “Elias one-if” both fail for Africa, and will for any warm continent with a blank cell.

The proposed “Best Min” solves this problem by adding a criteria of non-blank cells. (It also eliminates the “>0″ check of the multiplied conditions from Elias’s version as it seems to evaluate to the same thing.)

There is a runner-up at the end of the list: a GetPivotData formula looking into a pivot table, or perhaps just a pivot table itself (you’ll see it if you scroll down). It might be my first choice if if didn’t require refreshing.

One last thought

I just realized that the Max function is subject to the exact same false-zero-minimum problems if all values are equal to or below zero, the inverse of the Min issue.

yoursumbuddy - Doug Glancy's Excel Site