usage data tells you what happened, not why it happened

A blog post that ends with “Microsoft UI has officially entered the realm of self-parody” is going to get quite a lot of mileage.  I lost count of how many times I saw it go by on my twitter stream.  Laurie Voss posted a response to Steven Sinofsky’s MSDN blog post about the improvements to Windows Explorer that are coming in Windows 8.

Voss takes a look at the data that Sinofsky posted about the usage of various commands in Windows Explorer, and is less than impressed at how they’ve applied this data to the new design of Windows Explorer.  He has two major complaints.  First, he complains that even by Microsoft’s own data, many of the commands that are elevated in the new Windows Explorer design are ones that aren’t commonly used.  Second, he complains that Microsoft’s data says that the menu bar within the Windows Explorer is very infrequently used, so what’s the point of doing it at all?

Both of these complaints are a very common misuse of usage data.  Usage data only tells you what happened in the past.  It doesn’t tell you why it happened, nor does it tell you what will happen in the future.  Furthermore, usage data can often not be broken down very far, so we don’t know what types of users and usages it represents.  I’ve written about this before in my blog post about the usage fallacy:

Usage data is directional. It doesn’t tell you what action to take, it tells you that there might be an action to take.

This certainly applies here.  For example, Voss is upset that few users use the menu bar, instead using contextual menus.  Do we know why they don’t use the menu bar?  Are there commands that they’re more likely to use in the menu bar?  Is there a discoverability problem (that is, are there commands that users would like to use but can’t find them)?  None of these are questions that can be answered by usage data.  To answer these questions, you need to use other research methodologies.

The true irony of Voss’s lack of understanding of how to appropriately apply usage data is found in another recent blog post of his about the need for statisticians, in which he says the following:

The whole web industry is accumulating vast quantities of data and storing it, magpie-like, as if it has intrinsic value, aided by ever-falling prices for storage. But the data isn’t valuable. It doesn’t mean anything until somebody who knows what they’re doing looks at it, sifts through it, and produces a tool that lets others use it to draw valid and useful conclusions.

He’s right: data isn’t valuable until someone who knows what they’re doing looks at it and helps draw valid and useful conclusions.  It’s always amusing when bloggers officially enter the realm of self-parody.

being female online is being a target online

I’ve been online since 1993.  Through it all, I’ve used my real name, and I’ve never tried to hide my gender.  I’ve taken part in various online communities, mostly geek and music.  One of my earliest lessons in being online is that being female makes you a target for threats.  I’ve received rape threats, most notably after I’ve mentioned that I financially support RAINN.  I’ve received death threats.  The threat that bothered me the most was a threat from a rather unhinged music fan against my beloved cat, since said music fan lived in the same city that I did and so I felt like there was a chance of him actually trying to do something.

Threats are the reason why I moderate comments here.  When I first started blogging, I didn’t bother.  Spam was my only concern, when I first started blogging, and modern spam filters do a reasonable job of managing it.  I started moderating comments when I got one particular commenter who saw it as his mission to threaten to rape me.  At first, moderation didn’t change his comments, but it did keep an echo chamber from forming.  Finally, he got bored, and stopped posting.  Maybe he’s still reading, I don’t know, but he’s not making the threats any longer.  I still get threats on occasion, most often if I’ve seen an uptick in traffic because I’ve gotten linked elsewhere.

Sadly, it came as no surprise to me today when I saw another female blogger say that she’s gone into hiding as a result of death threats:

I got a death threat.

I got another death threat.

And then I got one telling me that if I went to a popular blogger’s conference in November, they would find me, they would kill me, and they would kill everyone around me.

[…]

As a reasonably prominent female face on the internet, I always knew the day would come when it would get very, very ugly.

I don’t know why being female online sparks this kind of response.  I don’t know why there’s an echo chamber that creates an ever-escalating environment.

And I don’t really know what to do about it, other than continue to be myself.  Yes, I’m female.  Yes, I’m a software engineer.  Yes, I have opinions, and I share them unapologetically.

every minute you talk is a minute your user isn’t giving you data

The summer intern program at VMware is robust.  My user experience team gets a few summer interns every year, and sets them loose on several forward-thinking projects.  They’ve got 10-12 weeks to go through the whole design process for their project.

This summer, I worked with one of our interns towards the end of her project.  She wanted to collect some user feedback about her work, and wasn’t sure how to go about it.  Her mentor suggested that she work with me to see where I could help her collect this feedback. After we had decided on our methodology, I assigned her the task of creating a walkthrough of her design to present to our users.

Together, we iterated on her walkthrough several times.  In her first attempt, she filled the entire time available to us by presenting her ideas and designs, not leaving enough time to gather feedback.  I gave her advice on what kind of research questions were appropriate and what kind of feedback she could get, and helped her hone her walkthrough so that she would get feedback that was directed and actionable.  I also gave her advice on her presentation style, how to answer questions, and what to do when she felt stuck.  Her final walkthrough was tight and focused, and she did a great job with it.

At the end of her internship, she gave a presentation to the whole user experience team about her project.  She talked about how she started, how she iterated on her design, how she collected feedback from stakeholders, where our research fit into her design.  At the end, she shared some quotes from team members which will stick with her.  One of them was from me:

Every minute you talk is a minute that your user isn’t giving you data.

(At that point, another designer leaned over to me and said, “yeah, that’s something you’d say”.  How well they know me!)

One of the things that we talked about as she worked on her walkthrough was brevity.  When you’re collecting data from your user, your goal is to collect as much actionable data as possible in the time that you have available.  This is just as true for a 5-minute survey as it is for a 2-hour usability study.  If you are talking, your user isn’t.  You can hear yourself talk any time, whereas you only have a very short window of opportunity with your user.  Therefore, when you’re collecting user data, you have to carefully craft what you present to them to maximize the data that they are able to give you.

Her first attempt at her walkthrough required a full hour for her to go through it.  What she actually presented to our users took about 10 minutes.  The rest of the time, we were able to ask questions and probe our users for feedback.  It took a lot of hard work to hone the focus of the walkthrough down to something that was so much shorter than the original, but it ultimately paid off.  We learned a lot, and were able to identify some important design improvements that could be made.

If I had to pick one message that a user experience intern would take away from working with me as we conducted research, I think this is a pretty good one.  I was very pleased to see her call that out as one of her lessons learned here at VMware.

what they don’t tell you about conducting research

Several times lately, I’ve been approached by a couple of the newer designers on my team who are conducting research for VMware for the first time.  In our conversations, it’s become clear that there’s a difference between what you learn about conducting research in school and what you learn when you’re actually out in the field conducting research.

Finding participants for research is always harder than you think it will be, and it takes longer than you think it will.  Not only do you have to get the right participant to take part in the study, scheduling them is difficult because you’ve got to work with your own schedule and theirs too.  When you’re asking others for leads, such as a Program Manager or a Technical Account Manager, you’re going to have to build a relationship with that person and help them understand your research goals to be able to get the right people.

You will always have more research questions than you can answer in the time available.  You’re going to have to prioritize your research questions.  While you’re actually conducting the research, you’re going to find additional research questions, which you might decide should be prioritized higher than the research questions that you’ve already identified.  Be aware that this will happen, and be flexible about it.

You have to get comfortable with silence.  Don’t rush in to fill space.  Let your participant think, let them consider what they’ll do or say next.  But don’t let the silence go on for too long, and don’t forget to prompt a participant who isn’t talking at all.  It’s a fine line to walk, and it takes some practice.

Conducting research isn’t just about having a solid research plan and protocol.  There’s a lot of mechanics surrounding conducting the research that can make or break your research.  The best research protocol won’t give you meaningful results if you can’t talk to the right people or if you have too many questions for the time available.

the road away from Quicken, part 4: using iBank

Now that I’ve got my data imported into iBank, it’s time to actually put it through its paces on my easiest set of financial data (the household accounts).

The UI is a single window.  I’ve never thought myself a fan of Quicken’s windows-everywhere approach, but I’m not sure if I like the single window either.  My household account list has 10 accounts in it, which takes up the majority of the space available.  I don’t want to lose the summary pane, since that’s where the upcoming scheduled transactions are listed, and I very  much want to see them at all times.  I seem to spend a lot of time scrolling through the left navigation pane, and it’s getting on my nerves.

There’s several other UI annoyances.  iBank should win a special award for Most Gratuitous Use of Cover Flow.  I can mostly ignore that, no matter how much it makes me want to snigger.  The main UI looks like a chequebook register, which is natural, but poor use of spacing and fonts make it difficult to read for any length of time.  Poor use of fonts also haunts the reporting function.

Entering data

Entering data manually seems difficult.  While the main window looks like a chequebook register, there’s no blank line.  You’re forced to either use the “new transaction” button in the bottom bar (not the most discoverable location) or cmd-N.  Normally, I like cmd-N for this sort of thing, but it gets annoying if you’ve got a whole account that needs manual entry.  I also don’t like that cmd-N creates a new transaction inline wherever you’ve currently got selected instead of at the top or bottom.  I kept on creating new transactions before I realized where they were located.

From years of Quicken, I’m used to hitting the + key to get the date of a transaction to increase by one.  Actually, I don’t have to hit the shift key, just hitting = results in this.  Muscle memory made me try that, leaving me with = in the date field.  I know that this is a minor detail, but it’s something that I’ve come to rely on, and the absence of this detail is also annoying.

Using scheduled transactions

I started to add in my scheduled transactions.  There are two ways to enter a scheduled transaction.  One is to create an instance of a transaction, and then use a menu selection (or contextual menu selection, if you’re so inclined) to create a scheduled transaction from that.  When using this option, I didn’t see a confirmation that my scheduled transaction had been saved. It took a couple of minutes of looking around to see that it’s in the summary area, in the lower-left corner of the window (you know, the least discoverable area of the screen).  Also, in the left navigation bar, there’s a category called “manage” which has an item “scheduled transactions”.  In either case, the confirmation that the scheduled transaction has been created is pretty subtle, which doesn’t make me feel entirely confident in it.  On the other hand, creating a new scheduled transaction isn’t something that I do frequently, so perhaps this wouldn’t be so annoying after I’ve got everything set up.

The other way to do it is in the list of scheduled transactions.  That’s in the left navigation bar under the “manage” category, which I only found after I’d been trying to create scheduled transactions via the direct manipulation method and wasn’t seeing the confirmation messages.  Adding a scheduled transaction here allows more flexibility than doing so from an existing transaction.  Using an existing transaction assumes that you want it on a monthly basis.  However, there’s thing missing from the scheduled transactions: the ability to have a flexible amount for the scheduled transaction.  For example, my power bill is never the same every month, but it’s always due on the same date.  I’ll have to manually update the transaction amount when I actually make it.  This isn’t a big deal (after all, I have to make that manual step today), but it feels odd to not be able to specify that the amount isn’t fixed.

I haven’t figured out the difference between “transaction templates” and “scheduled transactions”.  Or, rather, I haven’t figured out why both of them exist in the UI.  In fact, I don’t get most of what’s in the left-hand navigation bar.

Reconciling accounts

After I got my new transactions entered, it was time to reconcile.  On my first attempt at reconciling a bank statement, I discovered that I had to reconcile all of my old transactions.  All of my imported information doesn’t have the reconciliation information.  This made reconciling last month’s statement impossible.  On the one account that I tried, I didn’t want to have to go through all six years of statements and do them individually, so I tried to choose all of the old transactions.  I had to click through six bloody years of transactions individually.  And then the numbers didn’t match up, so it was a bloody half-hour of wasted effort.  I deleted the statement, and the account is still unreconciled.

More research told me that I was going to have to go through all of the statements to do this to make sure that all of the individual transactions imported properly.  You can imagine the amount of obscenities that I started spewing at this point.  I’ve only got six years of data on this one, and the idea of having to go through all of my bank statements for that period of time is making my skin crawl.  This might be a dealbreaker for using iBank.

Creating a budget

Although I was pretty upset at the idea of having to go back and reconcile all eleventy billion transactions manually, I also tried to set up a budget.  In doing so, this made me realize that I need to invest time into fixing my categories.  It correctly imported the names of all of my categories, but set them as expenses.  That took a couple of minutes to fix, and then I went through the budget wizard to get started.  Most of it was self-explanatory, until I got to the screen titled “starting cash in envelopes”.  Hunh?  That sent me to the manual to learn that iBank’s budget can use the so-called envelope system of budgeting.

I ended up having to create three budgets to get one that was correct.  The second screen, “select accounts to budget” tripped me up.  It gives a list of all of my accounts, and they’re all checked by default.  The manual is no help, telling me to “[c]heck off the ones you want to include”.  Well, okay, they’re all part of my financial picture, so I want them included, right?  Wrong.  One of my accounts tracks my mortgage.  If I want to be able to include my monthly mortgage payment in my budget, I have to remove my mortgage account from the list of accounts.  Mortgage interest is already included, since it’s got a category, but I can’t capture this payment to my mortgage if I list it as an account to include in my budget.  Conceptually, I can’t grasp this at all.

I set up my budget as an annual budget.  While most of my transactions are monthly, there are some important ones that are biannual or annual.  My car insurance and property tax payments are biannual, for example.  However, this means that the only view that I get of my mortgage is of the year.  This makes it hard to see if I’ve met my budget for, say, my power bill this month.  I can only get the picture of my year-to-date, not this month.

Issues in setting up my budget aside, I don’t like how it shows my budget information.  The visual display of the information is poorly-done.  There’s a progress bar for each individual expense category, which has five states: empty (no transactions), green (on track), yellow (ahead of forecast, but still under budget), red (over budget), and a checkmark (exactly at the budget).  It’s difficult to read a line and see what’s going on.  Overall, viewing my budget and figuring out what’s going on is something that I can’t do in the budget view.  I’d hoped that I could create a report, but there only seem to be four built-in reports that I can’t do anything with.

Conclusion

I’ve spent three weeks using iBank seriously, and I end up with a headache each time I try to use it.  I don’t mean that in an “ugh, this thing sucks” kind of way, I mean that my head physically aches after using it.  Some of it is simply trying to get up and running, some of it is trying to figure out what they’re conceptually doing (the budget thing is entirely perplexing), and some of it is poor visual choices.

After three weeks, I just don’t see how I can use this seriously.  There’s the major hump to get over of having to reconcile all of my accounts for all of their transactions.  I don’t even want to imagine what that time investment is like.  Additionally, the childish budget design and difficulty of using scheduled transactions make it unsuitable for my needs in planning.

This is part four of an ongoing series about trying to move away from Quicken 2007 for Mac.  The previous three parts of the series are here:

the road away from Quicken, part three: importing data into iBank

This is part three of an ongoing series about trying to move away from Quicken 2007 for Mac.  The previous two parts of the series are here:

As I’ve mentioned before, I maintain two separate Quicken accounts: one with my personal data, and one with my household data.  The household data is the smaller dataset of the two, so I’m using that as my test case for importing into potential Quicken replacements.

iBank importer: choose file to importiBank has a 30-day free trial available on its website, so I downloaded and installed it.  On first launch, it detected that I didn’t already have a data file, and gave me the opportunity to import my data file from Quicken.  I didn’t expect that I’d have explicitly export my Quicken data into a QIF.  Exporting to a QIF took a few minutes, and resulted in a 360-KB QIF file.  Then I dragged the file into the iBank importer, which (oddly) gave me a teensy preview of the file.  I’m not quite sure what the point of this preview is.

After selecting my QIF for import, I was then prompted to save my document.  I was surprised that I was expected to decide the filename here, and not entirely sure what “document” means.  I’ve been using Quicken since the dawn of time or thereabouts, so I’m not used to thinking about how it stores its data.  I’m not sure what lies behind the design decision to force the user to think about both the name and location of their datafile.  Shouldn’t they at least provide a reasonable default?

After going through the last question of currency, it started importing.  iBank showed a determinate progress indicator, but didn’t give me an estimate of how long it would take.  My file took about a minute to import.

After importing, I was given a list of my accounts and account types, and asked to choose the most appropriate type for each account listed below.  It noted that iBank supports more account types than QIF1, but didn’t give any guidance as to why I might want to select a different type of account.  For example, my mortgage imported as the “liability” account type, and iBank has both “liability” and “loan”.  Why would I choose one over the other?  Can I change this later?  This drove me to check the “Quick Start Guide”, which didn’t help, so I had to go grab the full manual.  There, I learned that a loan has a payment schedule and a liability doesn’t.  So I changed the account type for my mortgage to loan, and also changed a couple of accounts from chequing to savings, and left the others unchanged.

At this point, I was out of time, so I quit iBank for the evening.  Overnight, I realized that my account data isn’t the only data that I’ve got in Quicken, and wrote that blog post.  As I was writing it, I opened iBank quickly to make sure that I had a couple of details right, and was surprised to see that iBank prompted me with the “Welcome to iBank” screen that I got on my first run.  On doing some research, I discovered that it’s a known bug with a workaround.  Additionally, I noticed that iBank wasn’t saving my window size and location information.  That was pretty annoying.  The workaround that seems to have resolved the document-prompt issue also seems to have addressed this issue.   While I’m glad that it’s a known issue with a workaround, it undermines my confidence in the application to hit such a bug first thing out of the gate.

My confidence is shaken, but still I’ll soldier on.  Now that I’ve gotten my transaction data into iBank, the next step is to try to actually use it.  Wish me luck …

  1. But not Quicken? I’m not sure what it means here, or why it would draw a distinction between Quicken and QIF.

why is Twitter spam so obnoxious?

Twitter spam seems to raise our collective hackles more than any other kind of spam. I admit it: I whinge when I get Twitter spam.  But I observed last week, after engaging in such a whinge, that I’d never dream of doing the same every time I get email spam.  I’ve even asked here whether a particular behavior was Twitter spam1.

Every time I get Twitter spam, I dutifully report it.  I’m never sure if it does any good, but I do it anyway.  Last week, Marco Arment commented on Twitter spam and motivation to report it, concluding that he’s not going to bother to report it any longer because it just seems like a waste of time.  I don’t share his assumption that Twitter isn’t doing anything (or isn’t doing enough), so I still report.  His post is one example of a perception that Twitter spam is a big problem, and that it’s only getting worse.

I don’t have numbers about Twitter spam, so I’m not sure whether it actually is a big problem, let alone one that’s getting worse.  The perception of the problem is one that I find fascinating.  Why is it that Twitter spam bothers us so much?  My theory2 is that there are several aspects of it that, together, make it bloody obnoxious.

A lot of Twitter spam is keyword spam.  There are some keywords that are guaranteed to get you spam responses: iPad, iPhone, laptop, etc.  It’s really galling to know that I’m going to tweet something and be entirely (and often accurately) convinced that I’m going to get spammed in response.  Compare this to email spam: there’s no obvious link between my action and the spam.  The spam that I get in email has nothing to do with anything I’ve ever done, it’s just an indiscriminate firehose.  I’ve been getting a bunch of Viagra spam on my blog post about appearing on the Angry Mac Bastard podcast, but while I find that amusing, it’s probably coincidence.

Twitter spam is addressed directly to me, and mentions/replies to @nadyne are designed to get my attention.  They’re supposed to be so important that they get their own stream, and I never skip reading @nadyne references.  I’ve seen people get testy when there’s some kind of contest or promotion that asks people to tweet something, but that’s rare, and I haven’t observed the same level of vitriol about that behavior as I have for the usual Twitter spam.  Email spam, while it might be addressed directly to me, isn’t highlighted in the same way, it’s just part of my inbox.

I can’t really scan past Twitter spam because it’s identical to everything else.  When I read my email, I scan my inbox to see what I can immediately delete.  Email spam is often, although not always, obvious enough that I can just read the sender/subject and delete it without wasting any more attention on it.  On the other hand, Twitter spam is roughly identical to the rest of Twitter.  While I’m probably not devoting any more time to reading Twitter spam than I am reading the subject line of email spam, I don’t feel as annoyed by email spam because the time spent on the subject of the email is much less than the time that I’d spend actually reading the email.

Are there other aspects of Twitter spam that make it feel so much more obnoxious than other forms of spam?

  1. And, for the record, the overwhelming agreement was that it was spam. Most of the responses came in via Twitter, which isn’t surprising.
  2. As a researcher, I must point out that I have no data whatsoever to back up this theory.