I came up with this idea the other day and I figured it was worth sharing. It requires a web site that can run Google Analytics or a similar tracker, and a lot of web content. That means it really doesn’t work for everyone.
The scenerario is this — lots of apps produce error messages. When the user can’t understand the error message or warning, that’s a problem. This could be, for instance, an error message that says “Your veeblefritzer is demodulated”. What does that mean? Or you may have an error message you have a bug report on, but you don’t know that it’s occurring as much as it actually is. All of these errors in an app can increase user frustration to the point where they’ll (maybe) stop using your application — and initial impressions are important. Users will almost always google for information about these problems.
There’s a pretty good concept I latched onto called the “30 minute rule”. It means after 30 minutes of fighting with anything, your users are busy, and will probably give up. Often that means they need to achieve complete success in that 30 minutes, or at least see there’s some light at the end of the tunnel. So minimizing those annoyances is pretty darn key for any project.
So here’s the trick — if you filter through Google Analytics search results (I really should write a script to automate this), removing the genuine search terms, you can get a pretty good idea about what messages from your application are the most confusing. You can see how often they are searched for, over time, for arbitrary intervals. It helps especially if you have the Analytics tool installed in your bug tracker. It would be even better if it were installed on a mailing list archive, though I think in most cases that’s not going to happen.
Another thing some apps try to do is submit tracebacks to a server — which also is a good idea, but I’d really like to see a good OSS hosted service for OSS apps (i.e. free) for that kind of purpose. i.e. not something kernel or firefox specific, but that I can insert into any application and get error counts for each log line. Though that doesn’t neccessarily solve the case of confusing application messages which aren’t tracebacks. You could also write something like an OSS Splunk that submitted logs to a central server, though it would be very difficult to ensure the logs were not containing proprietary information — opt-in would be pretty low.
I think there’s lots of statistical noise here and it’s probably not perfect, but everything you can do helps.
The Google trick is pretty useful, but not necessarily to everyone … what do people think about open & generic traceback and error service? (Perhaps if it was configurable on a per-application opt-in basis and had very tight controls on who could read the output?)
Maybe call it “ErrorHub”.
