Our Search is the Best! Wait, is it?

So you’ve launched your new enterprise search implementation for your customers, after an astronomical amount of effort, painfully tight deadlines, and tearing through a large pile of cash. You’re quite proud of your new search, as you should be. One small question is starting to bother you however – now what?

How can you tell if your new search is being used at all?   Oh, I see – you gather usage statistics using one of the popular website statistics products on the market. Sounds like you have things covered. But wait – how do you know people are really finding what they’re looking for? Does your website statistics product tell you that? After all, you’ve invested in a search engine, so it would be nice if users could find something with its help. Otherwise, offering users a deficient search solution can sometimes be worse than not offering any type of search solution at all.  How do you measure the success of your search offering?   The answer is simple – by gathering good, relevant and detailed search statistics relating to the search engine itself, and proceeding to understand what the captured statistics mean.

Gathering Statistics

Capturing good statistical information about how your customers are using your search can really help protect and improve your search investment. It may be hard to get good user feedback from users directly, but user actions can often speak louder than words.

How should you go about capturing relevant search statistics?  You may find that the website analytics products you’re already using aren’t helping you out much beyond the basics, when it comes to search.  You are definitely not alone. The problem lies within the fact that website usage analytics products are great for tracking web navigation activities, but not great at tracking enterprise search usage patterns. At the other end of the spectrum, enterprise search vendors often have products that come with integrated search statistics engines. These engines record search specific information, such as what search queries were made, how long they took to process and find results, what the search terms were, etc. Unfortunately, what is lacking from most enterprise search statistics engines, is a way to bridge web usage patterns with core search specific captured information. Bridging the capturing and reporting of user navigation, with the knowledge of how it directly relates to search results and content usage patterns, would truly show just how effective your search implementation is.

How to Interpret Your Statistics

Having good search statistics can really help you protect your search investment, by ensuring users get the most of your search offering.   Good statistics can help pin-point where you should, or should not, invest more time and money to make your search truly shine.  This being said, as good as your search statistics are, they will bring little value to you if they are not interpreted properly. Below I’ve listed a few statistical areas that you should keep an eye on within your search implementation. I’ve also explored a few different ways to interpret these statistics, and the benefits that you can take advantage of from this knowledge. Some of the mentioned statistical areas can be found through popular analytics products, while others are harder to obtain (but just as, if not more, valuable).

Search Frequency

Knowing how many searches are made in a given time period is definitely a useful metric.  A low number of searches may mean your search engine lacks marketing, is not visible, or is not intuitive enough to be used properly.  Should you offer a search box in your website header, or is a link in your header to a search page good enough? Knowing which days of the week or time of the day your search is less busy could help you establish the best time for a search maintenance window, minimizing customer impact.  As for a high volume of searches, be careful – this number can be misleading.  It could be a sign of success, or it could mean that users have to perform many searches to find what they’re looking for.  The number of searches made in a given time period should definitely be analyzed in conjunction with other statistics.

Query Terms

Knowing what your top query terms are can help you with ad placement or suggested links (also called best bets, related links, promotions, etc.). For example, if you run a convenience store and you suddenly see a lot of searches for candles, you’ll likely want to make sure you have something relevant to show to users of your search. Showing and promoting a 10% discount coupon on candles, regardless of what your regular search results would show, will definitely help with sales. Another use for query terms involves using this captured statistical information to populate auto-suggest features of search boxes. This way you can ensure that your suggestions are relevant, based on frequently used terms. Query term metrics can also be used to keep track of what terms are being searched for, that might exist under a different name in your system. Armed with this knowledge you might want to create synonyms for these query terms to help ensure that your users find relevant results.  Seeing queries that generate no or few results strongly suggests that you should create synonyms for the query terms being used, or suggested links, to help reduce the dreaded “No results found” message.  Knowing the number of query terms that are usually used for searching can also influence your search implementation.  If you’re building a complex search condition triggered when encountering a combination of user query terms, you might stop and rethink this approach if you see that 99% of the submitted search queries contain one term only.

Performance

Performance metrics are relatively easy to get – it’s getting the right metrics, however, that can be tricky.  Are you more concerned with tracking your search server performance, or the performance of the end-to-end user experience (including network latency and page rendering)?  When you want to look at search engine performance only, to make sure that it’s not the cause of user reported “slow searches”, how do you go about it? For some search implementations one user query can translate into two, three or more search engine requests (one for result documents, one for facets, one to get synonyms, etc.).  Which search request one do you track? Should you combine them all? You might benefit from knowing how fast your search is, depending on the number of search terms provided.   If your search hangs on the occasional long search query, but is otherwise quite fast, you might want to consider limiting the number of user terms (or find other ways to limit the impact of long search queries). Related to this, finding out what the slowest search queries performed are, can also give you pointers towards areas that need to be improved.

Document Viewing

Getting statistics that tie search results to document viewing events can be harder to attain, but are definitely quite valuable. Knowing which queries resulted in having none of the search result documents clicked on for viewing, can help you determine whether there are issues with your search relevancy, whether there is a need synonyms or suggested links to be added, etc.   How often do users have to go beyond the first results page?  How many pages are viewed on average for a single search?  More importantly, you might want to know which results page a user was on when she decided to open a document. This shows whether users have to navigate through several pages of results to find what they’re really looking for. If resulting documents are frequently viewed several pages into the results, you might want to take a close look at the results that come before the viewed document. Your search relevancy might need to be adjusted, or you might want to consider investing in better guided navigation to help prevent extensive result browsing. Another important document viewing statistic that can be useful is knowing how many result documents are viewed by a user, for the same search. If a user has to open and read several documents to find what they’re looking for, there might be an issue with the way you’re indexing content or presenting the summary content in the results. Knowing what your most viewed documents are can also help you decide whether they should be promoted as suggested links, or have their relevancy boosted by other means, to make sure they appear higher in the results.

Users

Capturing user specific statistical information can definitely be a sensitive topic, especially from a privacy point of view.  For auditing purposes however (in high security environment), it may be required for you to see, sometimes in real-time, what is being searched on and by whom.  Statistics tying search users to their search queries can also help better overall search experiences. By knowing what your search user is interested in, or frequently searching on, you can tailor their search results accordingly. You could, for example, present them with content promotions relevant to their search behaviour.  Knowing who the users of your search are, and how they use the search, can also help you target the most frequent users of your search for feedback.

Functionality

Knowing which features of your search offering your clients are actually using, can really help show if you’ve made the right search functionality design decisions. By looking at what is used, you can determine if there are certain areas that should be improved to better serve your users, and/or certain areas that should be de-commissioned due to lack of use.  Tracking which features of your search are being used can help answer questions such as:

• Are people using your advanced search?
• Are users taking advantage of your guided navigation, and which ones do they click on (eg. facets, taxonomy, etc.)?
• How many users go beyond the basic search box?
• Are my suggested links effective?

Equipped with answers to these questions can help you decide where it makes the most sense to invest next.  Should you still add support for more search criteria in your advanced search if you realize it’s only used by 0.5% of your user base?  Or would it make sense to add more search facets and filtering options, since the ones already present are used quite a bit?

Conclusion

An enterprise search implementation is a like newborn infant – you have to pay close attention to it, and nurture it properly, so it grows healthily.  The ability to adapt quickly to user search trends, performance issues, and other related search metrics, highlights the differences between a great search implementation that keeps getting better, and a mediocre search implementation that barely services users’ needs. Capturing and acting on good statistics helps improve search maintenance, makes it easier to decide where to expand, allows you to keep your search engine efficient, and helps keep its content relevant.

I’m sure some of you make use of search statistics to help improve your overall search environment. We would definitely like to hear about the metrics you capture, and how you benefit from using them.