Enterprises typically have silos of search technology, though some vendors hope to tie all of these search sectors together. And with so much digitized content, search technology today is viewed as a primary navigation tool rather than something bolted onto a company's Web sites or databases.
In part one of a two-part interview, Guy Creese, a content management expert at Burton Group, a Midvale, Utah-based research company, weighed in on some of the basics about the evolution of enterprise search and offers some tips for IT managers.
SearchWinIT.com: What exactly is enterprise search?
More on enterprise search Microsoft server chief weighs in on key IT trends
Enterprise Search – 10 essential planning tips for CIOs
IT managers have to do both well. It's common for companies to concentrate on only one of the two. They spend a lot on search marketing, which brings people to the site, but because the site search is so poor it then drives people away. They may tune the site search well, but no one will use it because they don't come to your site.
Enterprise search covers both finding content on your website as well as searching behind the firewall.
Where are vendors in terms of developing tools that link both?
Creese: Bringing together all of the different search sectors is still a bit of a Holy Grail. Microsoft is developing toward that goal. The SharePoint group owns search – SharePoint search, MSN search and desktop search. Google has Google Desktop and the [Google Search] appliance.
Prior to 2000 there was a robust search market led by specialists. There were companies like Autonomy Corp. and Verity Inc., which have since merged. Today, there is Fast [Search and Transfer], which is now part of Microsoft; Enquiro [Search Solutions Inc.]; Endeca [Search Technologies Inc.]; and Mercado [Software]. There are specialized search engines. Some search for music.
Microsoft talks about segmenting the market in a Goldilocks fashion. You have low-end search with basic capabilities. There is a middle tier with start ups like SharePoint Search, which can cost from $20,000 to $100,000. On the high end, sophisticated search engines, such as Autonomy and Fast can cost from $250,000 to $500,000.
Who in the company is usually in charge of enterprise search?Creese: In a business unit, perhaps the marketing VP wants better search [capabilities] because of the corporate website. They may be losing visitors. Or it may be a corporate librarian that worries about internal knowledge management. Perhaps people can't find content on the internal portal.
Typically people buy search and do it without talking to each other. The CFO then realizes he is paying for four different search licenses. So IT gets involved because they let the business units solve their pain points but now they need a strategy.
Websites have become too complex for people to click their way around. With so much digitized content, you need to have better search and records management.
How are search tools sold?
Creese: There is search available as [software as a service]. Your website can sign up for something like CrownPeak [Technology]. It goes to your site, indexes your content and adds some Java script to your Web pages. It types in the query there, the query goes back to CrownPeak and CrownPeak goes back to the pages that are appropriate.
Sometimes it's sold as an appliance, like Thunderstone, Google and Microsoft. Pricing will differ based on how many documents are indexed. Google sells different sized appliances. Sometimes it's the same appliance but there is a switch in place that is based on how many documents you index. You might pay for a server. Large multinationals may not have a single search server but one in the U.S., one in Europe and one in Asia.
And some vendors sell by the number of connectors. I've seen one vendor that says you get certain connectors for free, but if you want to use, say, OpenText, you might have to pay extra.
What's does the acquisition of Norway's Fast Search and Transfer buy Microsoft?
Creese: Microsoft saw itself in the middle of the market. The [low end] Google Appliance remained a thorn in its side so [Microsoft] came out with Search Server  and Search Server  Express. These can be put on appliances – it's for people who are interested in search but not SharePoint [Enterprise] Search.
Microsoft bought Fast for two reasons. It wanted a Microsoft migration strategy so if you got into a Search Server Express you could move up to a middle or high-end Fast solution. Also, Microsoft wanted Fast to handle the high-end requirements and thought it would take too long to build.
What does the Fast technology give IT shops that they could not get from those other tools?
Creese: It helps in the sophistication of tuning the results. When someone submits a query, it knows that PDF docs are typically final versions of docs so it might put the PDF first since it's the more authoritative version.
There is a lot under the covers with Fast [technology]. Microsoft got a revamped search engine. Not that the one it had was deficient, but the CTO really went back to basics and rebuilt the thing. You can do a query via SQL, or an X query, which is designed for retrieving unstructured data. There is also Fast regular search APIs. What that does is let the Fast search engine handle structured and unstructured content.
We are seeing things like [business intelligence] and search start to combine. People want the information and don't want to think about whether it's stored in the database or in the content repository.