Latest Applications in Enterprise Search

In my previous post, I described the future of enterprise search. In this post, I will describe few new search applications that could be interesting.

Concept Searching

Founded in 2002, Concept Searching provides software products that deliver automatic semantic metadata generation, auto-classification, and powerful taxonomy management tools. Concept Searching is the only platform independent statistical metadata generation and classification software company in the world that uses concept extraction and compound term processing to significantly improve access to unstructured information. The Concept Searching Microsoft suite of technologies runs in all versions of SharePoint, Office 365, and OneDrive for Business.

The technologies are being used to improve search outcomes, deploy an enterprise metadata repository, enable effective records management, identify and secure sensitive information, improve governance and compliance, social tagging, collaboration, text analytics, facilitate eDiscovery, and drive intelligent migration.

Concept Searching, developer of the Smart Content Framework™, provides organizations with a method to mitigate risk, automate processes, manage information, protect privacy, and address compliance issues. This infrastructure framework utilizes a set of technologies that encompasses the entire portfolio of unstructured information assets, resulting in increased organizational performance and agility.

Lexalytics, Inc.

Lexalytics provides enterprise and hosted text analytics software to transform unstructured text into structured data. The software extracts entities (people, places, companies, products, etc.), sentiment, quotes, opinions, and themes (generally noun phrases) from text. Text is considered unstructured data which comprises somewhere between 31% and 85% of what is stored in any given enterprise.

Lexalytics is an OEM vendor of text analytics and sentiment analysis technology for social media monitoring, brand management, and voice-of-customer industries. The software uses natural language processing technology to extract the above-mentioned items from social media and forums; the voice of the customer in surveys, emails, and call-center feedback, traditional media, pharmaceutical research and development, internal enterprise documents, and others.

Lexalytics, provides a text mining engine that is used by a number of search partners like Coveo, Playence, and Oracle to add additional metadata to their search. This is additional intelligence around "just what do those words actually mean?" In other words, this engine is boosting the value of search by providing more information into the index. This enables other applications, and helps search be "smarter".

MaxxCAT

MaxxCAT provides enterprise search solutions for corporate intranets, web sites, databases, file systems and applications, and other environments that require rapid document retrieval from multiple data sources. The flagship products offered by MaxxCAT are the SB-250 series and the EX-5000 series network search appliances. Also available are series of cloud-enables storage appliances.

Basis Technology

Founded in 1995, this software company specializes in applying artificial intelligence techniques to understanding documents written in different languages. Their software enhances parsing tools by classifying the role of words and provides metadata on the role of words to other algorithms. Software from Basis Technology will, for instance, identify the language of an incoming stream of characters and then identify the parts of each sentence like the subject or the direct object.

The company is best known for its Rosette Linguistics Platform which uses Natural Language Processing techniques to improve information retrieval, text mining, search engines and other applications. The tool is used to create normalized forms of text by major search engines, and, translators. Basis Technology software is also used by forensic analysts to search through files for words, tokens, phrases or numbers that may be important to investigators.

dtSearch

Founded in 1991, this company specializes in text retrieval software. Its current range of software includes products for enterprise desktop search, Intranet/Internet spidering and search, and search engines for developers (SDK) to integrate into other software applications

LTU technologies

Founded in 1999, this company is in the field of image recognition for commercial and government customers. The company provides technologies for image matching, similarity and color search for integration into applications for mobile, media intelligence and advertisement tracking, ecommerce and stock photography, brand and copyright protection, law enforcement and more

Sematext Group, Inc.

This company's product SSA - Site Search Analytics - continuously monitors, measures, and improves the search experience. It identifies top queries, problematic zero-hit queries, common misspellings, etc. It measures and compares search relevance and improves conversion rates. It is available It is available on-premises and in the cloud.

Exorbyte

This is a privately held software company which was founded in 2000 in Konstanz, Germany, with an additional office in the United Kingdom (Bristol). The company develops intelligent software for search and analysis of structured and semi-structured data.

Their product MatchMaker is the leading error-tolerant search & match platform for huge master data volumes. The multiple award-winning software technology thinks, searches and finds like a human – but dramatically faster, in much more complex configurations and with no serious data restriction using keys or similar methods. It is available on-premises and in the cloud.

Federal authorities, insurance agencies, ICT firms and more use this software to identity a resolution in diverse, data-intensive business processes such as input management, enterprise search and data quality. It has easy customization and integration.

Inbenta

Founded in 2005,this company provides enterprise semantic search technology based on artificial intelligence and natural language processing. It offers intuitive search solutions and intelligent content support for website and corporate Intranets.

Content Analyst Company

This is a privately held software company which develops concept-aware text analytics software called CAAT, which is licensed to software product companies for use in eDiscovery. In 2013, five CAAT-powered products were named in the Gartner eDiscovery Magic Quadrant Report, and the analyst firm 451 Group referred to CAAT as The Hottest Product in eDiscovery.

Content Analyst's CAAT analytics software is a machine learning system based on latent semantic indexing technology. CAAT provides several text analytics capabilities using both supervised learning and unsupervised learning methods including concept search, categorization, conceptual clustering, email conversation threading, language identification, near-duplicate identification, auto summarization and difference highlighting.

SearchYourCloud

With SearchYourCloud and its patented, federated search technology, a single search request in Outlook simultaneously and transparently searches your email, desktop and all of your cloud storage sources and delivers highly targeted results. You get exactly the information you need with just one query.

Docurated

Docurated aggregates all your documents in one place, turning them into a searchable and customizable database. Docurated will now provide Dropbox integration as well. It accelerates sales in companies looking for fast growth by making the best marketing content readily available to Sales around the world. Docurated works with your existing content stores and uses machine learning to enable your team to find and re-use the most effective content with no manual tagging or uploading.

This is the next generation visual knowledge management platform which solves the information retrieval problem for leading companies like Clorox, Omnicom, Netflix, Weather Channel, and many others. Docurated enables sales, marketing, and technology teams to surface and use the exact chart or slide they need, no matter where it is stored, without slogging through folders and files. Docurated seamlessly integrates with existing folder-based repositories.

Lucene

Apache Lucene is a free open source information retrieval software. It is supported by the Apache Software Foundation and is released under the Apache Software License. While suitable for any application which requires full text indexing and searching capability, Lucene has been widely recognizedfor its utility in the implementation of Internet search engines and local, single-site searching.

At the core of Lucene's logical architecture is the idea of a document containing fields of text. This flexibility allows Lucene's API to be independent of the file format. Text from PDFs, HTML, Microsoft Word, and OpenDocument documents, as well as many others (except images), can all be indexed as long as their textual information can be extracted.

These are just few search applications that are currently on the market. There are many others. Choosing the right application is based on your organization's requirements.

Future of Enterprise Search

Enterprise search is a developing industry. In this post, I will describe the latest developments in enterprise search.

Effective enterprise search represents one of the most challenging areas in business today. The whole area of search has been revolutionized by Google. Employees now expect to be able to locate relevant data as easily as they navigate the web through Google. When this ease of search is not replicated in organizations' systems, it can be quite frustrating. As we create more content than ever before, the importance of effective search across the enterprise continues to grow.

Until recently, much of the enterprise search technology remained unchanged. The general purpose enterprise search offerings were fairly similar in technology and scope. There are now many software companies who direct their efforts towards enterprise search. The future will bring shorter innovation cycles, continuous user experience improvements, deeper integration with first- and third-party applications and more ETL-like (extract, transform and load) functionality to handle poor quality content.

In the second half of the 2000’s, the enterprise search companies were absorbed by the large software companies:

• Microsoft acquired FAST Search in 2008
• Adobe acquired Mercado in 2009
• Dassault Systèms acquired Exalead in 2010
• Hewlett Packard acquired Autonomy in 2011
• Oracle acquired Endeca in 2011
• IBM acquired Vivisimo in 2012

User experience is a broad topic in itself, with active trends including:

• Richer information about the user to determine context, such as their business context, social context, mobile device sensors, location, speech recognition, preferences and historical usage.
• Advances in visualization such as HTML 5.
• Natural language processing as in the trends seen with Wolfram Alpha and smart phone digital assistants, such as Apple’s Siri, Microsoft’s Cortana and Google Now.
• Richer results that look less like a page of links and more like answers to questions.
• Elements of knowledge management that add meaning to queries and results.
• Enterprise search products will become increasingly and more deeply integrated with existing platforms, allowing more types of content to be searchable and in more meaningful ways. It will also become less of a dark art and more of a platform for discovery and analysis.

The future of enterprise search seems destined to continue with simple keyword and Boolean searching, augmented by faceted navigation based on metadata. Virtually every e-commerce web site today offers guided navigation based on metadata.

This ubiquitous model now appears in most of the leading enterprise search products and users immediately understand how a simple text query can quickly be focused to a specific domain by clicking on a metadata filter. This updated search model is increasing demand for auto-classification products which can generate descriptive metadata automatically based on an analysis of the document’s unstructured content.

Open source software has made significant improvements, displacing many of the traditional search vendors. Lucene and its supporting companies like LucidWorks provide solid search functionality at a hard-to-beat price. Where vendors are seeing success is in four main areas:

• Providing functionality beyond typical "search" – extending to facets, true knowledge management, multimedia search, and other functionality.
• Focusing on vertical-specific applications like fraud and supply-chain management.
• Working with larger, more conservative enterprises.
• Providing a SaaS, one-stop-shop for zero (or low) touch functionality.

A few major factors are going to drive the industry going forward:

• Open source will continue to get better and drive out inefficiency in the market .
• More, better information about the searcher: location awareness, profile sharing, time dependence, deeper understanding of the context and content of the search. With this information, you can provide better, more relevant results. 
• Lower tolerance for hassle: people expect search to "just work" – not understanding that it can be just as complicated as any other major IT initiative. By having low-touch solutions, SaaS providers will make major progress in the small/medium business world.
• Search all the things!: Integrated understanding of objects, video, speech, as well as traditional semantic sources like text will combine together better into a whole that allows for information retrieval no matter what the format.

Another area for future development is machine to machine consumption of information and sharing. Search providers are increasingly applying advanced analytics of text and other media so their users’ desires are more deeply satisfied through relevant search results. Search will be increasingly entity-centric and collaborative.

Future of search will include more semantic understanding of both content and queries. For example Exorbyte is focused on searching in structured master data – people, products and places, and its ability to query this data without use of restrictive match-keys for both lexicographical and semantic similarity is globally unique.

The future of search goes through natural language processing while on the other hand it will entail the capability of providing advanced information analysis during indexation time.

The facility to search within the document itself is becoming vital. The Docurated platform caters for instant access to the most relevant page or slide without even having to open the document.

Effective enterprise search can eradicate inefficiency. Enterprise search will become instant and intuitive, paving the way for increased productivity across the enterprise.

In my next post, I will highlight few search applications that could be worth looking into...

SharePoint 2013 Improvements

In this post, I will describe few improved features in SharePoint 2013.

Cross-Site Publishing

SharePoint 2013 has cross-site publishing. In the previous versions of SharePoint, it was not possible to easily share content across sites. Using cross-site publishing, users can separate authoring and publishing into different site collections: authored content goes into an indexable "catalog", and you can then use FAST to index and deliver dynamic content on a loosely coupled front end.

This feature is required for services like personalization, localization, metadata-driven topic pages, etc. An example of its use is a product catalog in an e-commerce environment. It can be used more generally for all dynamic content. Note that cross-site publishing is not available in SharePoint Online.

Here is how it works. First, you designate a list or a library as a "catalog". FAST then indexes that content and makes it available to publishing site collections via a new content search web part (CSWP). There are few good features put into creating and customizing CSWP instances, including some browser-based configurations. Run-time queries should execute faster against the FAST index than against a SharePoint database.

Cross-site publishing feature could significantly improve your content reuse capabilities by enabling you to publish to multiple site collections.

Templates

Creating templates still begins with a master page which is an ASP.NET construction that defines the basic page structure such as headers and footers, navigation, logos, search box, etc. In previous versions, master pages tended to contain a lot of parts by default, and branding a SharePoint publishing site was somewhat tricky.

SharePoint 2013 has new Design Manager module, which is essentially a WYSIWYG master page/page layout builder. Design Manager is essentially an ASP.NET and JavaScript code generator. You upload HTML and CSS files that you create and preview offline. After you add more components in the UI (for example, specialized web parts), Design Manager generates the associated master page. Page layouts get converted to SharePoint specific JavaScript that the platform uses to render the dynamic components on the page.

You can generate and propagate a design package to reuse designs across site collections. There are template snippets that enable you to apply layouts within a design package, but they are not reusable across design packages.

This process is more straight forward than the previous versions, but it still would likely involve a developer.

Contributing Content

SharePoint 2013 enables contributors to add more complex, non-web part elements like embedded code and video that does not have to be based on a specific web part. This feature is called "embed code". Note that if you are using cross-site publishing with its search based delivery, widget behavior may be tricky and could require IT support.

With respect to digital asset management, SharePoint has had the ability to store digital assets. However, once you got past uploading a FLV or PNG file, there was scant recourse to leverage it. SharePoint 2013 brings a new video content type, with automatic and manual thumbnailing.

Creating image renditions capability has also improved. It allows you to contribute a full fidelity image to a library, and then render a derivative of that image when served through a web page.

Other added features include better mobile detection/mobile site development and an improved editing experience.

Metadata and Tagging Services

SharePoint 2013 has solid metadata and tagging services with improved and simplified the term store. However, there is still no versioning, version control or workflow for terms.

Big improvement is that using FAST, you can leverage metadata in the delivery environment much more readily than you could in previous versions. You can use metadata-based navigation structures (as opposed to folder hierarchies), and deploy automated, category pages and link lists based on how items are tagged.

Enterprise Content Management and Mobile Devices

With mobile devices becoming increasingly powerful, users want to access their documents while on the move. iPads and other tablets in particular have become very popular. Increasingly, employers allow employees to bring mobile devices of their choice to work.

"Bring Your Own Device" (BYOD) policy became wide spread in organizations and users started expecting and demanding new features that would enable them to work on their documents from mobile devices. Therefore, the necessity to have mobile access to content has greatly increased in recent years.

As with most technology, mobile and cloud applications are driving the next generation of capabilities in ECM tools. The key capabilities in ECM tools are the ability to access documents via mobile devices, ability to sync documents across multiple devices, and the ability to work on documents offline.

Most tools provide a mobile Web-based application that allows users to access documents from a mobile’s Web browser. That is handy when users use a device for which the tool provides no dedicated application.

The capabilities of mobile applications vary across different tools. In some cases, the mobile application is very basic, allowing users to perform only read-only operations. In other cases, users can perform more complex tasks such as creating workflows, editing documents, changing permissions or adding comments.

Solutions and Vendors

Solutions emerged that specialize in cloud based file sharing capabilities (CFS). Dropbox, Google Drive, Box.com, and Syncplicity (acquired by EMC) provide services for cloud-based file sharing, sync, offline work, and some collaboration for enterprises.

There is considerable overlap of services between these CFS vendors and traditional document management (DM) vendors. CFS vendors build better document management capabilities (such as library services), and DM vendors build (or acquire) cloud-based file sharing, sync, and collaboration services. Customers invested in DM tools frequently consider deploying relevant technology for cloud file sharing and sync scenarios. Similarly, many customers want to extend their usage of CFS platforms for basic document management services.

DM vendors which actively trying to address these needs include Alfresco (via Alfresco Cloud), EMC, Microsoft (via SkyDrive/ Office 365), Nuxeo (via Nuxeo Connect), and OpenText (via Tempo Box). Collaboration/social vendors like Jive, Microsoft, and Salesforce have also entered the enterprise file sharing market. Other large platform vendors include Citrix which acquired ShareFile. Oracle, IBM, and HP are about to enter this market as well.

Key Features

Number of Devices - Number of devices that the ECM vendor provides mobile applications for is very important. Most tools provide specific native applications for Apple’s iPhone and iPad (based on iOS operating system) and Android-based phones and tablets. Some also differentiate between the iPhone and iPad and provide separate applications for those two devices. Some provide applications for other devices such as those based on Windows and BlackBerry.

File sync and offline capabilities - Many users use more than one device to get work done. They might use a laptop in the office, a desktop at home, and a tablet and a phone while traveling. They need to access files from all of those devices, and it is important that an ECM tool can synchronize files across different devices.

Users increasingly expect capabilities for advanced file sharing, including cloud and hybrid cloud-based services. Most tools do that by providing a sync app for your desktop/laptop, which then syncs your files from the cloud-based storage to your local machine.

Most tools require users to create a dedicated folder and move files to that dedicated folder, which is then synced. A few tools like Syncplicity allow users to sync from any existing folder on your machine.

A dedicated folder can be better managed and seems to be a cleaner solution. However, it means that users need to move files around which can cause duplication. The other approach of using any folder as a sync folder allows users to keep working on files in their usual location. That is convenient, but if users reach a stage when they have too many folders scattered around on their laptop and other synced machines, they might have some manageability issues.

Some tools allow users to selectively sync. Rather than syncing the entire cloud drive, users can decide which folders to sync. That is useful when users are in a slow speed area or they have other bandwidth-related constraints. In some cases, they can also decide whether they want a one-way sync or a bi-directional sync. Once they have the files synced up and available locally, they typically can work offline as well. When they go online, their changes are synced back to the cloud.

Most tools that provide a dedicated mobile applications can also sync files on mobile devices. However, mobile syncing is usually tricky due to the closed nature of mobile device file systems.

While most ECM and DM vendors provide some varying capabilities for mobile access, not all of them can effectively offer file sync across multiple devices.

Your options should be based on your users' requirements. Access them very carefully before deciding on a suitable solution for your organization.

Success in Enterprise Content Management Implementations

A successful enterprise content management (ECM) implementation requires an ongoing partnership between IT, compliance, and business managers.

Strict top-down initiatives that leave little for users' requirements consideration result in ECM systems that users don’t want to use.

Similarly, an ad hoc, overly decentralized approach leads to inconsistent policies and procedures, which in turn leads to disorganized, not governed, not foundable content. In both extremes, the ECM initiative ends with a failure.

Whether your organization uses an agile, waterfall or mixed approach to ECM deployment, ECM leaders must think about program initiation, planning, deployment, and ongoing improvement as a process and not as isolated events. Team composition will change over time of ECM project planning and roll-out, as different skill sets are needed.

For example, a business analyst is a key member of the team early in the project when developing a business case and projecting total cost of the project, while legal department will need to get involved when documenting e-discovery requirements.

But, there is often no clear location in the org chart for fundamental content management responsibilities, and that can contribute to weakened strategy, governance and return on investment (ROI).

Approach to ECM

Successful ECM initiatives balance corporate governance needs with the desire of business units to be efficient and competitive, and to meet cost and revenue targets.

Organizations should determine the balance of centralized versus decentralized decision making authority by the level of industry regulation, jurisdiction, corporate culture and autonomy of business units or field offices.

A central ECM project team of content management, business process, and technology experts should define strategy and objectives and align with the technology vision. Local subject matter experts in business units or regional offices can then be responsible for the execution and translation of essential requirements into localized policies and procedures, along with the business unit’s content management goals.

Business managers can help to measure current state of productivity, set goals for improvement, contribute to a business case or forecast total cost of a CMS ownership over a number of years. A trainer will be needed during pilot and roll-out to help with change management and system orientation. Legal department should approve updates to retention schedule and disposition policies as practices shift away from classification schemes designed for paper to more automated, metadata-driven approach.

Project Roles

The following roles are essential for an ECM project:

• Steering committee is responsible for project accountability and vision. Their role is to define an overall vision for an ECM project and outline processes and procedures to ensure integrity of information.

• Project manager is responsible for the ECM project management during CMS deployment. The project manager's role is to create project plans and timetables, identify risks and dependencies, liaise with business units, executive sponsors, IT, and other teams.

• Business analyst is responsible for outlining the desired state of CMS implementation and success metrics. This role is to gather business and technical requirements by engaging with business, technical, and legal/compliance stakeholders. They need to identify the current state of operations and outline the desired future state by adopting a CMS system.

• Information architect's role is to define and communicate the standards to support the infrastructure, configuration, and development of ECM application.System administrators - their role is to define and implement an approach to on-premises, cloud, or hybrid infrastructure to support a CMS.

• CMS administrator is responsible for the operation of the CMS. This role is to define and implement processes and procedures to maintain the operation of the CMS.

• User experience specialist's role is to define standards for usability and consistency across devices and applications, and create reusable design and templates to drive users' adoption.

• Records and information managers' role is to define and deploy taxonomies, identify metadata requirements, and to develop retention, disposition, and preservation schedules.

Core competencies will be supplemented by developers, trainers, quality assurance, documentation, and other specialists at various phases of the ECM deployment project. It is important to provide leadership during the deployment of a CMS. The team should bring technical knowledge about repositories, standards and service-oriented architectures, combined with business process acumen and awareness of corporate compliance obligations.

Information architects will be important participants during both the planning and deployment phases of the project. Communication and process expertise are essential for ongoing success. IT, information architect, and information managers should learn the vocabulary, pain points, and needs of business units, and help translate users' requirements to technical solutions so that the deployed CMS could help to improve current processes.

Compliance subject matter experts should communicate the implications and rationale of any change in process or obligations to users responsible for creating or capturing content.

Project plans, budgets and timetables should include time for coaching, communication, and both formal and informal training. Even simple file sharing technology will require some investment in training and orientation when processes or policies are changed.

Strategic Asset

ECM is a long-term investment, not a one-time technology installation project. Enterprises can often realize short-term ROI by automating manual processes or high-risk noncompliance issues, but the real payoff comes when an enterprise treats content as a strategic asset.

A strong ECM project team demonstrates leadership, communication skills and openness to iteration, setting the foundation for long-term value from the deployment efforts.

For example, a company aligned its deployment and continuous improvement work by adopting more agile approaches to project delivery, as well as a willingness to adopt business metrics (faster time to market for new products), instead of technology management metrics (number of documents created per week). That change allowed the company to better serve its document sharing and collaboration needs of sales teams in the field.

The project team must engage directly with the user community to create systems that make work processes better. It is a good idea to include hands-on participation and validation with a pilot group.

Recommended Practices

Follow best practices from completed ECM projects. Review processes, applications, forms, and capture screens to identify areas of friction when people capture or share content. User experience professionals have design and testing experience, and they need to be included in the ECM deployment team.

User participation is valuable throughout the ECM deployment project. Direct input on process bottlenecks, tool usability and real-world challenges helps prioritize requirements, select technologies and create meaningful training materials.

Senior managers who participate on a steering committee, or are stakeholders in an information governance strategy, should allow their teams to allocate adequate time for participation. That might mean attending focus groups, holding interviews, attending demos and training, or experimenting with new tools.

Be Proactive

A sustainable and successful ECM initiative will be responsive to the changing behavior of customers, partners and prospects, changing needs of users, and corporate and business unit objectives. Stay current with ECM and industry trends. ECM project team members should keep one eye on the future and be open to learning about industry best practices.

Businesses will continue to adopt mobile, cloud and social technologies for customer and employees communication. Anticipate new forms of digital content and incorporate them into the ECM program strategy proactively, not reactively.

Proactively push vendors for commitments and road maps to accommodate those emerging needs. Stay alert to emerging new vendors or alternative approaches if the needs of business stakeholders are shifting faster than current ECM technology. Aim for breadth as well as depth of knowledge, and encourage team members to explore adjacent areas to ECM to acquire related knowledge and think more holistically.