Pavel Ignatov - Fotolia


Merging your view of structured and unstructured data in SharePoint

Companies are seeing the merits of merging structured and unstructured data in SharePoint. But gaining the resulting business insight will take work.

SharePoint can provide a unified platform for converting disparate data points into valuable insights, but the...

learning curve will be steep for many companies.

When it's set up correctly, SharePoint can merge both structured and unstructured data to provide a unified view of previously siloed information, where the sum is greater than the parts. But that may require integrating fragmented systems and implementing common taxonomies, which are no small tasks.

First, let's distinguish between structured and unstructured data. Structured data resides in a database. It is numerical in nature and has inherent structure. Unstructured data, by contrast, doesn't reside in a database. It may involve company contracts, video files or even comments posted on a social media platform. And historically, merging these two kinds of data in a way that reveals patterns and trends has been difficult with SharePoint -- or any application, for that matter.

Historically, SharePoint has offered impoverished search options. But more recently, Microsoft has developed tools to merge these data types for analysis, such as PowerBI to analyze Google Analytics and the new Delve service. Consequently, businesses of nearly every size can create a more unified view of their enterprise content.

By merging this view, companies may be able to gather new business insights. A roofing company, for example, could potentially merge structured and unstructured data by marrying information in a customer relationship management system with sales materials, such as proposals, contracts and satellite photos of roofs it stores in SharePoint.

The company can then use analytics tools like Power BI to identify how documentation helps sell more; for instance, in their sales proposals, do successful salespeople use more photos or more carefully document work to be done than do less successful salespeople? These are still emerging concepts, but by marrying structured and unstructured data, the possibility is there.

Managing and accessing content in SharePoint

SharePoint has had an evolving relationship with structured data. In 2007, Microsoft introduced the Business Data Catalog (BDC). This service allowed administrators to define structured data sources that could be identified and retrieved through SharePoint search results. Unfortunately, BDC was limited to read-only access; it was difficult to configure and few organizations made true use of it. But BDC was the first step toward a unified view of structured content and related unstructured content that enhanced business decision making.

Since 2007, Microsoft has continued down two complementary paths: enhancing its underlying data capabilities through BDC and subsequent PerformancePoint, Power BI and Power Pivot (à la SQL Server). When all three were combined, Microsoft had created what is arguably the most business-centric and accessible structured data analysis tool in the marketplace.

The combination of structured and unstructured data features give companies real opportunities.

Simultaneously, Microsoft's pursuit of search, metadata and taxonomy has created equally powerful tools for unstructured content. Microsoft acquired FAST Search and Transfer near the time SharePoint2010 was released. That search technology has since been fully absorbed into SharePoint and offered as a service through Azure Search. With it, Microsoft is stitching together findability mechanisms such as Microsoft Delve, which shows contextual and automated results. With Microsoft Delve, instead of searching, content is displayed through a dashboard based on your work habits and interactions with others. For example, it automatically displays what you've been working on and which projects are relevant to your work by combining results from OneDrive, Exchange, and SharePoint.

Master data and enterprise entities

In business, master data management (MDM) provides a common point of reference for the processes, governance, policies, standards and tools used to manage enterprise content. Master data describes the concepts, products and services your business markets and sells. It can also give a comprehensive view of organizational data or isolate desired information, which helps when reviewing success among products, sales people or other entities.

You can access master and nonmaster data in a variety of ways. An external list, for example, is SharePoint's more evolved method of showing structured content. Using the successor to the Business Data Catalog (now called Business Connectivity Services), you can define these concepts and constructs, expose them in a list, and even allow two-way updates of certain data. In this way, structured and master data make their way into your document and collaboration management tools and, consequently, other opportunities emerge.

Bringing it all together

The combination of structured and unstructured data offers real opportunities in SharePoint 2013 and SharePoint Online. First, it can display structured and unstructured content within the same interface – such as displaying data visualizations alongside related documents further describing the data. Furthermore, new integrations with data sources like Google Analytics give business leaders a more holistic view of their enterprise and reduce the risk of making critical decisions with only partial information.

How does this all happen? It starts with basic business taxonomy. In the data world, this is a firm's master data. The terms and concepts that comprise important business concepts can be surfaced in SharePoint through either an external list or through the Term Store (managed metadata). Next, Power BI and BCS can visualize data across numerous systems within your firm; a complete view is now available for leaders.

Once the structured data is in place, search and other content display tools within SharePoint can show related documents, spreadsheets and presentations. Using Delve as a model, your IT team can configure SharePoint functionality, like the Content Query Web Part, to accept parameters -- like master data terms -- to filter unstructured content through concepts displayed in the data views. Instead of just seeing charts and graphs, executives are given access to the narrative that provides a deeper context behind the numbers.

While the technology supports what's described here, there are challenges. For most organizations, even common concepts about their products and services are often fragmented among systems. Many firms retain product, financial, marketing and sales data across many systems -- some of which may be bespoke. This creates several potential problems, including master data governance, integration difficulty, and access. Executives in these firms often receive hand-crafted reports resulting from a lot of manual work to provide data clarity. As such, creating a unified view in SharePoint, while technically feasible, can be daunting without clear vision and the will to first organize structured and unstructured content.


Firms that use SharePoint have a real opportunity to change the way they see themselves. By combining services like Power BI, Business Connectivity Services, Delve and search, organizations can develop more complete composite enterprise content views. Master data terms become the "tags" used to identify and relate unstructured content to the structured data. The combination of Web parts within SharePoint allow you to configure these interfaces simply by using parameters.

However, as previously mentioned, there are serious challenges to implementing this view. Firms wishing to undertake this kind of initiative should prepare to rationalize their structured data, create reliable integrations points into source systems and develop an enterprise taxonomy that can effectively be applied to both structured and unstructured data.

Next Steps

SharePoint usability no small issue

Migrating to SharePoint Online still a minority concept

Usefulness vs. feature-rich SharePoint?

Dig Deeper on Enterprise portal software strategy