Highly Competitive - software industry insights

Ungoverned Spaces: Innovating Information Flow

 

Agile Governance to Ensure that Reliable Information is Available Anywhere Anytime

  

A good many companies now have Information functions in the Cloud, for a variety of reasons. There is the purposeful selection of cloud services for the usage and management of data, content, records and documents – in other words, Information – that supplements or replaces systems inside the firewall. There is also the proliferation of data and content – more Information – created in many cloud-based sites not under the control of companies. But companies want and need all of this Information -- and also need energetic ways to govern all of this Information as it flows in and out of the spaces used by enterprises.

Going with the notion that for most enterprises knowledge is power, information-led business innovation and competitiveness is becoming a recognized path for potential success. In parallel, services in the cloud are talked about more and more as enablers of innovation, as an agile, dynamic and cost-effective environment for new business direction and experimentation.

With so much information created anywhere-anytime, information governance has a vital purpose: ensuring the flow of reliable information to the right people at the right time. This can also mean reducing former barriers and changing processes, to make more of this information and data are available to more people in the enterprise, sometimes including partners and customers. What should be emphasized is governance that aligns with how information is used in the enterprise, so that governance also connects to business value and corporate strategic goals.

So when we’re talking about innovating information governance, we’re no longer in a vacuum or in a silo, off somewhere in that “information department” – we’re talking about the entire enterprise of integrated business processes and management infrastructure, running on-premises, in the Cloud, and in between. Everyone in the enterprise has to be on board, from top to bottom, across teams, internally and externally, working in concert to enact effective information governance.

  

Priming Information Flow – Anywhere Anytime

There is no doubt that most companies must extend their information strategies to capitalize on highly relevant information that is continuously created in the cloud.  Strong information governance policies and practices must realistically enable the flow of information to continuously bring information to the enterprise -- to improve decision-making, to better understand current and desired customers, improve competitiveness, and support all significant goals and strategies in the enterprise. Information Governance can be a vital part of the ‘framework’ for business agility, responsiveness to competitive pressures, support for new product and services directions, and innovation shepherding.

Enterprises need to break free of old perceptions and implementations of information governance where silos and disconnectedness dominate, and where governance has been viewed as difficult, esoteric, and somebody else’s problem. Companies generally have not taken a holistic view of business information governance, its impacts across end-to-end processes or business units, or the technologies to support it. Instead enterprises have treated information as islands within specific system silos, and have taken the same approach to its governance. Information Governance overwhelms many – many get lost in the details.

If information has become essential to corporate success, then Information Governance must become part of a company’s overall business strategy.  An upper management team, comprised of cross-departmental members, has to set the high level ‘culture’ to enable agile Information Governance:  a corporate ‘infrastructure’ of cross-team collaboration, an enterprise-wide commitment to levels of excellence when connecting with and serving customers, recognition that content from social sites matters, the dynamic means to process all potentially useful data and content no matter the source, and the ability to provide on-demand data and analytics ready for any device.

Individual responsibilities for different components of Information Governance can be segmented to include most groups in the enterprise and to enhance accountability. Services-oriented Information Governance may be an optimal approach with partnerships across teams to enable governance functions and processes. Legal and financial teams could take key responsibility for risk management and compliance, as well as retention policies and practices. IT teams can take on key responsibilities for security, storage, resource provisioning, and user access. It is also now the responsibility of enterprise users of information to understand, partake in and help enforce policies and practices for security, information quality, delegated access, sharing, and so on – these responsibilities are no longer “somebody else’s problem”.

 

    Creating Information Advantage in a Cloudy World

 

Detours For Information Anywhere Anytime

Not surprisingly, persistent issues for Information Governance and Management are mostly the same, whether the repositories are in the cloud, on-premises, or in a hybrid flow based on the two:

 

• No clear holistic strategy for governance of distributed information sources
• Prolific growth of information sources and systems
• Need for extensive integration between information sources, often existing as silos
• Synchronizations to eliminate redundancies and to update with most current information
• Unreliable quality of information, content, data
• Timeliness of information
• Fragmentation that will block information flow and devalue assets

The creation of content and data in cloud repositories has added complexity to ownership and the chain of custody. Enterprises that strongly manage complex chains of information ownership and usage, both internally and externally, will help corporate goals and achieve advantage from all information assets.

Companies must take on the responsibility of monitoring vendors of cloud services to ensure that the vendor is meeting SLA provisions for security, privacy, backup and so on. This monitoring falls both to business users and IT groups. But companies must also develop comprehensive Information Governance plans to supplement what cloud service vendors provide. Companies are liable for proper information governance regardless of where the repositories sit. Business users as well as IT teams must understand the implications of “anywhere anytime” information flows.

 

            IBM: Enabling Multifaceted Innovation

 

Cloud Inspires Governance Innovation

Innovation often relates to finding new ideas and / or applying existing ideas in new ways to bring about beneficial change that affects many aspects of what an enterprise does. Innovation is not limited to the products and services in an enterprise, but also can be applied to business model and strategy, operations, employee and partner enablement.

Again, enterprises need to break out of seeing governance only in terms of risk containment and security (which are important) to also see how it can be a means to innovation. Governance is valuable for setting up policies and practices to guide information usage but it can also provide scope for experimentation and value creation through the orchestrated flow of information to and from many sources. Governance is evolving to handle the challenges of data in the cloud – extending information governance to cloud sources can also help innovate how governance is done. This is a new challenge for teams across the enterprise to work differently and creatively when governing vital information that must constantly flow, while ensuring that the enterprise meets reliability, compliance and risk requirements. Making accountability a distributed function across the enterprise extends the value of information governance and takes it out of its usual silo.

Timely, reliable information in the right hands is more essential as business cycles shorten and become more competitive, and as customer perspective and buy-in become more important. Information created in the cloud along with cloud services comprise a new platform for quickly sharing data and content that matters to the enterprise. “Working the problem” of information governance from the focus of ensuring the open flow of reliable information to be used by many teams in the enterprise anywhere-anytime is a positive move into innovation that will make a difference.

  

Related Links:

Information, Intelligence and Process:  Combining Forces to Better Answer Business Needs

The Ever-Expanding Importance of Analytics–True Intelligence for Business

Goals and Metrics – Measuring Customer Engagement on the Web

Findability Inside The Firewall – Still Trying To Find The Information We Need

The incongruous worlds of “data integration” – Making data accessible for any-size companies

Custom Code or Data Integration software tools: Strategic Use of Technology for Company Competitiveness

Shallow Thinking Ensures Bad Business Decisions

Connecting Data Governance to Business Outcomes That Matter

Data Management: Reaching into the Cloud

 

About the author: Julie Hunt is a software industry solution strategist and analyst, providing market and competitive insights. Her 25+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Strategic Product & Market Intelligence Services    

 

 

Connecting Data Governance to Business Outcomes That Matter

Originally published on Hub Designs Magazine September 2011

 

Process-centric initiatives are often optimal approaches for the things that companies want to accomplish. Business processes make it possible to directly connect work to business outcomes that lend themselves to various metrics that provide clarity for business performance. On the other hand, data and data-oriented solutions, by themselves, don’t usually map directly to business outcomes, and as such, usually can’t be measured by value contributed to desired outcomes and achievement of key corporate goals. But -- business processes and data are inextricably bound together. Processes create, modify and consume data – one might even say processes live and breathe data. It is through business processes that data can be mapped to business value.

Companies generally have not taken a holistic view of business information, technologies to improve it, or its impacts across end-to-end processes or businesses, instead treating data as information islands within specific system silos.

Kalido, The State of Data Governance Maturity 2011   

 

Notions like “Big Data” and “Information as an Asset” have stimulated thinking in many enterprises about extracting more value from the data already in enterprise repositories to benefit the business and its goals. But to extract that value from data, it’s not about the data nor is it about data governance. The key focus should be the business processes that impact revenue, business agility, competitiveness, and overall positive performance. However:  both reliable data and effective data governance are key enablers of gaining value from business processes.

Data governance is an important strategy both to maintain data in its most usable condition and to ensure consistency in how processes work effectively across the enterprise, by sustaining the quality, accessibility and relevance of data used by processes. Instead of many silo’d processes working on silo’d data, data governance contributes greatly to connecting essential pieces to synch significant business processes to the bigger enterprise strategic picture.

Martin Atherton on data governance as an enabler of business value:   

For many companies, it is common to focus on the minimum standards required to achieve compliance, instead of seeking to incrementally expand the effort and reap the benefits of being compliant more broadly across the business. In a previous research note, we used the phrase ‘value driven compliance’ to highlight the difference between approaching regulatory compliance as a chore versus seeing it as a business enabler.

 

Some of the difficulty of transitioning how enterprises derive value from business processes (and the data running through them) is the fact that most data-related initiatives have been driven by IT instead of by more business-oriented roles. To enable better business outcomes, data governance needs to bring business roles and IT teams together, as well as incorporating the support of various levels of management and strategic decision-makers. Too many times, data governance has not been implemented strategically. Instead it has been marooned by ad hoc decisions and “policies” tied more to daily operational situations. Without involving data governance in key corporate strategies, enterprises are not in good position to handle internal data issues, let alone address the ever-growing proliferation data from many new sources inside and outside the enterprise.

Ross Dawson expands on the notion of “governance as an enabler”: 

Governance should focus as much on enabling innovation and taking useful risks as about managing and mitigating risk...What needs to happen is that governance, risk, and compliance become part of the process of creating business value...More generally, I think it is important to note that IT governance and corporate governance have long been treated as separate. As I pointed out in my keynote this morning, technology – including in data management – is increasingly central to corporate strategy.

 

With good commitment from management, business units and IT teams alike, data governance initiatives are a very good way to manage data more effectively for greatly improved use in business processes. Data governance should provide the means to coordinate efficient and non-redundant business processes, along with the necessary data assets, across the enterprise, between lines of business. From data governance coordination, standards for reliable data and optimized business processes can be implemented, maintained and changed as needed. Data governance is also an enabler of acquiring important insights into both data and business processes.

 In my interactions with Forrester clients, I get the sense that data governance is receiving the most senior-management-level attention today than I've seen throughout my 18+ year data management career.

Rob Karel, Forrester

 

Tangible Value

Data alone carries no value. Unusable and poorly managed data carries no value. Real cost savings obviously result when data sources are streamlined and cleaned up, when processes are more responsive to business needs. Great value can result when business intelligence and other processes are based on the right data that can be trusted to lead to decisions that are more accurate and timely. The tangible benefits and value of data governance and data are further revealed through their connections to business processes and the desired outcomes from those processes that connect to corporate strategies and goals.

  

Related Links

Data Management: Reaching into the Cloud

Information, Intelligence and Process:  Combining Forces to Better Answer Business Needs

The Ever-Expanding Importance of Analytics–True Intelligence for Business

Goals and Metrics – Measuring Customer Engagement on the Web

The incongruous worlds of “data integration” – Making data accessible for any-size companies

Custom Code or Data Integration software tools: Strategic Use of Technology for Company Competitiveness

Levels of Social – and Integration – Cut Across Enterprises 

 

About the author: Julie Hunt is a software industry solution strategist and analyst, providing market and competitive insights. Her 25+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Strategic Product & Market Intelligence Services   

 

 

 

Data Management: Reaching into the Cloud

Originally published on Hub Designs Magazine August 2011

 

In a form of “shadow IT”, Line-of-Business groups have been turning to cloud-based services to quickly set up technology solutions that support their business needs and objectives. IT teams are already carrying heavy workloads with ever-shrinking staffing levels, and frequently don’t have the resources to immediately respond to time-sensitive LOB needs. However without IT involvement, these LOB groups do not usually have the expertise to understand the implications of creating data in the cloud and its relationship not only to data in other SaaS / cloud offerings, but to on-premises systems as well. These business users need a partnership with IT to ensure that comprehensive data management processes are put into practice.

It’s clear that enterprises will continue to increase usage of cloud and SaaS offerings to find new ways to operate more competitively and efficiently. Unfortunately the rapid creation and adoption of cloud and SaaS services have led to increased data silos in the cloud. The same challenges that enterprises face for on-premises data management obviously apply to data repositories in the cloud: data quality and reliability, integration, governance, security and compliance, MDM, as well as improved data accessibility. There is now a bigger picture for data management, where all of these domains need governance as an interoperating whole.

If enterprises fail to address a rigorous data management strategy for cloud computing initiatives, deterioration in data quality and reliability of cloud data are likely to kick in quickly. Data that cannot be trusted nor aligned with enterprise datasets will destroy the value and cost savings that enterprises want from cloud services. One of the most important requirements of business data generated through cloud services is security for that data. Not all data created in the cloud requires security, but most of it does. Comprehensive data management policies are needed in the cloud as well as for on-premises data repositories to ensure secure data processes.

The concerns of both business users and IT regarding data in the cloud have been articulated in many surveys including one recently completed by IDC where issues that relate directly to data governance and MDM initiatives covered security, data integration, operational costs and compliance – all pretty much to be expected when considering management strategies for cloud data. The IDC study goes on to discuss an important point that again requires IT and LOB groups working together: Business Relevance.

The next two challenges – the perceived difficulty of cloud services integration, and the limited ability to customize – are both related to the important issue of business relevance.  While customers certainly enjoy the economic and operational benefits of the off-the-shelf, standardized nature of many cloud services, this survey shows they nonetheless want greater ability to “fit” cloud services more tightly into the context of their specific business.

MDM and data governance provide clear value when they reflect the actual activities and processes of the enterprise. This approach also makes it easier for corporate decision-makers to understand the role MDM and data governance play to support strategic goals and even how they might map (at the high level) to key business processes. Malcolm Chisholm and Fabio Corzo flesh out these ideas in The True Role of MDM: The importance of master data is not just data values or quality, but the real-life concepts they represent… We are looking for fundamental business transformation that will take the business into new ways of perceiving, understanding and developing customers, employees, suppliers, assets and securities. We are looking for business transformation that moves in line with the business goals. And though executives understand that data is at the heart of proper and well-executed business transformation, experience tells us that more often than not, the business transformation is limited to improved data quality side effects that fail to realize the true value of an MDM program. The above concepts become even more imperative when establishing management of data-driven business processes between the cloud and on-premises systems. There will need to be an evolution of data management and governance approaches to deal with new challenges that come from data in the cloud and from an integrated approach to managing data across disparate data domains, many of which are not owned by the enterprise. Cloud services have been very effective in converting (or commoditizing) IT practices through well-formed repeatable processes. To accommodate proper data management for cloud and on-premises solutions, IT and business users will need to work together to build out data management solutions through practices and technologies that support the creation, use and storage of data “anywhere anytime”. Both IT and Business are important to setting the right strategies, including the crafting of a data management architecture that will kick in before an enterprise’s cloud data becomes too distributed for thorough governance. While more and more data will be owned by non-enterprise entities in the cloud, the ownership of comprehensive data management processes and practices continues to fall squarely on the enterprise through business users and IT working to ensure reliable data availability.     Related Links The incongruous worlds of “data integration” – Making data accessible for any-size companies Custom Code or Data Integration software tools: Strategic Use of Technology for Company Competitiveness Levels of Social – and Integration – Cut Across Enterprises  Information, Intelligence and Process:  Combining Forces to Better Answer Business Needs The Ever-Expanding Importance of Analytics–True Intelligence for Business Goals and Metrics – Measuring Customer Engagement on the Web     About the author: Julie Hunt is a software industry analyst, providing strategic market and competitive insights for vendors and buyers. Her 25+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Strategic Product & Market Intelligence Services

Information, Intelligence and Process: Combining Forces to Better Answer Business Needs

Originally published as a 4-part series on Hub Designs Magazine May 2011

Data governance can be seen as formalized policies, practices and processes set up to manage voluminous data assets across enterprises. Data governance also sits on an important growing convergence that encompasses multiple, and frequently separate, disciplines: data quality / data integration, master data management (MDM), business process management (BPM), business intelligence (BI) and analytics. The solutions surrounding the management of information, data and intelligence are emerging from artificial silos to acknowledge the greater overlap and interrelationships of these technologies and practices. Deliberate initiatives to tighten up the convergence of these solution spaces will not only improve the better overall functioning of all of these solutions but will help both IT and Business users see how it all works together. A greatly beneficial result should be the elimination of duplicate demands on business users and IT for implementing and managing these systems.

Forrester’s Rob Karel on aligning Business Process and Data Governance Initiatives:

Data governance is not – and should never have been – about the data.

High-quality and trustworthy data sitting in some repository somewhere does not in fact increase revenue, reduce risk, improve operational efficiencies, or strategically differentiate any organization from its competitors. It’s only when this trusted data can be delivered and consumed within the most critical business processes and decisions that run your business that these business outcomes can become reality.

So what is data governance all about? It’s all about business process, of course.

Let’s add this note to Karel’s assessment: business process is about optimizing the Business.

The current state of the increasing interrelationships between Data Governance, MDM, BPM, BI / Analytics (as well as DQ, DI) is also the story of the teaming up of Business users and IT as collaborative partners in doing the work that help businesses meet strategic goals, bring value to company customers, add to competitiveness. Both sides of the story must start with the business problems and the jobs to be done, plus the desired outcomes and benefits. Then decisions must be made regarding the role to be played by the software solutions to be implemented. Starting with the business also encourages the exploration and creation of the specific business cases that reflect needs and requirements. With such business cases defined, monitoring outcomes of data governance, BPM, BI, etc., should have clear basis.

While business processes are a major focus for strategies around data governance, BI and BPM, data still retains a significant role as a strategic asset that enterprises can use to differentiate themselves. These two entities (process and data) must be accorded respect when developing interrelated approaches that incorporate all of these solutions. Just as these solutions are best in synch with one another, so must the enterprise itself work with these solutions across departments and diverse teams. For business and intelligence processes to yield useful, timely and accurate results, the underlying data must be trustworthy.

Knocking down the Silos

Data, information and content permeate most enterprises where business users must become much more directly involved in creating the overall solution for handling these vital resources. Business users usually hold the key to the context for data and processes, as well as future usage considerations. The integrated components of MDM / data governance, BI / analytics, and BPM need to deliver real results to business users but are also highly dependent on participation of business users to reflect reality and provide on-target solutions. True collaboration between Business and IT teams starts with requirements, results, procedures, and then leads to collaborative work on joint projects. Such collaboration fosters reuse, reduces duplication of effort and processes, and builds a network of Business and IT staff that is aligned with how systems support key business objectives, initiatives and future direction.

Commonality also crosses the business drivers for data governance, MDM, BI, and BPM where key benefit areas consist of growing revenue, cost efficiency, different levels of agility, and compliance. Efforts should provide improvements for:

• Faster, more accurate decision-making - real-time intelligence
• Identifying and engendering new markets, new products, new customers
• Enhancing and supporting customer relationships / customer service
• Operational procedures
• Customer cross-selling / up-selling
• Ease of compliance, audits, risk management

Rajan Chandras, Data Gaps Plague Process Initiatives:

For one, the genesis of these types of initiatives is often very different. BPM is undertaken when organizations find themselves beset with process shortcomings, and with the primary purpose of better understanding, improving and integrating business processes. Data governance initiatives, on the other hand, are usually justified by shortcomings in data quality, consistency and integrity.

Then, data governance is often closely intertwined with master data management, and thus organized alongside important "master" business data entities -- customer, product, organization and such.

That said, Rob Karel, Forrester analyst and lead author of the report, suggests that IT must learn how to better educate and evangelize data issues in a language and a context that matters to the business -- a responsibility that Karel puts squarely on IT, "because IT often has a more cross-enterprise view than siloed business units and functions."

For data governance and process governance efforts to be successful, they both (business and IT) must frame their priorities and business value in the context of which business processes they are aiming to improve, transform and optimize, says Karel.

From Forrester study, September 2010

Source:http://alteryx.com/How-Alteryx-Performs/Pages/White-Papers/Forrester-Report-on-BI-Platform-Requirements.aspx

Intelligence, People and Process

MDM is a reflection of how an enterprise uses data, information – and content - for business purposes. The creation and management of master data touches more than the information itself. By creating data repositories and processes that reflect business functions, enterprises should develop access to the information that is so necessary to effectively and efficiently achieve business goals. It’s extremely important to analyze, manage and provide access to all forms of information – structured and semi-structured data, as well as “unstructured” content.

For BI and analytics solutions to provide real intelligence, they must be based on accurate and timely data sources. BI outputs are based on data aggregations, until recently, from structured data in data warehouses, with delivery in report or dashboard form to enterprise users. Analytics add the dimension of mathematics and formulas, with variants addressing forecasting and prediction.

Neil Raden responds to the ebizQ question: How does master data management change BI?:

…to describe how organizations collect data as part of a process, but manage to make it infinitely more valuable by using it for other purposes.

What does this have to do with BI and MDM? In our research, we have found that most knowledge workers shun BI for two reasons (not performance or ease-of-use): relevance and understanding. MDM adds nothing to address these concerns because its representational framework, a relational schema, is inadequate. It's a representation of a model. An ontology is a model. Until MDM rejects the relational model as its underlying schema, it will be unable to add the rich meaning, relationships and even reasoning that an ontology can do.

So, the point is, if you're going to go to the massive effort and expense of an MDM solution, take some advice from 21 years ago…Make the data useful for people, not just governors and black belts.

One newer area of interest for BI / analytics solutions is the inclusion of collaborative activities to add contextual and qualitative layers to the output of BI processes. To achieve authentic intelligence, contextual / qualitative layers can provide a strong basis to test, fine tune and filter the artifacts of analytics. Analytics can benefit greatly from human filters that bring experience, knowledge, creative thinking. Context has a big role here: context for sources, context for outcomes, context for usage with other data points, to achieve the optimal Intelligence for “making better business decisions”.

The possibilities for new applications of analytics increase with collaboration. Inviting in many-to-many interactions also opens up processes to new ideas from participants. Gartner found that social venues and collaboration help to track and capture outcomes of the decisions made based on BI / analytics:

Gartner's user surveys show that improved decision making is the key driver of BI purchases. However, most BI deployments emphasize information delivery and analysis to support fact-based decision making, but fail to link BI content with the decision itself, the decision outcome, or with the related collaboration and other decision inputs. This makes it impossible to capture decision-making best practices. Solutions are emerging that tie BI with social software and collaborative tools for higher-quality, more transparent decisions that will increase the value derived from BI applications.

With convergence, employees in the enterprise should operate more effectively, where improved data governance/MDM lead to better BI, where collaborative processes also enhance and validate BI, where a collaboration setting for BPM works to deliver more of the information that the enterprise needs. A possible approach to marrying collaboration, data and intelligence to business processes, and, more importantly, to the way people work, can be seen in Tibco’s re-work of tibbr. Dennis Howlett provides this description of tibbr:

It intelligently marries people, process and context, delivering information the way people want to consume

Tibco connects tibbr to business processes and event-triggering that are then exposed in tibbr for taking action. While tibbr is built more for real-time information streams than archives of content and information, tibbr and Tibco have created a platform with a lot of potential for improving overall information findability that adheres to context and worker roles. It connects to process, people/workers, collaboration venues, data and information streams, centralizing all event streams into one dashboard. Information can be organized by subject or topic, rather than by people. tibbr enables users to create, contribute to, and subscribe to the real-time event streams that matter most to them.

Convergence Clarifies Benefits and Value

In a recent Gartner report covering predictions for data integration and MDM:

Through 2015, 66% of organizations that initiate an MDM program will struggle to demonstrate the business value of it.

It’s probably not a stretch to extend this “struggle for value” to BPM and BI / analytics. But through the convergence of MDM with BI / analytics and BPM, an analyst like Sandy Kemsley sees increased business value, in this Loraine Lawson interview:

Lawson: It's become more important with MDM to identify pain points and a real business case. Yet MDM is still pretty IT-centered. Does connecting it to BPM allow it to be more business-focused as a project?

Kemsley: Absolutely – connecting it to BPM, and also connecting it to business intelligence and analytics, because that's one of the things that many business users are interacting directly with analytics, whether it's just simple reporting or dashboards or a more complex, drill-down. You can get better analytics if you have consistent data models and those consistent data models are going to be helped by MDM, that's one of the key business benefits. When you want to have a report that gathers data from a number of different systems, it's not going to be some huge, big, long project because somebody has to figure out what data maps onto what other data. It's going to be a much more straightforward thing because while the data may not all be in one place, at least it will be in a consistent format so that you can easily bring it together.

 

Individually these solutions can achieve some value and success for an enterprise, but at the cost of much duplicated effort and resources. In convergence, with initiatives for business improvement and better decision-making, an integration of BI/analytics, BPM, data governance and MDM has definite synergistic outcomes that can greatly increase business value and reduce wasted effort. Integrating business insights into process execution will significantly improve the quality of decisions and lead to continuous process optimization. And process events kicking off BI and analytics for real-time decision-making increases beneficial outcomes.

All of these solutions should be about solving frequently complex business problems and supporting the work to be done in the enterprise. Each solution brings a significant piece of the “big picture”: better decisions for the business with reliable data feeding both intelligence and process. Ultimately the biggest payoff is how this convergence feeds directly into business agility and continuous change to constantly fine tune sustainable competitiveness.

Related Links

The Ever-Expanding Importance of Analytics–True Intelligence for Business

Goals and Metrics – Measuring Customer Engagement on the Web

I have a confession to make: I am a Hybrid

Findability Inside The Firewall – Still Trying To Find The Information We Need

About the author: Julie Hunt is an accomplished software industry analyst, providing strategic market and competitive insights. Her 20+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing. Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt For more information: Julie Hunt Consulting – Strategic Product & Market Intelligence Services

 

The Ever-Expanding Importance of Analytics – True Intelligence for Business

“Intelligence” is a word much bandied about in enterprises. Frankly few enterprises use it properly to denote what constitutes authentic intelligence.

"Intelligence is nothing if not an institutionalized black market in perishable commodities"

Master spy novelist John LeCarré

While the above notion is intended to refer to information gathered by spies regarding enemies, this sort of thinking has colored how enterprises view intelligence: the secret information that will solve all problems and kill all enemies. It doesn’t matter if you’re talking business intelligence or market / competitive intelligence. The apparent allure of the military and spy industries for enterprises has badly skewed how many companies view intelligence.

The kind of intelligence that truly benefits enterprises derives from classic definitions of the word: learning, understanding, applying knowledge and experience to new situations to make better decisions. For both business intelligence and market/competitive intelligence, the frequent catchphrase is “making better business decisions”. Both involve data, analysis, recommendations. Both can have strategic and tactical applications. Both benefit from qualitative as well as quantitative dimensions. And both should strive for authentic INTELLIGENCE, not just resultant collections of information.

Business Intelligence methodologies and tech solutions have been used to generate actionable views of a company, frequently rooted in business operations. Until recently, BI outputs have been primarily reports – some might say too many non-contextual reports, many of which may be trivial, pointless, or worse, misleading. The BI world includes analytics, data mining, text mining and predictive analytics.

And sometimes BI is considered to be ‘competitive intelligence’, hence my references to market intelligence. Market/competitive intelligence and business intelligence should be complementary, but are not the same thing. In my view, a certain amount of BI output contributes ‘intelligent data’ for market intelligence initiatives. Overall, market intelligence gleans from many ‘data’ sources from many disparate sources inside and outside the firewall, both qualitative and quantitative, while filtered by specific strategic contexts. Until recently, BI has mostly derived from practices and technology that address structured data and business processes inside the firewall. BI will now have to include unstructured or content sources in analytics, to add in missing subtleties, context, and insight that cannot be pulled from structured data sources.

Analytics Everywhere

In addition to overt BI-style analytics, many other analytics solutions have been proliferating to help make better business decisions. Web analytics were silo’d for a while as relevant only to website usages. But web analytics now have many forms and have become very sophisticated. Data captured by web analytics is now important to many teams across the enterprise and is now becoming more integrated into other “business intelligence” endeavors.

With fast-growing social media applications, SM analytics are now emerging – again the data generated is essential and must be integrated into the overall intelligence base of the enterprise. Content analytics have become even more important as unstructured content proliferates from many sources.

These seemingly diverse analytics cannot be confined to individual silos in enterprises. “Intelligence for the Business” that results from all analytics is needed at LOB / department levels, as well as at the über-enterprise level. And mid-market enterprises need such business intelligence capabilities as well. So all kinds of analytic tools need to become more accessible to many roles and organizational segments that have not necessarily been targets of such solutions before.

Unfortunately, keeping pace with the enormous proliferation of data, and even with the various analytical outputs, is overwhelming many enterprises. More integration between what began as silos of analytics must now happen. Enterprises have to get better at the behind-the-scenes / get-your-hands-dirty work that is essential to make realistic and effective use of all the data (structured and unstructured) pouring in from so many sources. The Semantic Web will likely contribute overall to the integration of structured and unstructured data to add to intelligence relevance.

Predictive Analytics – The Bold New LOB Tool

Considered a subset of BI, but also present in other categories of analytics solutions, predictive analytics may become much more in demand than traditional BI solutions, especially with the new directions of innovative predictive analytics solution vendors.

Eric Segal of Information Management on predictive analytics – citing a marketing use case:

Predictive analytics is data mining technology that uses your customer data to build a predictive model specialized for your business. …The real trick is to find the best predictive model.

…a careful combination of predictors perform better customer prediction by considering multiple aspects of your customers and their behaviors. Predictive analytics finds the right way to combine predictors by building a model optimized according to your customer data.

Predictive analytics builds models automatically, but the overall business process to direct and integrate predictive analytics is by no means automatic - it truly needs your marketing expertise.

Predictive analytics can be part of “traditional” BI offerings – it’s also part of the web analytics world. Baynote provides ‘Adaptive Web’ solutions that include analytics, personalization and social search for customer-focused relevance on websites, to help optimize engagement and retention. Baynote’s Collective Intelligence Platform provides companies with predictive analytics based on the implicit patterns of customers visiting their websites. Omniture, WebTrends and other web analytics solutions also provide capabilities for predictive analytics.

Content Analytics, Content Intelligence

Enterprises have been challenged for years on the content intelligence front – it has been very difficult and very messy to extract meaning (and thus retaining context) from unstructured content to be used as “data”. The amount of enterprise content is enormous, with geometric expansion in play. For enterprises that publish content on the web, content analytics also offer the means to better connect with target markets and customers:

"Content analytics" can be seen as business intelligence (BI) for/from content, as text (rather than number) crunching that generates insights to improve business outcomes. The two practices, content analytics and BI, certainly share motivations. If you don't analyze your content/data, you may be missing opportunities and running risks.

For content publishers, analytics drives better targeted content delivery, expanded audiences, and secondary uses and new distribution channels. These outcomes add up to profit. On the flip side, they are matched by reduced risk and cost avoidance given possibilities for more complete, more accurate compliance screening, e-discovery, and storage management.

Analytics also boosts value for users. Semantic search, faceted navigation, and content annotation/enrichment create findability and improve user experience and value for users. They also let users treat content like data. Call the goal "content intelligence," enabled by "smart content."

Seth Grimes on Content Analytics

 

“Intelligence” Hits The Wall

Stephen Few’s graphic for “BI Has Hit the Wall” is quite expressive of where BI – and all analytics - need to head to deliver evolving value to enterprises. Stephen Few is a leading expert in creating effective data visualizations that communicate real intelligence and don’t just function as “eye candy”:

Contained in these early definitions was the seed of an inspiring vision that caused people like me to imagine a better world, but the business intelligence industry has done little to help us achieve the vision of the people who coined the term. When Thornton May was interviewing people for his book “The New Know”, he asked a prominent venture capitalist known for his 360-degree view of the technology industry what he thought of when he heard the phrase business intelligence. His response was “big software, little analysis.”

For information to be useful, we must explore it, analyze it, communicate it, and monitor it, but the BI industry’s attempts to support these activities with few exceptions have been tragically comical. The technology-centric, engineering-oriented perspective and skill set that has allowed the industry to build an information infrastructure is not what’s needed to support data sense-making. To use the data that we’ve amassed, a human-centric, design-oriented perspective and skill set is needed.

New World Focus on BI and Predictive Analytics

Some of the significant changes to analytics solutions relate to how final artifacts are “formatted”. There has been a growing transition from traditional report formats to interactive visualization tools, and to collaborative processes that enhance the final artifacts. More companies want BI to help with current and future needs and goals, rather than measuring the past. And these companies should be inviting in more and more individuals (via collaboration) to help refine the accuracy and contributions of analytics outputs. Newer on-demand BI / analytics solutions are providing “right now” intelligence, though still these are still works-in-progress, for the most part.

A number of articles lately have made the point that BI initiatives have not been highly successful for many customers, while making another important point: that “failure” actually is part of the BI refinement cycle. But for costly enterprise-style BI solutions, “failure” can be quite expensive. Gerry Brown of Bloor comments:

However, there is little doubt that the number of BI users overall is increasing. With low-cost operators such as QlikTech, Tableau, LogiXML, Pentaho and Jaspersoft you get an awful lot of BI software for $25,000. Conversely, most enterprise BI vendors don't traditionally get out of bed for deals of less than $50,000. Low cost (or 'free') BI software encourages trial and experimentation with little risk, and this is what most enterprises prefer. After all, BI project 'failures' are common.

Referring again to the graphic for “BI Has Hit the Wall”, Stephen Few sees that “the traditional BI software vendors and most of the industry’s thought leaders are stuck on the left side of the wall” – while the future of BI will come from more non-traditional BI solutions:

The software vendors that are providing effective data sense-making solutions—those that make it possible to work in the realm of analytics on the right side of the wall—have come from outside the traditional BI marketplace. Vendors like Tableau, TIBCO Spotfire, Panopticon, Advisor Solutions, and SAS tend to either be spin-offs of university research or companies that have ventured into the BI marketplace from a long history of work in statistics.

In a very interesting and thought-provoking Gartner Magic Quadrant for BI platforms, the analysts see significant changes in the solutions sought by enterprises:

…there is significant, if not euphoric, satisfaction with, and accelerated interest in, pure-play BI platforms. This is particularly true for smaller, innovative vendors filling needs left unmet by the larger vendors. To understand this paradox, it is necessary to consider a number of factors that are driving the BI platform buying decision today.

Other vendors to watch for new ways to perform and use BI / analytics include Lyzasoft, Predixion Software and GoodData. Lyzasoft and Predixion recently became partners for cloud to cloud analytics-as-a-service. GoodData provides BI PaaS to help speed up solution deployments.

BI SaaS adoption, while very low today, will grow steadily as maturing BI SaaS solutions are delivered in private and public clouds and in on-premises and off-premises configurations by trusted vendors. …innovative, pure-play vendors offering highly interactive and graphical user interfaces built on alternative in-memory architectures to address their unmet ease-of-use and rapid deployment needs. The perceived benefit is so compelling that business users are making this choice, despite the risk of creating fragmented silos of applications and tools.

Business users in particular showed a growing impatience with the time to deploy and complexity of traditional enterprise tools, which led to a rise in departmental buying of alternatives.

Collaboration and the Human Element

One newer area of interest for BI / analytics solutions is the inclusion of collaborative activities to add contextual and qualitative layers to the output of BI processes. To achieve authentic intelligence, contextual / qualitative layers can provide strong basis to test, fine tune and filter the artifacts of analytics. Analytics can benefit greatly from human filters that bring experience, knowledge, creative thinking. Context has a big role here: context for sources, context for outcomes, context for usage with other data points to achieve the best Intelligence for “making better business decisions”.

Collaboration is far more than distributing and sharing documents. It is interactive, inclusive, cross-enterprise. If collaboration is to be part of analytics, then iterative collaborative processes should be established throughout analytics cycles. Collaboration for analytics means bringing in disparate people to test assumptions, validity of data sources, accuracy and relevance of the outputs. These additional participants should provide a wealth of experience, complementary concepts and other essential data and perspectives. Having to prove the relevance and accuracy of analytics results to sympathetic as well as less sympathetic individuals should strengthen BI / analytic processes and projects.

The possibilities for new applications of analytics increase with collaboration. Inviting in many-to-many interactions also opens up processes to new ideas from participants. Gartner found that social venues and collaboration help to track and capture outcomes of the decisions made based on BI / analytics:

Gartner's user surveys show that improved decision making is the key driver of BI purchases. However, most BI deployments emphasize information delivery and analysis to support fact-based decision making, but fail to link BI content with the decision itself, the decision outcome, or with the related collaboration and other decision inputs. This makes it impossible to capture decision-making best practices. Solutions are emerging that tie BI with social software and collaborative tools for higher-quality, more transparent decisions that will increase the value derived from BI applications.

Vendors embracing collaboration as an essential part of BI / analytics include Lyzasoft, Predixion and Tibco Spotfire. Keep in mind that these vendors are all in early stages of building in collaboration as a significant aspect of their analytics solutions.

Analytics Aren’t Always Fun and Games

While there are very interesting analytics solutions that are “friendly” to business users, it is essential that safeguards are in place to ensure that business users understand what they are doing with analytic models and whether the resulting “intelligence” artifacts are correct, meaningful and useful. Collaboration with others who know the data, understand analytics, visualize the big picture, and so on, is one safeguard to ensure reliable analytics outcomes and correct usages.

In a particular enterprise, do enough people know what to do with analytics, both to start processes in meaningful ways and to audit outcomes to validate accuracy and relevance? Are users chasing the right problems or questions, providing the right data sources, including enough pieces? Do they have the understanding to work in a “big-picture” sense? Yes, analytics vendors should build in methods for validation, testing, guidance, but is that enough? Bob Warfield addresses this concern:

Few enterprises have the right analytical talent.

I was musing not long ago with VC and fellow EI Evangelos Simoudis that very few people actually know how to ask questions in a way that solves problems. It is something of a Sherlock Holmes conundrum. All the data is available. It is shatteringly obvious once someone connects the dots. Yet, very few know how to step across the stones that peek above the raging torrent of data to get to the other side where the answer lies without falling in and getting wet.

“You see Watson, but you do not observe.”

Beyond the traditional users of BI / analytics, there is an additional “challenge” to vendors to reach potential users who “need predictive analytics but don’t know it”. The challenge comes in two parts: first to find them and connect meaningfully with them; and then to provide guidance for every step of using the analytics solution to ensure proper outcomes. To connect with these “new” analytics users, vendors have to help them understand why they need analytics and how to use it. Self-serve training modules must be available for using the solution but also for understanding what analytics mean to the user’s role, industry and business needs. Detailed templates and best practices organized by industry scenarios are also important for success. It will be interesting to see how successful various vendors are for empowering such non-traditional users.

The Elephant in the Room: Reliable Structured Data

Although I’m not addressing data warehousing in this article, I do want to point out the need for basing any data-driven intelligence / analytics processes on reliable data. While it’s the right direction to provide analytics tools that work well for business users and increase ease-of-implementation and ease-of-use, diligence must be exercised to guarantee that data sources fed into analytic processes are trustworthy. Otherwise GIGO.

Teams comprised of IT and business must collaborate to create and maintain excellent processes for data profiling, data cleansing / quality, and data integration. The business users know how source data is used and the context for analytic processes. IT is the partner to help with the data profiling, data quality and integration steps. Data profiling tools can be used by business users to audit data sources, test for errors and/or for the lack of the right data, and to understand different data sources.

"Intelligence is nothing if not an institutionalized black market in perishable commodities"

A lot of data and content that matter for better “Intelligence for the Business” come with a short shelf life of significance. The new direction for many analytics solutions should bring this information more quickly into intelligence processes. When enterprises succeed in achieving richly integrated knowledge and context, authentic INTELLIGENCE is likely to result for “making better business decisions”.

Related Links

Market Intelligence – Making Better Business Decisions at the Core of the Company

Shallow Thinking Ensures Bad Business Decisions

Software companies still fail to tap into top benefits from market & competitive intelligence programs

Levels of Social – and Integration – Cut Across Enterprises

The incongruous worlds of “data integration” – Making data accessible for any-size companies

Custom Code or Data Integration software tools: Strategic Use of Technology for Company Competitiveness

Content Marketing Strategy for B2B Software Vendors: Starring the ‘New’ White Paper

Disclaimer: Julie Hunt is not affiliated with any of the vendors in this article.

About the author: Julie Hunt is an accomplished software industry analyst, providing strategic market and competitive insights. Her 20+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing. Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt For more information: Julie Hunt Consulting – Strategic Product & Market Intelligence Services

Custom Code or Data Integration software tools: Strategic Use of Technology for Company Competitiveness

 

 

Data Integration software solutions have been in the spotlight these past few years as the explosion of BI and data-driven businesses processes push through companies of all sorts.  It has become clearer how essential trustworthy and timely data is to business, and how important it is for Business users as well as IT teams to participate in decisions regarding data-driven processes, data access, data integration, and overall data quality. Equally important is the collaboration between IT and Business teams as part of the process to select the right data integration software solutions. Business users of applications, and the underlying data, understand the business context and utilization for that data in different systems, and can help decide advantageous ways to manage access and integration strategies.

 

Until very recently, data integration activities belonged almost exclusively to the IT domain. And for some time, data integration has become much more than moving data between disparate systems. While there are many DI software offerings for many different kinds of projects, IT groups have frequently used custom code instead of purchasing DI tools.

 

When focusing on DI software offerings versus custom code, a deeper, important issue for many enterprises comes forward: the need to align IT and Business teams to work on data initiatives from strategic business perspectives, not just from technology infrastructure perspectives. Many line-of-business (LOB) managers are taking on more responsibility for IT projects, setting goals, outlining requirements, providing insight from the business POV, and providing cost justification for projects. These recent changes add momentum to using technology more strategically -- but must involve IT as the key partner in running business processes. Data integration projects can benefit greatly from this sort of dynamic shared ownership.

 

Strategic collaborations between Business and IT for DI initiatives tie directly to the 80/20 IT spend “paradigm”: companies continue to report a problematic metric where 80% of annual IT spend/resources are dedicated to the maintenance of the status quo, while only 20% has been ponied up to drive tech innovations that would likely benefit the competitiveness of the business. With the fast pace of business change and disruption, agility and efficiency in tech utilization are key to building long term success and sustainability. 

 

Transitioning data integration projects from custom code to implementing more DI software tools that provide the majority of the solution - with much less coding – lends significant momentum to help swing IT resource allocation towards innovation rather than maintenance. The reusability aspects and other value points of DI tools add weight to this shift.

 

IT Culture of Custom Code for DI Projects – Tactical status quo?

In a 2008 interview with Philip Russom, TDWI, Lorraine Lawson for IT Business Edge, Russom indicated that at least 50% of DI projects are still hand coded, even though IT managers understand the clear, and frequently significant, savings that usually result from implementing DI software solutions:

“Whenever he explains his case, people tell him: Wow, I get the economics. Wow, thank you for enlightening me. BUT we still have this culture of hand-coding.”

 

Lorraine Lawson followed with additional posts in 2009 – here summarizing the comments generated by the discussion of custom code for data integration projects:

“Last week, I wrote a post on Why Companies Still Hand Code, But Don't Have To, despite the numerous data-integration tools of all price ranges available on the market. In the piece, I summed up the reasons cited by Rick Sherman, founder of Athena IT Solutions, which included ignorance of market costs, a lack of resources and a misplaced adherence to corporate standards.

“Several readers commented on the post, adding that Sherman overlooked one critical reason why so many data-integration projects are hand coded: The human factor.

“When it comes to data integration, the missing human factor is simply that developers like to code. They're comfortable with the tools and, in that moment, it's easier to hand code than evaluate, choose, buy, install, learn and use a new tool – even if that tool might shave months off projects on down the line.”

 

The seeming preference for custom code does point to a weakness in the differentiation of certain DI software offerings:  these tools are still designed to be used as “developer tools”, and indeed involve an amount (sometimes a lot) of coding themselves. Why would developers adopt a DI solution platform that on the surface looks quite a bit like something else that they are already using? Obviously most DI software tools offer a lot more than a “code platform” -- but to have clear differentiation, DI software vendors need to offer more focused solutions that solve specific business problems for specific customer scenarios.  DI tools need to become faster to learn, easier to use, with ready-to-go components, instead of providing general platforms. For better adoption, DI software offerings should be tools that any team will want to use over and over. 

 

The very important functions of data profiling and data quality are significant components of data integration software solutions that cannot be emulated very well by custom code.  DP and DQ processes help derive more value from data and to help render data more trustworthy and usable. Data profiling tools in particular provide a platform for both Business and IT usage and collaboration. DP and DQ capabilities have become important differentiators for DI software offerings, provided they are cost-effective and lean towards ease-of-use.

 

Vendors: Market DI software offerings to be “more findable”

There are DI software tools aplenty for any sort of company initiative – and any sort of company.  They can be on-premise licensed, SaaS subscription, cloud-based metering, open source.  They may manifest as platforms, appliances, mashups, dashboards, APIs, templates, pre-packaged industry-specific solutions, and so on.

 

Since major analyst publications primarily cover only a select group of high-end DI / ETL tools, the notion that companies and their IT and Business teams are not aware of the many options available has validity. However this is really an opportunity for DI software vendors to do more to make their solutions much more “findable” for their target markets and customers. With the upsurge of interactive marketing and social media for B2B selling, “findability” provides software vendors methods to improve the ability of the right customers to connect with the vendor.

 

Competitive advantage can be gained for vendors who do a better job of marketing and connecting what the tools offer to customer business problems. So it’s not just a matter of being findable as a “brand” or category of DI software, but findable as the right fit for a specific DI / business problem, and for a specific customer segment (a segment that could be Business and/or IT). DI software vendors also own the ability to do a better job of providing good solutions that are ready-to-go, to handle specific industry scenarios. No more one-size fits all type of DI platforms – instead, differentiate from developer code platforms by offering focused tools that solve pre-defined problems, solutions that target specific markets/industries/scenarios.

 

When DI software vendors have done a good job of revealing solutions that matter to their markets while showing that these solutions truly solve important problems, the intrinsic value of those DI software offerings should be a “slap in the face” for targeted Business and IT users. The value should be so apparent that IT will rarely want to hand code DI projects again – at least not the important ones.

 

Clear strategic purpose, distinct solution context, and a customer-centric orientation can take DI software further into helping IT change the usual 80% allocation for the tactical-for-status-quo – on to achieving the goal of a larger chunk of resources dedicated to competitive innovation. For DI projects, connecting IT to Business users of company data empowers everyone to work smarter. Savvy companies want to get more from their IT investments and want to rely on their data assets for competitive advantage. Through collaboration the company benefits: Business users know what they need from data and understand the context for data; IT provides the know-how to implement the DI processes to make it all happen. 

 

 

 

Flipping the 80/20 IT spend paradigm – understanding the TCO of custom code for DI

  

Talend Open Source DI Looks at ROI of Custom Code

Whether you agree with all of the Talend conclusions, this study introduces concrete parameters and data about the real cost of custom-coded DI projects.  One valuable takeaway is that this study shows the need to evaluate custom code on the same basis as any software offerings – and why. I find the Talend data believable, since most IT groups overlook lots of hidden costs related to custom coded projects when conducting comparisons to acquiring DI software tools.  Talend is at least opening the door on points that should be considered when selecting the right DI solution. 

 

Further details can be found in the complete report, The Return on Investment of Open Source Data Integration, 2008.

 

Talend ROI Project Category:  Large project: real-time enterprise data integration

· The "Large" project is a complete (large) enterprise data integration project.

· Twelve different databases, two packaged applications (one ERP and one SCM), an application from a partner (accessed through Web Services), a LDAP directory, and a Software-as-a-service CRM all need to be kept synchronized in real-time.

· This project involves 200 complex data flows.

  

Cost to complete this project using custom code:

 

 

Summary of 3 approaches to completing the “large DI project”

 

 

And finally:

 

 

 

About the author: Julie Hunt is an accomplished market intelligence analyst, providing strategic market and competitive insights for the software industry. Her 20+ years as a software professional range from the very technical side  to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Market & Competitive Intelligence Services  

 

The incongruous worlds of “data integration” – Making data accessible for any-size companies

 

Companies of all sizes derive competitive advantage from technology systems that in turn support numerous business applications. Many variations on software solutions populate different kinds of enterprises. Customer relationship management (CRM), sales force automation (SFA), enterprise resource planning (ERP), supply chain management (SCM), and so forth, provide necessary tools for most businesses.  As implemented by enterprise-level, mid-market, and even SMB, companies, the complexity of applications continues to increase, as has the need for continuously integrating and transforming data that provide the life blood for these applications.

Enterprises probably are not spending enough time, money or resources on data integration (DI) software/services – part of the blame may fall to the broad range of options for DI solutions. The current data integration software/services market, while robust, is also highly fragmented and somewhat confusing.  From Enterprise Data Warehousing (EDW) and Master Data Management (MDM) costing millions -- to data mashups, and dashboards based on Excel -- data integration software solutions span a broad spectrum of cost and complexity for all sizes of companies.  DI offerings are proliferating to meet the demands of diverse data management and integration problems.

 

Companies are faced with making decisions about which DI solution, or solutions, to choose – while weighing current and future requirements, cost, usability, reusability, speed to solution.  Instead of purchasing software, many times companies will simply try to build a DI process or solution based on custom code. Custom code has been cited by the industry as the #1 competitor for DI software offerings of any sort, for several years now. But custom code is usually not the best avenue to travel for most DI projects.

 

As has been noted by Rick Sherman, mid-market / SMB companies may not be aware of many newer DI tools that are the right fit for their needs and budgets, and therefore have not seriously pursued the purchase of data integration software solutions, instead of hand coding.  The solutions that are most publicized are the high-end solutions from Oracle, IBM, Informatica, SAP-Business Objects, etc., so it may seem to mid-market and smaller companies that DI tools are expensive and difficult to use.  The high-end vendors are also the usual focus of industry analyst benchmark reports, such as the Forrester Wave and the Gartner Magic Quadrant. But there are many DI software alternatives that could be the right fit for many companies, that are cheaper and easier to use. 

 

 

Source: Gartner

 

 

 

Larger Enterprise Data Integration Software Tools – Gartner definition for 2009 Magic Quadrant

Gartner has defined several classes of functional capabilities that vendors of data integration tools must possess to deliver optimal value to organizations in support of a full range of data integration scenarios:

· Connectivity/adapter capabilities (data source and target support)

· Data delivery capabilities

· Data transformation capabilities

· Metadata and data modeling capabilities

· Design and development environment capabilities

· Data governance capabilities (data quality, profiling and mining)

· Deployment options and runtime platform capabilities

· Operations and administration capabilities

· Architecture and integration

· Service-enablement capabilities

 

Does The Gartner MQ Help Most Companies Select Data Integration Tools?

With its focus on primarily high-end vendors and solutions, the Gartner MQ itself presents only a fraction of vendors that are operating in the overall competitive landscape.  While the Gartner document has provided an extensive list of 40+ vendors that are NOT included in the MQ, there is not enough information to assist with buying decisions. 

 

So the Gartner MQ for DI Tools focuses on solutions meant primarily for larger projects:  EDW, MDM, CDI, B2B data exchanges. It’s not clear if any of these vendor offerings would be cost-effective for less-complex needs. The 2009 MQ does include some DI alternative vendors such as Talend (OSS, cloud options) and Pervasive (on-premise and SaaS/cloud options) – but again, there are many more options available that have not been covered by the Gartner MQ and that companies should definitely explore.

 

 

 

Data Integration Variations – lots of options, which to choose?

 

 

 

 

Integrating data in a way that is meaningful to each enterprise

No matter which integration tools and methods are adopted, data is still the key – whether handled in concrete or abstract fashion. Understanding from a strategic point of view, how the company’s data matters not only to immediate projects/processes, but how it impacts the long-term plans and success of the enterprise, is essential to understanding which tools will be the best choice.  Until recently, most companies have used data integration solutions to solve tactical problems but it’s really time to focus on strategic planning and utilization. 

 

Many larger enterprises have traditionally thought it best to build a foundation of data warehousing and ETL processes, which have proven to be expensive, time-consuming, and frequently insufficient for overall data access needs in the enterprise. This sort of DW foundation can be difficult to implement, maintain, change, or even use. Other difficult – and costly – data initiatives have been introduced into the larger enterprise: master data management (MDM), customer data integration (CDI), complex data.  MDM in concept makes sense, but the pain of making it happen so far has kept many companies from implementations.

 

Companies of all sizes need alternative solutions and new methods that are faster to implement, easier to deploy and use, with more functionality based on OOTB and/or service-oriented architecture.  There has been an increase in pre-packaged, lower-priced offerings that solve specific DI business problems – hopefully this trend will continue.  Industry-specific integrations based on pre-fab templates and integration appliances for CRM, SFA, B2B data exchanges, are also increasing. All of these approaches usually provide tools/services for customizations.

 

Beyond the tools for effecting integration processes, easy-to-use tools are needed for simply understanding and planning strategic architecture for data access and applications, especially tools suitable for mid-market/SMB companies with fewer IT/developer resources. From the “big picture”, companies can design smaller, incremental projects and processes for essential data integration and management.

 

Important to the any-size company is the integrity of the data – “bad” data will render any process or application useless.  Less expensive tools for data profiling and data quality must become available.  Right now most data quality software solutions are expensive and complex. (And hopefully few enterprises are attempting to hand-code data quality processes, as most custom code approaches to DQ come up painfully short.)

 

Overall Data Integration Software Market Outlook Great

Data Integration initiatives, and the sale of DI tools, are proliferating in healthy numbers, especially compared to other software solution segments.  This reflects the value of data accessibility for many company needs, including improving competitiveness via analytics/BI for any-size companies.  Companies are setting priorities for obtaining better, cleaner, and more timely data out of their operational systems.  Data integration software solutions usually provide the best path to achieve such goals.

 

Gartner’s Mark McDonald recently pointed out increases in DI-related spending, commenting that customers will invest "in business intelligence applications and information consolidation in order to raise enterprise visibility and transparency, particularly around sales and operational performance."

 

Phillip Russom of TDWI: "One of the interesting wrinkles here is that these (BI / DI solutions) are now being considered an infrastructure, whose maintenance needs occasional augmentation. This new mindset -- by both technical and business people -- which considers DI/DQ/MDM to be infrastructure [i.e., akin to networks and broadband] is helping these technologies get -- or keep -- resources, despite a lock-down on general IT spending."

 

An IDC report estimates that the worldwide data integration and access software market will grow to US$3.8 billion in 2012, reflecting 8.7 percent CAGR from 2007 to 2012.

 

In the 2009 Data Integration Tools Magic Quadrant, “Gartner estimates the size of the market for data integration tools at approximately $1.34 billion as of the end of 2008, and forecasts growth at a five-year compound annual rate of approximately 9.4%. While the forecast growth has been substantially curtailed due to current economic conditions, this growth rate is very healthy compared with most other software segments.”  Gartner growth estimates calculated earlier in 2008 forecast the data integration tools market at nearly $2 billion by end of 2008, with projected 17% annual growth rate for software and supporting services revenue. (Market Trends: Data Integration Tools, Worldwide, 2007-2012)

 

 

 

About the author: Julie Hunt is an accomplished market intelligence analyst, providing strategic market and competitive insights for the software industry. Her 20+ years as a software professional range from the very technical side  to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Market & Competitive Intelligence Services