Companies of all sizes derive competitive advantage from technology systems that in turn support numerous business applications. Many variations on software solutions populate different kinds of enterprises. Customer relationship management (CRM), sales force automation (SFA), enterprise resource planning (ERP), supply chain management (SCM), and so forth, provide necessary tools for most businesses. As implemented by enterprise-level, mid-market, and even SMB, companies, the complexity of applications continues to increase, as has the need for continuously integrating and transforming data that provide the life blood for these applications.
Enterprises probably are not spending enough time, money or resources on data integration (DI) software/services – part of the blame may fall to the broad range of options for DI solutions. The current data integration software/services market, while robust, is also highly fragmented and somewhat confusing. From Enterprise Data Warehousing (EDW) and Master Data Management (MDM) costing millions -- to data mashups, and dashboards based on Excel -- data integration software solutions span a broad spectrum of cost and complexity for all sizes of companies. DI offerings are proliferating to meet the demands of diverse data management and integration problems.
Companies are faced with making decisions about which DI solution, or solutions, to choose – while weighing current and future requirements, cost, usability, reusability, speed to solution. Instead of purchasing software, many times companies will simply try to build a DI process or solution based on custom code. Custom code has been cited by the industry as the #1 competitor for DI software offerings of any sort, for several years now. But custom code is usually not the best avenue to travel for most DI projects.
As has been noted by Rick Sherman, mid-market / SMB companies may not be aware of many newer DI tools that are the right fit for their needs and budgets, and therefore have not seriously pursued the purchase of data integration software solutions, instead of hand coding. The solutions that are most publicized are the high-end solutions from Oracle, IBM, Informatica, SAP-Business Objects, etc., so it may seem to mid-market and smaller companies that DI tools are expensive and difficult to use. The high-end vendors are also the usual focus of industry analyst benchmark reports, such as the Forrester Wave and the Gartner Magic Quadrant. But there are many DI software alternatives that could be the right fit for many companies, that are cheaper and easier to use.
(Originally a 2009 MQ was here - but Garnter demanded its removal. JH - 6/12/2014)
Larger Enterprise Data Integration Software Tools – Gartner definition for 2009 Magic Quadrant
Gartner has defined several classes of functional capabilities that vendors of data integration tools must possess to deliver optimal value to organizations in support of a full range of data integration scenarios:
·Connectivity/adapter capabilities (data source and target support)
·Data delivery capabilities
·Data transformation capabilities
·Metadata and data modeling capabilities
·Design and development environment capabilities
·Data governance capabilities (data quality, profiling and mining)
·Deployment options and runtime platform capabilities
·Operations and administration capabilities
·Architecture and integration
Does The Gartner MQ Help Most Companies Select Data Integration Tools?
With its focus on primarily high-end vendors and solutions, the Gartner MQ itself presents only a fraction of vendors that are operating in the overall competitive landscape. While the Gartner document has provided an extensive list of 40+ vendors that are NOT included in the MQ, there is not enough information to assist with buying decisions.
So the Gartner MQ for DI Tools focuses on solutions meant primarily for larger projects: EDW, MDM, CDI, B2B data exchanges. It’s not clear if any of these vendor offerings would be cost-effective for less-complex needs. The 2009 MQ does include some DI alternative vendors such as Talend (OSS, cloud options) and Pervasive (on-premise and SaaS/cloud options) – but again, there are many more options available that have not been covered by the Gartner MQ and that companies should definitely explore.
Data Integration Variations – lots of options, which to choose?
Integrating data in a way that is meaningful to each enterprise
No matter which integration tools and methods are adopted, data is still the key – whether handled in concrete or abstract fashion. Understanding from a strategic point of view, how the company’s data matters not only to immediate projects/processes, but how it impacts the long-term plans and success of the enterprise, is essential to understanding which tools will be the best choice. Until recently, most companies have used data integration solutions to solve tactical problems but it’s really time to focus on strategic planning and utilization.
Many larger enterprises have traditionally thought it best to build a foundation of data warehousing and ETL processes, which have proven to be expensive, time-consuming, and frequently insufficient for overall data access needs in the enterprise. This sort of DW foundation can be difficult to implement, maintain, change, or even use. Other difficult – and costly – data initiatives have been introduced into the larger enterprise: master data management (MDM), customer data integration (CDI), complex data. MDM in concept makes sense, but the pain of making it happen so far has kept many companies from implementations.
Companies of all sizes need alternative solutions and new methods that are faster to implement, easier to deploy and use, with more functionality based on OOTB and/or service-oriented architecture. There has been an increase in pre-packaged, lower-priced offerings that solve specific DI business problems – hopefully this trend will continue. Industry-specific integrations based on pre-fab templates and integration appliances for CRM, SFA, B2B data exchanges, are also increasing. All of these approaches usually provide tools/services for customizations.
Beyond the tools for effecting integration processes, easy-to-use tools are needed for simply understanding and planning strategic architecture for data access and applications, especially tools suitable for mid-market/SMB companies with fewer IT/developer resources. From the “big picture”, companies can design smaller, incremental projects and processes for essential data integration and management.
Important to the any-size company is the integrity of the data – “bad” data will render any process or application useless. Less expensive tools for data profiling and data quality must become available. Right now most data quality software solutions are expensive and complex. (And hopefully few enterprises are attempting to hand-code data quality processes, as most custom code approaches to DQ come up painfully short.)
Overall Data Integration Software Market Outlook Great
Data Integration initiatives, and the sale of DI tools, are proliferating in healthy numbers, especially compared to other software solution segments. This reflects the value of data accessibility for many company needs, including improving competitiveness via analytics/BI for any-size companies. Companies are setting priorities for obtaining better, cleaner, and more timely data out of their operational systems. Data integration software solutions usually provide the best path to achieve such goals.
Gartner’s Mark McDonald recently pointed out increases in DI-related spending, commenting that customers will invest "in business intelligence applications and information consolidation in order to raise enterprise visibility and transparency, particularly around sales and operational performance."
Phillip Russom of TDWI: "One of the interesting wrinkles here is that these (BI / DI solutions) are now being considered an infrastructure, whose maintenance needs occasional augmentation. This new mindset -- by both technical and business people -- which considers DI/DQ/MDM to be infrastructure [i.e., akin to networks and broadband] is helping these technologies get -- or keep -- resources, despite a lock-down on general IT spending."
An IDC report estimates that the worldwide data integration and access software market will grow to US$3.8 billion in 2012, reflecting 8.7 percent CAGR from 2007 to 2012.
In the 2009 Data Integration Tools Magic Quadrant, “Gartner estimates the size of the market for data integration tools at approximately $1.34 billion as of the end of 2008, and forecasts growth at a five-year compound annual rate of approximately 9.4%. While the forecast growth has been substantially curtailed due to current economic conditions, this growth rate is very healthy compared with most other software segments.” Gartner growth estimates calculated earlier in 2008 forecast the data integration tools market at nearly $2 billion by end of 2008, with projected 17% annual growth rate for software and supporting services revenue. (Market Trends: Data Integration Tools, Worldwide, 2007-2012)
About the author: Julie Hunt is an accomplished market intelligence analyst, providing strategic market and competitive insights for the software industry. Her 20+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing. Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt For more information: Julie Hunt Consulting – Market & Competitive Intelligence Services