Highly Competitive - software industry insights

Cloud Expands Value of Purpose-Built Data Management Services

 

The ongoing value of purpose-built technology products stems from the savings in productivity and cost that come from delivering only the features and functions needed to get a certain job done. Purpose-built software solutions are an opportunity for cloud service providers to deliver well-differentiated services that are built on domain expertise for explicit problems related to specific industry scenarios. This approach often results in a strong competitive market presence for the cloud services provider.

As data management platforms and solutions continue to emerge as cloud services, purpose-built data management capabilities can be an important area both for the providers and consumers of cloud services. Data integration, in particular, has taken on immense importance due to the continuously proliferating data silos in the cloud and the need to integrate cloud data with on-premises systems. Purpose-built data integration is usually designed to meet a well-defined business need, with specific capabilities bundled in the solution, frequently delivered as an appliance.

Taking the purpose-built route for data integration services aligns with top advantages of cloud computing:

• Up and running quickly to handle integration tasks
• Enterprise-quality capabilities available to organizations of all sizes
• Always-on availability from any device or location
• The most current capabilities, without the need to purchase new licenses or new versions
• Pay-as-you-go, pay for only what is used

There is clear advantage for cloud services that support specific point-to-point integrations that can be performed easily and inexpensively, enabling less-technical business users to handle certain integrations between applications. A common integration need involves CRM and ERP systems, such as data flows between Salesforce.com and SAP. Services are built for focused use cases where business users do not have to make lots of decisions about how the integrations will be accomplished. Wizards are often employed to walk users through the process. However, business users must have good knowledge of the source and target datasets, and basic understanding of the data integration task, to work confidently and accurately through the purpose-built approach.

Cloud services for integration usually include data replication, migration, and synchronizations for customer data. Typically integrations involve certain cloud / SaaS applications as "pre-set" sources and targets, as well as on-premises systems. Data integration cloud services should also include data quality tools to involve business users in this important task.

Data integration solutions that can handle data located anywhere, as well as security and on-demand considerations, are important for connecting cloud services to on-premises systems. For many companies, the business technology that underpins those organizations is going through a transition from an infrastructure of all systems inside the firewall to more of a "hybrid" infrastructure comprised of part cloud-based services and part on-premises.

However, not all integration processes can be reduced to "simplicity". That's why data integration is still a tough nut to crack. In these situations, integrations will require the help of IT teams, consultants, and/or managed services using more sophisticated approaches. For organizations with broad needs, purpose-built data integration quickly becomes less useful as business needs change and the types of projects fluctuate. For such needs, a comprehensive and flexible data integration platform usually provides the best approach to handle current projects and to scale to address future integration challenges.

 

 

 

 

 

 

 

 

 

 

 

 

Image source: pomeroy.com

 

 

 

This post was brought to you by IBM for Midsize Business and opinions are my own. To read more on this topic visit IBM's Midsize Insider. Dedicated to providing businesses with expertise, solutions and tools that are specific to small and midsized companies, the Midsize Business program provides businesses with the materials and knowledge they need to become engines of a smarter planet.

 

About the author: Julie Hunt understands the overlap and convergence of many business processes and software solutions that once were thought of as "separate" – and how this impacts software Vendors and Buyers, as well as the strategies that enterprises implement for how technology supports the business and its customers. Julie shares her takes on the software industry via her blog Highly Competitiveand on Twitter: @juliebhunt For more information: Julie Hunt Consulting – Strategies for B2B Software Solutions: Working from the Customer Perspective

 

Data Management: Reaching into the Cloud

Originally published on Hub Designs Magazine August 2011

 

In a form of “shadow IT”, Line-of-Business groups have been turning to cloud-based services to quickly set up technology solutions that support their business needs and objectives. IT teams are already carrying heavy workloads with ever-shrinking staffing levels, and frequently don’t have the resources to immediately respond to time-sensitive LOB needs. However without IT involvement, these LOB groups do not usually have the expertise to understand the implications of creating data in the cloud and its relationship not only to data in other SaaS / cloud offerings, but to on-premises systems as well. These business users need a partnership with IT to ensure that comprehensive data management processes are put into practice.

It’s clear that enterprises will continue to increase usage of cloud and SaaS offerings to find new ways to operate more competitively and efficiently. Unfortunately the rapid creation and adoption of cloud and SaaS services have led to increased data silos in the cloud. The same challenges that enterprises face for on-premises data management obviously apply to data repositories in the cloud: data quality and reliability, integration, governance, security and compliance, MDM, as well as improved data accessibility. There is now a bigger picture for data management, where all of these domains need governance as an interoperating whole.

If enterprises fail to address a rigorous data management strategy for cloud computing initiatives, deterioration in data quality and reliability of cloud data are likely to kick in quickly. Data that cannot be trusted nor aligned with enterprise datasets will destroy the value and cost savings that enterprises want from cloud services. One of the most important requirements of business data generated through cloud services is security for that data. Not all data created in the cloud requires security, but most of it does. Comprehensive data management policies are needed in the cloud as well as for on-premises data repositories to ensure secure data processes.

The concerns of both business users and IT regarding data in the cloud have been articulated in many surveys including one recently completed by IDC where issues that relate directly to data governance and MDM initiatives covered security, data integration, operational costs and compliance – all pretty much to be expected when considering management strategies for cloud data. The IDC study goes on to discuss an important point that again requires IT and LOB groups working together: Business Relevance.

The next two challenges – the perceived difficulty of cloud services integration, and the limited ability to customize – are both related to the important issue of business relevance.  While customers certainly enjoy the economic and operational benefits of the off-the-shelf, standardized nature of many cloud services, this survey shows they nonetheless want greater ability to “fit” cloud services more tightly into the context of their specific business.

MDM and data governance provide clear value when they reflect the actual activities and processes of the enterprise. This approach also makes it easier for corporate decision-makers to understand the role MDM and data governance play to support strategic goals and even how they might map (at the high level) to key business processes. Malcolm Chisholm and Fabio Corzo flesh out these ideas in The True Role of MDM: The importance of master data is not just data values or quality, but the real-life concepts they represent… We are looking for fundamental business transformation that will take the business into new ways of perceiving, understanding and developing customers, employees, suppliers, assets and securities. We are looking for business transformation that moves in line with the business goals. And though executives understand that data is at the heart of proper and well-executed business transformation, experience tells us that more often than not, the business transformation is limited to improved data quality side effects that fail to realize the true value of an MDM program. The above concepts become even more imperative when establishing management of data-driven business processes between the cloud and on-premises systems. There will need to be an evolution of data management and governance approaches to deal with new challenges that come from data in the cloud and from an integrated approach to managing data across disparate data domains, many of which are not owned by the enterprise. Cloud services have been very effective in converting (or commoditizing) IT practices through well-formed repeatable processes. To accommodate proper data management for cloud and on-premises solutions, IT and business users will need to work together to build out data management solutions through practices and technologies that support the creation, use and storage of data “anywhere anytime”. Both IT and Business are important to setting the right strategies, including the crafting of a data management architecture that will kick in before an enterprise’s cloud data becomes too distributed for thorough governance. While more and more data will be owned by non-enterprise entities in the cloud, the ownership of comprehensive data management processes and practices continues to fall squarely on the enterprise through business users and IT working to ensure reliable data availability.     Related Links The incongruous worlds of “data integration” – Making data accessible for any-size companies Custom Code or Data Integration software tools: Strategic Use of Technology for Company Competitiveness Levels of Social – and Integration – Cut Across Enterprises  Information, Intelligence and Process:  Combining Forces to Better Answer Business Needs The Ever-Expanding Importance of Analytics–True Intelligence for Business Goals and Metrics – Measuring Customer Engagement on the Web     About the author: Julie Hunt is a software industry analyst, providing strategic market and competitive insights for vendors and buyers. Her 25+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Strategic Product & Market Intelligence Services

The incongruous worlds of “data integration” – Making data accessible for any-size companies

 

Companies of all sizes derive competitive advantage from technology systems that in turn support numerous business applications. Many variations on software solutions populate different kinds of enterprises. Customer relationship management (CRM), sales force automation (SFA), enterprise resource planning (ERP), supply chain management (SCM), and so forth, provide necessary tools for most businesses.  As implemented by enterprise-level, mid-market, and even SMB, companies, the complexity of applications continues to increase, as has the need for continuously integrating and transforming data that provide the life blood for these applications.

 

Enterprises probably are not spending enough time, money or resources on data integration (DI) software/services – part of the blame may fall to the broad range of options for DI solutions. The current data integration software/services market, while robust, is also highly fragmented and somewhat confusing.  From Enterprise Data Warehousing (EDW) and Master Data Management (MDM) costing millions -- to data mashups, and dashboards based on Excel -- data integration software solutions span a broad spectrum of cost and complexity for all sizes of companies.  DI offerings are proliferating to meet the demands of diverse data management and integration problems.

 

 

Companies are faced with making decisions about which DI solution, or solutions, to choose – while weighing current and future requirements, cost, usability, reusability, speed to solution.  Instead of purchasing software, many times companies will simply try to build a DI process or solution based on custom code. Custom code has been cited by the industry as the #1 competitor for DI software offerings of any sort, for several years now. But custom code is usually not the best avenue to travel for most DI projects.

 

 

As has been noted by Rick Sherman, mid-market / SMB companies may not be aware of many newer DI tools that are the right fit for their needs and budgets, and therefore have not seriously pursued the purchase of data integration software solutions, instead of hand coding.  The solutions that are most publicized are the high-end solutions from Oracle, IBM, Informatica, SAP-Business Objects, etc., so it may seem to mid-market and smaller companies that DI tools are expensive and difficult to use.  The high-end vendors are also the usual focus of industry analyst benchmark reports, such as the Forrester Wave and the Gartner Magic Quadrant. But there are many DI software alternatives that could be the right fit for many companies, that are cheaper and easier to use. 

 

 

(Originally a 2009 MQ was here - but Garnter demanded its removal. JH - 6/12/2014)

 

 

  

Larger Enterprise Data Integration Software Tools – Gartner definition for 2009 Magic Quadrant

Gartner has defined several classes of functional capabilities that vendors of data integration tools must possess to deliver optimal value to organizations in support of a full range of data integration scenarios:

·Connectivity/adapter capabilities (data source and target support)

·Data delivery capabilities

·Data transformation capabilities

·Metadata and data modeling capabilities

·Design and development environment capabilities

·Data governance capabilities (data quality, profiling and mining)

·Deployment options and runtime platform capabilities

·Operations and administration capabilities

·Architecture and integration

·Service-enablement capabilities

 

 

 

 

Does The Gartner MQ Help Most Companies Select Data Integration Tools?

With its focus on primarily high-end vendors and solutions, the Gartner MQ itself presents only a fraction of vendors that are operating in the overall competitive landscape.  While the Gartner document has provided an extensive list of 40+ vendors that are NOT included in the MQ, there is not enough information to assist with buying decisions. 

 

 

 

So the Gartner MQ for DI Tools focuses on solutions meant primarily for larger projects:  EDW, MDM, CDI, B2B data exchanges. It’s not clear if any of these vendor offerings would be cost-effective for less-complex needs. The 2009 MQ does include some DI alternative vendors such as Talend (OSS, cloud options) and Pervasive (on-premise and SaaS/cloud options) – but again, there are many more options available that have not been covered by the Gartner MQ and that companies should definitely explore.

 

 

 

Data Integration Variations – lots of options, which to choose?

 

 

 

 

Integrating data in a way that is meaningful to each enterprise

No matter which integration tools and methods are adopted, data is still the key – whether handled in concrete or abstract fashion. Understanding from a strategic point of view, how the company’s data matters not only to immediate projects/processes, but how it impacts the long-term plans and success of the enterprise, is essential to understanding which tools will be the best choice.  Until recently, most companies have used data integration solutions to solve tactical problems but it’s really time to focus on strategic planning and utilization. 

 

 

Many larger enterprises have traditionally thought it best to build a foundation of data warehousing and ETL processes, which have proven to be expensive, time-consuming, and frequently insufficient for overall data access needs in the enterprise. This sort of DW foundation can be difficult to implement, maintain, change, or even use. Other difficult – and costly – data initiatives have been introduced into the larger enterprise: master data management (MDM), customer data integration (CDI), complex data.  MDM in concept makes sense, but the pain of making it happen so far has kept many companies from implementations.

 

 

Companies of all sizes need alternative solutions and new methods that are faster to implement, easier to deploy and use, with more functionality based on OOTB and/or service-oriented architecture.  There has been an increase in pre-packaged, lower-priced offerings that solve specific DI business problems – hopefully this trend will continue.  Industry-specific integrations based on pre-fab templates and integration appliances for CRM, SFA, B2B data exchanges, are also increasing. All of these approaches usually provide tools/services for customizations.

 

 

Beyond the tools for effecting integration processes, easy-to-use tools are needed for simply understanding and planning strategic architecture for data access and applications, especially tools suitable for mid-market/SMB companies with fewer IT/developer resources. From the “big picture”, companies can design smaller, incremental projects and processes for essential data integration and management.

 

 

Important to the any-size company is the integrity of the data – “bad” data will render any process or application useless.  Less expensive tools for data profiling and data quality must become available.  Right now most data quality software solutions are expensive and complex. (And hopefully few enterprises are attempting to hand-code data quality processes, as most custom code approaches to DQ come up painfully short.)

 

 

 

Overall Data Integration Software Market Outlook Great

Data Integration initiatives, and the sale of DI tools, are proliferating in healthy numbers, especially compared to other software solution segments.  This reflects the value of data accessibility for many company needs, including improving competitiveness via analytics/BI for any-size companies.  Companies are setting priorities for obtaining better, cleaner, and more timely data out of their operational systems.  Data integration software solutions usually provide the best path to achieve such goals.

 

 

Gartner’s Mark McDonald recently pointed out increases in DI-related spending, commenting that customers will invest "in business intelligence applications and information consolidation in order to raise enterprise visibility and transparency, particularly around sales and operational performance."

 

 

Phillip Russom of TDWI: "One of the interesting wrinkles here is that these (BI / DI solutions) are now being considered an infrastructure, whose maintenance needs occasional augmentation. This new mindset -- by both technical and business people -- which considers DI/DQ/MDM to be infrastructure [i.e., akin to networks and broadband] is helping these technologies get -- or keep -- resources, despite a lock-down on general IT spending."

 

 

An IDC report estimates that the worldwide data integration and access software market will grow to US$3.8 billion in 2012, reflecting 8.7 percent CAGR from 2007 to 2012.

 

 

 

In the 2009 Data Integration Tools Magic Quadrant, “Gartner estimates the size of the market for data integration tools at approximately $1.34 billion as of the end of 2008, and forecasts growth at a five-year compound annual rate of approximately 9.4%. While the forecast growth has been substantially curtailed due to current economic conditions, this growth rate is very healthy compared with most other software segments.”  Gartner growth estimates calculated earlier in 2008 forecast the data integration tools market at nearly $2 billion by end of 2008, with projected 17% annual growth rate for software and supporting services revenue. (Market Trends: Data Integration Tools, Worldwide, 2007-2012)

 

 

 

About the author: Julie Hunt is an accomplished market intelligence analyst, providing strategic market and competitive insights for the software industry. Her 20+ years as a software professional range from the very technical side  to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt   For more information: Julie Hunt Consulting – Market & Competitive Intelligence Services