Data Integration software solutions have been in the spotlight these past few years as the explosion of BI and data-driven businesses processes push through companies of all sorts. It has become clearer how essential trustworthy and timely data is to business, and how important it is for Business users as well as IT teams to participate in decisions regarding data-driven processes, data access, data integration, and overall data quality. Equally important is the collaboration between IT and Business teams as part of the process to select the right data integration software solutions. Business users of applications, and the underlying data, understand the business context and utilization for that data in different systems, and can help decide advantageous ways to manage access and integration strategies.
Until very recently, data integration activities belonged almost exclusively to the IT domain. And for some time, data integration has become much more than moving data between disparate systems. While there are many DI software offerings for many different kinds of projects, IT groups have frequently used custom code instead of purchasing DI tools.
When focusing on DI software offerings versus custom code, a deeper, important issue for many enterprises comes forward: the need to align IT and Business teams to work on data initiatives from strategic business perspectives, not just from technology infrastructure perspectives. Many line-of-business (LOB) managers are taking on more responsibility for IT projects, setting goals, outlining requirements, providing insight from the business POV, and providing cost justification for projects. These recent changes add momentum to using technology more strategically -- but must involve IT as the key partner in running business processes. Data integration projects can benefit greatly from this sort of dynamic shared ownership.
Strategic collaborations between Business and IT for DI initiatives tie directly to the 80/20 IT spend “paradigm”: companies continue to report a problematic metric where 80% of annual IT spend/resources are dedicated to the maintenance of the status quo, while only 20% has been ponied up to drive tech innovations that would likely benefit the competitiveness of the business. With the fast pace of business change and disruption, agility and efficiency in tech utilization are key to building long term success and sustainability.
Transitioning data integration projects from custom code to implementing more DI software tools that provide the majority of the solution - with much less coding – lends significant momentum to help swing IT resource allocation towards innovation rather than maintenance. The reusability aspects and other value points of DI tools add weight to this shift.
IT Culture of Custom Code for DI Projects – Tactical status quo?
In a 2008 interview with Philip Russom, TDWI, Lorraine Lawson for IT Business Edge, Russom indicated that at least 50% of DI projects are still hand coded, even though IT managers understand the clear, and frequently significant, savings that usually result from implementing DI software solutions:
“Whenever he explains his case, people tell him: Wow, I get the economics. Wow, thank you for enlightening me. BUT we still have this culture of hand-coding.”
Lorraine Lawson followed with additional posts in 2009 – here summarizing the comments generated by the discussion of custom code for data integration projects:
“Last week, I wrote a post on Why Companies Still Hand Code, But Don't Have To, despite the numerous data-integration tools of all price ranges available on the market. In the piece, I summed up the reasons cited by Rick Sherman, founder of Athena IT Solutions, which included ignorance of market costs, a lack of resources and a misplaced adherence to corporate standards.
“Several readers commented on the post, adding that Sherman overlooked one critical reason why so many data-integration projects are hand coded: The human factor.
“When it comes to data integration, the missing human factor is simply that developers like to code. They're comfortable with the tools and, in that moment, it's easier to hand code than evaluate, choose, buy, install, learn and use a new tool – even if that tool might shave months off projects on down the line.”
The seeming preference for custom code does point to a weakness in the differentiation of certain DI software offerings: these tools are still designed to be used as “developer tools”, and indeed involve an amount (sometimes a lot) of coding themselves. Why would developers adopt a DI solution platform that on the surface looks quite a bit like something else that they are already using? Obviously most DI software tools offer a lot more than a “code platform” -- but to have clear differentiation, DI software vendors need to offer more focused solutions that solve specific business problems for specific customer scenarios. DI tools need to become faster to learn, easier to use, with ready-to-go components, instead of providing general platforms. For better adoption, DI software offerings should be tools that any team will want to use over and over.
The very important functions of data profiling and data quality are significant components of data integration software solutions that cannot be emulated very well by custom code. DP and DQ processes help derive more value from data and to help render data more trustworthy and usable. Data profiling tools in particular provide a platform for both Business and IT usage and collaboration. DP and DQ capabilities have become important differentiators for DI software offerings, provided they are cost-effective and lean towards ease-of-use.
Vendors: Market DI software offerings to be “more findable”
There are DI software tools aplenty for any sort of company initiative – and any sort of company. They can be on-premise licensed, SaaS subscription, cloud-based metering, open source. They may manifest as platforms, appliances, mashups, dashboards, APIs, templates, pre-packaged industry-specific solutions, and so on.
Since major analyst publications primarily cover only a select group of high-end DI / ETL tools, the notion that companies and their IT and Business teams are not aware of the many options available has validity. However this is really an opportunity for DI software vendors to do more to make their solutions much more “findable” for their target markets and customers. With the upsurge of interactive marketing and social media for B2B selling, “findability” provides software vendors methods to improve the ability of the right customers to connect with the vendor.
Competitive advantage can be gained for vendors who do a better job of marketing and connecting what the tools offer to customer business problems. So it’s not just a matter of being findable as a “brand” or category of DI software, but findable as the right fit for a specific DI / business problem, and for a specific customer segment (a segment that could be Business and/or IT). DI software vendors also own the ability to do a better job of providing good solutions that are ready-to-go, to handle specific industry scenarios. No more one-size fits all type of DI platforms – instead, differentiate from developer code platforms by offering focused tools that solve pre-defined problems, solutions that target specific markets/industries/scenarios.
When DI software vendors have done a good job of revealing solutions that matter to their markets while showing that these solutions truly solve important problems, the intrinsic value of those DI software offerings should be a “slap in the face” for targeted Business and IT users. The value should be so apparent that IT will rarely want to hand code DI projects again – at least not the important ones.
Clear strategic purpose, distinct solution context, and a customer-centric orientation can take DI software further into helping IT change the usual 80% allocation for the tactical-for-status-quo – on to achieving the goal of a larger chunk of resources dedicated to competitive innovation. For DI projects, connecting IT to Business users of company data empowers everyone to work smarter. Savvy companies want to get more from their IT investments and want to rely on their data assets for competitive advantage. Through collaboration the company benefits: Business users know what they need from data and understand the context for data; IT provides the know-how to implement the DI processes to make it all happen.
Flipping the 80/20 IT spend paradigm – understanding the TCO of custom code for DI
Talend Open Source DI Looks at ROI of Custom Code
Whether you agree with all of the Talend conclusions, this study introduces concrete parameters and data about the real cost of custom-coded DI projects. One valuable takeaway is that this study shows the need to evaluate custom code on the same basis as any software offerings – and why. I find the Talend data believable, since most IT groups overlook lots of hidden costs related to custom coded projects when conducting comparisons to acquiring DI software tools. Talend is at least opening the door on points that should be considered when selecting the right DI solution.
Further details can be found in the complete report, The Return on Investment of Open Source Data Integration, 2008.
Talend ROI Project Category: Large project: real-time enterprise data integration
· The "Large" project is a complete (large) enterprise data integration project.
· Twelve different databases, two packaged applications (one ERP and one SCM), an application from a partner (accessed through Web Services), a LDAP directory, and a Software-as-a-service CRM all need to be kept synchronized in real-time.
· This project involves 200 complex data flows.
Cost to complete this project using custom code:
Summary of 3 approaches to completing the “large DI project”
And finally:
About the author: Julie Hunt is an accomplished market intelligence analyst, providing strategic market and competitive insights for the software industry. Her 20+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing. Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter: @juliebhunt For more information: Julie Hunt Consulting – Market & Competitive Intelligence Services
Comments