Are Data Warehouses Still Relevant?
by Brittian Warner, on May 1, 2020
The future of data warehousing is changing, fast. Technology is giving us the ability to move quicker and accomplish things we could never imagine before. But is our modern data warehouse strategy evolving with new technology? Is the purpose of a data warehouse the same as before? I have been thinking about these questions quite a bit lately.
More specifically, I’ve been thinking about where marketing data fits in with traditional data warehousing. Can you blame me? Here at DemandJump, data-driven marketing is important to us. We make all of our decisions based on data. And as the Data Architect, I need to make sure our data warehouse can fulfill not only our needs, but our customer’s needs as well. So in this blog, let’s talk about data warehousing!
What is data warehousing with example?
Before we get too far, let’s define what a data warehouse is exactly. Thanks, Google; a large store of data accumulated from a wide range of sources within a company which is used to guide management decisions. I like this definition because it highlights that a data warehouse is not just about accumulating data from various sources but it is also there to help management make decisions.
There is a ton of data and it is never in the place you need it. This is especially true if you are a large enterprise where one report could be made from 10+ data sources requiring a cross-channel analytics solution. Bringing data from disparate sources together is no easy task, and marketing data is no exception to this. I would even argue that marketing is full of some of the most disparate data sources out there.
Marketing departments are unique because they need data from not only internal sources but also external sources as well, and most of the sources that I have encountered with marketing require hitting some sort of API in order to automate ingestion. So what happens when your team does not know how to go about bringing marketing data together?
Do you go find a consultant or hire a new internal resource? What data warehouse architecture do you use? These are important questions that need to be answered if you are considering a data warehouse in your organization.
Are data warehouses obsolete?
I have been consulting with data in some form or fashion my entire career. From spreadsheets, flat files, databases, and API’s, I have experienced pulling data together many different ways. Almost every experience includes some form of a data warehouse. The most common form of data warehouse I have experienced is a spreadsheet. Yes, I called a spreadsheet a data warehouse and almost every organization has done it.
I have had that pleasure of working with many talented people that can do wonderful things with Excel, but there are many limitations with spreadsheets; one of those being its shortcomings as a single source of truth. This is arguably the most common reason for a data warehouse. How do you know your data is correct? So you have an analyst that built an amazing dataset and visuals for a presentation. That’s great! What happens when you have another trusted employee that built his or her own dataset but got different results. Which employee is correct? Well, that’s not always an easy question depending on the dataset.
In order to answer this question, you have to dive into the data and figure out what each employee did to generate their results. This could take weeks to figure out and in turn lots of money spent. How do you avoid this? The simple answer is a data warehouse. This will provide a single source of truth for all of your employees to use for their datasets.
I feel this is especially prevalent in marketing. There are so many different ways to slice and manipulate marketing data, so how do you know it’s correct? For example, how do you currently attribute conversions? Do you use a 30, 14, or 7 day lookback window? Sometimes marketing data is not as clear as something like order data from a CRM application - where you have order #123 from a specific customer on a specific date. You could even argue that marketing data is abstract to a certain degree depending on your organization’s KPI’s and rules around viewing data. This level of abstraction is even more reason to have a defined process around what happens to your data from extraction to presentation.
Anyone that analyzes the data must understand what they are looking at. We at DemandJump leverage an enterprise data warehouse for all of our data requirements. We have a single source of truth that has been validated and tested for all of our employees needs. Doing so allows our employees to move fast and gain insights without all the overhead of worrying about gathering data and hoping the result is correct.
Do we still need a data warehouse?
So far, we have discussed data warehousing and some of the reasons why this is still an important topic. Since joining DemandJump, some of my views have changed on this subject but the core has still stayed the same. In my previous experience, I was a consultant where I would work with customers and build enterprise data warehouses for them to take over and maintain themselves. This is how many organizations are functioning today where they either hire a consultant or hire an internal resource to build a data warehouse (or multiple) and have a business intelligence team that maintains/expands it.
Not all data warehouses are created equal. Many organizations across the globe are spending millions of dollars to get their data under control - only to find out that their data warehouse needs to be updated a few years later and sometimes even rebuilt!
Some companies get it right and are set for the next 3-6 years, whereas other companies may never even get a working product! Are you surprised? With technology rapidly advancing and the amount of data growth we have seen in recent years, it is not unusual for leadership to want to change directions. Change is inevitable, so how do you take all of this into consideration when deciding which solution to choose?
A data warehouse was always the solution when I was a consultant. Most organizations today absolutely need some sort of data warehouse but what happens when your organization cannot afford a consultant or internal team to build and maintain a data warehouse?
Or maybe there is a better solution out there? What if there was a single source of truth that did not require any development and your analysts could start leveraging data instantly? This sounds like a dream but this is exactly what we are doing with marketing data at DemandJump. A single source of truth that has been validated, tested and requires no development time. Not only do you get all of your marketing data in one place, you also get access to one of the top visualization tools (Looker) and the most talented marketing team in the country that supports you through every step. This is the future of data warehousing!
So do you still need a data warehouse? Yes, you need a data warehouse, but the exact solution depends on many different factors including but not limited to technology stack, internal resources, internal knowledge, data governance and budget. I am so excited about the future of data warehousing and where things are headed. The things we are doing at DemandJump are just the beginning and we have more exciting features coming in the near future.
If your organization is thinking about how to handle marketing data or data warehousing in general, just remember that you are not alone! There are a ton of resources out there to help! I hope you enjoyed this article and if you have any questions or want to talk more with the DemandJump team you can do so here.