What's the point of business intelligence in 2023 and beyond
Pushing the boundaries of a solved problem
Maybe you haven’t heard yet, but Microsoft and Power BI have officially closed the book on business intelligence, so we can all move on to more important things. The combination of an easy to use excel-ish interface, top notch enterprise integrations and (most importantly) a seemingly free price point make it practically impossible to choose another tool. And why would you want to - nothing is as simple, as widely adopted and as powerful at creating dashboards and simple data apps as Power BI.
Power BI: the cause of and solution to all of BI’s problems
So job done? In a way, yes. If you define ‘business intelligence’ as the art of quickly creating dashboards on questionable data infrastructure to be consumed like candy which gives a temporary high, a brutal crash and a ton of empty calories then Microsoft has nailed it. This is the 2010s way of doing BI and its apex achievement is Power BI and the data analyst - driven workflow that it enables. Give a flexible tool to someone close the business and let it rip, problem solved.
I may sound dismissive of this way of working. It’s because I am. But not because it’s ineffective - in fact it solved a very, very real problem that plagued business intelligence in the Cognos and Business Objects era - the fact that accomplishing anything required an interminable wait for IT to do even simple modeling or reporting tasks. We can’t return to that world. And the people close to the business SHOULD be empowered to solve data problems.
The BI pipeline is broken
But neither can we continue to live in a world of one-off Snowflake views feeding one-off data models feeding one-off dashboards, where every metric is ‘bespoke’ in the worst sense of the word, where most visualizations are useless and most dashboards collecting cobwebs. The need to deliver something - anything - as quickly as possible so that we can say we incorporated data into our decision has resulted in a lot of rushed, repetitive and unscalable work.
It’s not fun. It’s not fun to build, it’s impossible to maintain, and for business people it’s not fun to use. We have a conundrum - any wait is too long and yet moving fast delivers the ‘break things’ part, but very little of lasting impact gets made.
Self-service can’t solve it
I talk to a lot of data leaders, and they feel increasingly burned by this. You might think the obvious answer is self-service and yet they question whether the investment in ‘self-service’ has really resulted in anything. Becoming ‘data driven’ has driven them mad, and the infinite dashboard sprawl of BI work has become an end unto itself.
The reality is that the idea of self-service - that we could just push the creation of business metrics and the dashboards that depend on them directly to business people - has created a noble yet ungovernable mess. Our current crop of tools - lead by Power BI - have become a brutal combination of too complex for business people to use to create anything, and yet too simple for data people to build the kind of robust metrics delivery systems of the past which can guarantee timely, accurate data in a repeatable, auditable, scalable way.
This may be a distressing message. Many data teams are struggling mightily to deliver self-service, and yet here I am arguing that it won’t solve their problems. Well - it won’t. So what will? I’m the first to admit, I don’t know for sure. But here’s where I see things going.
Step 1: Remember the point of BI
In the pre-Tableau era, BI was not defined by visual fidelity. Humans are, of course, visual creatures and many a BI contract was inked based entirely on an awesome demo of 3d column charts. But the purpose of the BI system was widely acknowledged to be the delivery of timely and accurate metrics. This was accomplished by modeling business logic directly withing the BI tool and exposing that logic to EVERY report or dashboard in the system. There was no tight coupling of data model to output - instead a single model would feed sometimes thousands of downstream objects.
Of course you could build report-level metrics, but it was often frowned upon, and great care was given to backing those metrics out of the report and into the BI model as soon as they were broadly useful. This took time, it took expertise and (from the business’ perspective) it took freaking forever, but it did result in 10k reports that used the exact same calculation for ‘revenue’ guaranteed, no matter what.
Tableau came around, and the focus shifted to communicating with data - telling stories. This was a critical and badly needed advancement in BI. However, the bedrock purpose of BI was neglected and now we have sprawling BI deployments with absolutely zero high quality data models. Recognizing this need is step 1.
Step 2: Rediscover the BI practice
The only way to build the kind of high quality metrics distribution system I describe is develop a broadly accepted, repeatable set of steps that you can hone over time - aka, a practice. I encourage you think explicitly in these terms.
A practice is something you do again and again, keeping what works and discarding what does not. But the key is, you do not start at square one every time. You build upon what came before. This means, practically, modeling data for reuse. Maintaining accurate metrics over time. Always using the metric from the model - not a metric scoped to a single report - whenever possible. It means following standards in how you visualize data, how you design dashboards. It means providing these high-quality, reusable models as the basis for self-service.
The key is to figure out how your team can do these things fast, and then doing it over and over until you’re really, really good at it. It means learning as you go.
Step 3: Treat BI (sort of) like software
Business intelligence is just woefully behind its cousins data science and data engineering in embracing code-first approaches to modeling and visualizing data, to our extreme detriment. And I’m not entirely sure why - maybe it’s that BI attracts less technically inclined people, maybe it’s a result of the over-focus on self-service, or maybe it’s an historical accident. But in practice, BI people are often stringing a chain of UI-bound tools together via vendor-provided integration points that are constantly breaking and have almost zero auditability and which require immense effort to build, deploy and maintain.
This has to stop. These problems have been solved using software engineering techniques. Modern BI tools should offer declarative APIs, SDKs and integrations with CI/CD. And modern BI teams should embrace this way of working, because while it seems more complex at first, it’s actually far simpler to copy, diff and edit code and config files than it is to build a whole new BI object in a GUI.
Step 4: Embrace the data model and metrics layer
The only way BI can fulfill it’s purpose of distributing high quality metrics to aid in decision making is to focus, first and foremost, on the metrics. Not the visuals. And that means building a BI model via a semantic or metrics layer.
There are lots of them to choose from. Some are tied to BI tools (GoodData has such a layer) and some are standalone software. The key is to select one that is open, meaning that many applications can access it via SQL and API/SDK endpoints, rather than one that is proprietary, meaning it is locked to a single BI front end. Like most of them.
The reason for this is obvious - the promise of BI is only fulfilled if you can maintain metrics quality across applications. In order to do so, you need a single place to define the metric, where you can update the metric and see that update automatically flow through to all downstream uses - AI, BI, data science, custom applications - anywhere.
You may be tempted to say, ‘my data warehouse is my metrics layer!’ Well… maybe. The issue is, the data warehouse itself must make concessions to accommodate the needs of ETL/ELT and often contains structures that exist explicitly to assist in data processing and storage. An independent metrics layer abstracts away this complexity and gives you a single place that contains metrics from many systems. The underlying physical reality of your data is undoubtedly very complex, but a metrics layer can make it appear simple to external users and applications
The pendulum swings back
Ultimately, what am I advocating here? In many ways this is a return to the wisdom of a previous age, but with modern tools and techniques. The big reason we left all this ‘metrics layer, BI practice’ stuff behind is because it was too slow and complex. However while the BI world was focused on building ever-prettier charts, the engineering world developed techniques and software to handle this complexity.
What this means is that it’s realistic once again to embrace modeling, embrace standards, embrace accuracy without sacrificing so much speed as to make your business unhappy. This is how you move forward in 2023 and beyond to build big, bold and audacious BI systems that leave desktop BI tools like Power BI in the dust. Or, at least, supply them with high quality metrics so that they aren’t churning out so much beautiful garbage.