The global economy is shifting
For a long time, corporations and businesses are developing different analytical systems and collecting data from different operational systems. Data in operational systems were actually describing steps and results of different internal business processes – materials, orders and deliveries from suppliers, manufacturing, customer orders, invoices, payments, and financials. Data was loaded and consolidated in a data warehouse and analytical systems, usually on a daily basis, to be prepared for management reporting and analytics.
The first big shift in analytical systems was triggered by the change from product-focused to customer-focused organizations. Customer relationship management, data about customer behavior and preferences, customer segmentation, and information about contacts with a customer through various sales and service channels over time become much more important sources of analytical information than operational data. This shift was important, but it was just a herald of change that will come later.
The next big shift is happening now, in the last few years. Everything around us is becoming more and more driven by events than by defined business processes, especially in B2C relationships. In every aspect of your life as a consumer, you can experience that corporates and different service providers are reacting to your recent actions with customized offerings. For example, if you browse some booking site looking for accommodation in New York and leave without booking, you will receive an e-mail with selected hotels in New York within an hour. You have playlists customized for you at your favorite music and video streaming services. You have countless examples of such assertive events from providers triggered by your actions.
Rise of event-driven analytics
For such events and actions, external data becomes more important than internal data. Location of the customer, time of day, weather, a device he is using, and many more that describe the context of the event and all that huge volume of varied structured, semi-structured and unstructured data is collected, streamed, analyzed and stored in real-time by analytical systems.
And now we have four main reasons that revolutionized the analytical landscape – event-driven economy, external data, complex and unstructured data, real-time processing – and completely changed analytical systems architecture and functionality. Everything became more complex and much larger, emphasizing critical architectural problems that also grew larger – data architecture and modelling, data governance, data privacy and security, metadata, master and reference data… A lot of problems, but how to solve them?
Change of the way of thinking in the event-driven world
Well, the usual answer that you can hear is to use Data Lake or Data Mesh to collect all the data you may need, to enable smart Data Scientists to use the schema-on-read approach, but my five cents go to the change of the mindset. Instead of a legacy monolithic mindset defined by business processes, we need to apply methods of creative, critical, and design thinking in new analytical systems architecture.
Creative thinking will ensure that we will always ask why something happened in our business and create a large number of different ideas and scenarios.
Critical thinking will always debate ideas using objectivity, effective communication and problem-solving abilities of the team, until ideas are validated or discarded.
Design thinking will enable you to prototype and develop your ideas for analytical data products in a fast, efficient and risk-free way.
All those three approaches and thinking methodologies are necessary for modern data engineering and data science teams.
When you have the strong team with essential thinking, design and development skills, on top of that you can build flexible and bulletproof analytical systems. The foundation of such a system shall be an industry-standard, comprehensive data model that will cover all standard out-of-the-box business requirements and can be easily extended with a new domain or functionality that will seamlessly integrate with existing entities and domains. On top of that, you need to define a Data Governance framework and set of tools that will take care of processes, responsibilities, data assets, definitions and business glossaries and assure data quality in your analytical systems and make sure that it will not turn in Data Swamp.
Think first, design and develop later
The world is changed and now most of the business is event-based. Analytical systems are also changing and you cannot solve new problems just by adding new technologies and systems. The first step you need to do is to step back and ensure that you have the right mindset and understanding that will make sure that you will design and implement analytical systems that will build on top of what you already have and will be future proof for many years to come.
How did the shift from a process-driven economy to an event-driven economy affect your business? Do you want to know more about this topic or have a question?
