Author: Lynton Mack
The BEST Data Architecture in the world!
What makes the “best” architecture for data in this industrial age, the 4th Industrial Age? We are in the midst of the 3rd Industrial Revolution, and the way in which we work, play and relate to one another and the world is changing rapidly.
Back in the 3rd Industrial Age, at the turn of the 20th century, which is wrapping up right now, we saw the centralisation of workforces into cities as factories grew and the world as we knew it changed dramatically. At the time, cities were seen to be full of the worst conditions for humanity – pestilence, poverty, crime and disease, to name a few – all created by unplanned and rapid development.
The world’s leading architects of the time turned to designing utopian cities that dealt with the problems created by this unplanned growth, accumulation, and exploitation of people and other resources. The three most prominent of them had different approaches to delivering the perfect city but also shared a few similarities.
The visions and plans of these architects can be transferred to our data architectures to make them the best in the world, too – hear me out…
Radiant City
Le Corbusier, the father of “modern” architecture, had the idea of the “Radiant City”, which was all about centralised power and control.
It was a fairly socialist model consisting of tower blocks of minimal housing that met basic human needs, while the rest of the city was designed to provide everything else that a good comrade needed—gardens, open space, entertainment, transport, work, and more. It was designed to have functional components like a body that interrelated with one another.
Le Corbusier was pretty popular in communist countries, and his influence is seen in places like Sophia and Bulgaria, where simple housing is surrounded by parklands for communal use.
Broad Acre City
Frank Lloyd Wright was pretty much the polar opposite of Le Corbusier and had a vision of a libertarian community called the “Broad Acre City”.
It was, in fact, an anti-city where people were organised outside of dense urban environments with the aim of distributing resources fairly, reducing inequality and being kind to the earth. Every house would have a significant piece of land from which they could grow their own produce and have their own open space. The changes in methods for transit (the car) and communication (phone) would change the way we interacted with one another. Since everyone would have cars, broader blocks were OK because we didn’t need to walk everywhere. Or we could just call a neighbour instead of walking over. His vision was, effectively, suburbia on steroids and its focus was on independence.
Garden City 
Ebeneezer Howard’s approach to solving the same problem in a different way was the “Garden City”.
He thought that cities were too big to support their population, but towns were too small to provide the work and support that people required, like schools and hospitals. His idea was to have a central town surrounded by a ring of other towns, all connected to one another via canals and rail, but all independently managed but interdependent on one another.
Think of it as spokes on a wheel, with the end of each spoke and the hub being a town reliant on the towns around them.
So, comparing these three approaches to revolutionising urban design… which one is the best one?
Well, it depends. It depends on your resources, the size and shape of your nation, the problems you are trying to solve, and the desires of your people and their willingness to be independent, conforming, or collective participants.
Le Corbusier’s Radiant City was all about central and regimented control, Wright’s Broad Acre (anti)City was about distributed living and individualism, and Howard’s Garden City was about collectivist control with interdependence across independently managed small towns.
Despite their differences, these Architects share commonalities as well. They all had the optimism that a utopian vision could be created, that it could not be done without starting from scratch and that geometry and open space were important factors in the betterment of humanity. Planning and commitment to that plan were keys to success. Oh, and also, none of them were actually certified architects, just very smart people who tinkered in all manner of things.
So, moving forward to this Industrial Age and considering the “megatropolises” of data we have created, we can see some of the 2nd Industrial Revolution problems clearly running rampant in our data estates. We have created unplanned, sprawling, confusing, and stressful places to work. We can’t find things, and support tickets are never-ending as things break and new developments are constantly needed. There are plenty of people out in the world of data who are stressed, just keeping the lights of their servers on!
I worked in Ulaanbaatar, Mongolia, and the power station that used to be on the edge of the city is now pretty much smack bang in the middle as the city grew. While this creates a mess, it does allow for central heating, so it’s not all bad, apart from the plumes of radioactive coal-fired smoke. It’s the epitome of “organic growth,” which is a term we hear all too often in relation to data.
We can consider the same approaches as these architectural greats to solve the problems we have inherited and created over time in our organically grown data microcosms (honestly, created by people like me too!).
So, what’s the plan? What’s the best architecture for our utopian data estates?
Well, it’s going to depend on a number of things.
- The size and diversity of your organisation (country),
- How it is managed (levels and styles of governing),
- The desires and commitment of the people (citizens) to the plan and
- The products that the data is needed to produce (GDP).
These are going to be varied, and it’s not a simple “one size fits all”, or even “one size fits most”. When we are looking at a pattern for a data platform, we need to consider all of those facets (and more). Importantly, we are barely even considering the data at this point in time! We are trying to create a digital and data-centric world that people interact with that best supports their specific needs.
Those three greats created architectural patterns for cities that describe “centralised”, “distributed” (decentralised) and “federated” data architectures. It’s not just the data that is architected in this way but also the people, methods, locations of control, responsibility, and how communication and transport work.
Centralised Data Architecture
A centralised data architecture aligns with Le Corbusier’s “Radiant City,” where everything data-related is centrally controlled. The central controlling body has the right to dictate terms and responsibilities to deliver services or face a civil uprising.
It can work. In fact, there are plenty of examples where this works exceedingly well but it is dependent on the quality of the central body and the willingness of the data citizens to be governed this way. If the central body is not up to meeting the citizen’s demands, revolution ensues – let them eat cake.
This works where the citizens’ technical capabilities are not up to the task of managing and governing the data and its products.
Distributed Architecture
A distributed architecture aligns with Frank Lloyd Wright’s “Broadacre City,” where individuals manage their own holdings. This freedom and liberty to try new things in their own patch gives individuals the freedom to work independently and choose to barter and trade their services, knowledge, and resources with others.
Resources, such as infrastructure, training, and assistance, go to those who need them. In an organisation, it might not go down to the level of an individual but to a group capability or geographic region or line of business, where a small group has the independence to set its own rules, methods, language, values, and visions.
This level of liberty has risks that come with self-governance, where things may just fall apart. On the other hand, having many independent centres can lead to more rapid R&D due to the reduction in red tape and individuals’ commitments to their values, leading to nimbleness in growth and delivery.
Think that such a thing is ridiculous and cannot really exist and thrive as an organisation? Well, that’s a common model where the population are highly technical and data professionals – data analytics consultancies, for example. Even Google. Individuals are not simply left to their own devices; there is still a flat management structure floating around, but the freedoms that they have, create conditions for higher velocity of delivery, agility in methods and rapid prototyping and product development. The citizens know what they are doing, need little direct management, and are trusted to work towards common goals.
Federated Data Architecture
A federated data architecture aligns with Howard’s “Garden City”, where a city is broken into smaller towns that are interdependent on one another. None of them are larger or more potent than any of the others; they all have their own functions and act as a collective for the betterment of the whole.
The central hub provides a level of governance and control to the spokes to support and protect them as well as manage and distribute resources across each of the towns, but the towns are still responsible for what they produce, all with some independence.
Anyone who has looked at Data Mesh patterns would see the similarities in the design.
This sort of architecture works well in organisations that have multiple data professionals who apply SME knowledge within their data domain and share data products with other towns. For example, a Finance team with its own data and analytics team provides General Ledger data to an Assets team living in another town with its own data team. The central hub coordinates the sharing and ensures that the products adhere to standards and governance.
So, what does Industry 4.0 look like?
So, here we all are, mid-way through the 3rd Industrial Revolution, trying to work out what Industry 4.0 looks like for us and how we evolve to survive it! We are looking back at the vast unplanned and organically grown structures of the last age, which are filled with data monoliths, schemas that are unable to keep up and legacy technology which we need to decide on whether to support or burn down and just start all over again – “Tabula Rasa” (clean slate).
When we are determining the best architecture, we need to consider the people, the state of the nation, language, values, people, resources, vision, capabilities, mindsets, and much, much more. There is so much more to it than just selecting a vendor or platform, which are simply the building blocks of the city/state/nation/continent/world/universe you want to create.
Data is part of all of this; in fact, it is the key resource in this Age, but it is only one aspect of so many that we use to design a suitable architecture. Our ability to value-add to data in a controlled, repeatable, and meaningful way is essential for an organisation’s survival and relevance. Not to mention leveraging it to improve our organisation’s value and even how we interact with other organisations (“orga-nations”? Nah…).
So, the BEST data architecture is one that is planned—a plan that aligns with the people and the organisation’s vision and is committed to suiting this data-driven Industrial Age.
PS If you look closely at Garden City, you will see areas outside of the towns like “Home for Inebriates” and “Insane Asylum”. We call these people “Data Scientists” now 😉