Dark Matters. How are you analysing the data you can’t see?

If you are looking for some mental stimulation, watch Jim Al-Khalili’s twin documentaries on Light and Dark. They are mind boggling. Did you know there are enough stars to fill the night sky with light? The only reason we see any dark is because most of them are so far away the light hasn’t reached us yet. And in the long-distant future, there will be no stars visible because dark matter and energy will have expanded the universe so much that the stars will be too far away from earth to see.  

Don’t ask me to explain what dark matter and energy are. I’m not a physicist and am still perfectly willing to believe the reason why 100,000 ton battleships don’t sink is because they are supported by magic.

All you need to know is that there is a lot of darkness. 27% of the universe is made up of dark matter and 68% is dark energy. Visible matter is only 5%, just over one fifth of dark matter. And the dark share will grow only bigger.

This is also true of data. Gartner defines dark data as ‘the information assets businesses collect, process, and store during regular business hours and generally fail to use for other purposes.’ According to IBM, 93% of all data will be dark by the end of 2020. This is also mind boggling.

Since business leaders spend a lot of time struggling to understand the data they can get, it is understandable there is not too much concern about the data they can’t. But there are risks to this. There is also another kind of data: business data that exists but you don’t have access to. And this can be the crucial data you need to make a decision.

Abraham Wald was a mathematician who was part of the Statistical Research Group at Columbia University during the Second World War. They applied statistical analysis to military problems, such as where to add armour to US bombers to reduce losses. The group analysed the damage on returning bombers and recommended that the wings be armoured, as they had sustained the most significant damage from flak.

Wald argued the opposite. The data actually proved that bombers could make it back with wings like Swiss cheese. They didn’t have the data that mattered: what had stopped the bombers that didn’t make it back from returning safely? His deduction was that their crews had been killed, and so the right place to protect was the underside of the fuselage. His recommendation was eventually accepted and bomber losses were reduced, saving thousands of lives.

Founders face many big questions: Why did existing customers leave? Why aren’t new prospects converting? What will it take to achieve product market fit? Which is the right PMF for their business?

There is probably dark data in your business that can help answer all these questions. There is definitely invisible data outside it.

What should you do about it? Firstly, you can map. Map every type of interaction that occurs between businesses and customers, between internal departments, between people and systems. This is your internal ecosystem. You can then label what data is being captured where. Then you can augment this with an external view by mapping how are your customers are interacting with other players and the invisible data that might exist somewhere.   

Secondly, you can illuminate. Fortunately there are also lots of tools available to help you do this: market research; spiders; social media monitoring; shopper ethnography; mystery shopping; online chat prompts; specialist software and machine learning.

Finally, the next time you face a major decision, you can ask yourself these questions: What data do I need to make this decision? What is the data I don’t have? What might it tell me?

None of this gathering and analysis will be as easy or clear cut as your existing management dashboard. Like Abraham Wald, you need to be prepared to make decisions on the analysis of data that is completely invisible to you. And the next time you look at the stars, savour them. Because they won’t be there for ever.

UP AND TO THE RIGHT.

References: https://science.nasa.gov/astrophysics/focus-areas/what-is-dark-energy