Why humans aren’t optional in a data-driven world

Businesses are now placing more emphasis on building a data culture than they ever have in the past, with spending on big data and business analytics expected to reach $448 billion. by 2027. Beyond traditional analytics and business intelligence, organizations are also leveraging AI technology to gain new insights from their data.

Tools like TensorFlow and PyTorch have dramatically lowered the threshold for implementing machine learning (ML). This means that even a mid-sized organization or tech start-up can now turn to AI to improve customer experience or better understand consumer trends.

And the momentum behind AI is growing rapidly. Just last week, we reported that Meta was building the world’s fastest AI supercomputer with some 22,000 Nvidia Tensor Core GPUs for 20 times the performance of its current systems. In China, SenseTime recently launched its Artificial Intelligence Data Center (AIDC) to more than double its global computing capacity and mainstream AI.

How Machine Learning Works

Have you ever wondered how ML models work? While most of us will probably never fully understand the math and algorithms highlighted in AI research papers, the supervised ML commonly deployed in enterprises today is relatively straightforward.

Basically, it is purely mathematical and works by analyzing an increasing amount of data with an appropriate algorithm. The algorithm does not change throughout the training process, although the various internal weights and biases that influence its outputs do.

Once a model is trained, it can provide an expected response for any given input. An ML model trained using metrological data from the last 20 years could therefore offer a prediction on the chance of rain tomorrow given inputs such as temperature and other parameters today.

At the business level, companies have used ML to improve efficiency or business operations. Google, for example, has connected its data centers to ML models to manage the hundreds of systems, from air exchangers to chillers, based on a plethora of parameters that include external conditions such as weather.

Some of the recommendations even came across as counterintuitive to seasoned experts, the senior executive in charge of his data centers told me in an interview a few years ago. But the results speak for themselves, and Google has saved millions of dollars a year through improved energy efficiency.

The problem of missing data

But ML is not magic and missteps are possible. Take Zillow Offers, which for eight months bought homes in the United States at prices recommended by an AI engine. It didn’t work and Zillow suffered a $304 million inventory write-down with up to 2,000 jobs lost.

As I noted earlier in “At the Limits of the Data,” the failure stems from an inability to accurately predict future home prices for up to six months out, in a tumultuous and pandemic market unprecedented in the recent history. In short, crucial data was missing.

This is why data scientists spend most of their time on data munging, a process of transforming data from erroneous or unusable forms into useful forms. This may involve filling it with a median or average value or using more advanced algorithms to close the gap through predictions.

When it comes to records with missing data fields and outlier data points, sometimes the easiest thing to do is to get rid of them altogether, explained the professor of a Python data science course I took. attended National University of Singapore (NUS) last year.

But what if the key indicators are completely absent? For example, hidden issues such as structural flaws can dramatically skew a house’s prices. But an AI working only on the data given to it cannot “see” it. And that’s why we need humans to intervene.

Bring in the humans

An article on Fortune summarized the situation with the current state of AI. As Aleksandar Tomic, Associate Dean for Strategy, Innovation, and Technology at Boston College, has observed: “Data models, especially data science and training models, are not good for things that have not happened before. A computer will do anything you ask it to do, but [the outcome] depends on what you ask for.

In the same report, Oliver Yao, professor and associate dean of graduate programs at Lehigh University’s College of Business, warned that the technology has limits.

“I think people sometimes rely too much on big data, on technology. The time has come, and they don’t have to think, they don’t have to do much. They just have to trust what the data tells us to do,” he said. According to him, the data is “absolutely useful”, but one cannot “rely 100% on it”.

As business leaders increasingly turn to data for insights, they must not only focus on how to use AI and data analytics, but also on “why”. and the “when” to use them. And they should keep in mind that they will always need human employees to ask the right questions – and fill in the inevitable gaps.

Paul Mah is the editor of DSAITrends. A former system administrator, programmer and professor of computer science, he enjoys writing code and prose. You can reach him at [email protected].​

Photo credit: iStockphoto/monsitj

Comments are closed.