Machine Learning and an Enterprise-Worthy AI Platform

A few months ago, I wrote about the need to take a platform approach to succeeding with artificial intelligence (AI). Engaging with those who are keen to derive greater value from their enterprise data and drive a truly game-changing digital transformation program has strengthened my belief about the need for working with a comprehensive platform. It is also clear that any such enterprise-worthy platform would have to include sophisticated machine-learning (ML) capabilities. Further, for success in delivering value there needs to be a good degree of clarity about ML and enthusiasm to embrace it.

Let us start with the part played by data science because it is a foundational element in this discussion. Proliferation of data ­has caused organizations issues and concerns. On the other hand, it has also provided them with a treasure trove of possibilities if they know how to uncover them. Data science is ultimately about working with data ─ that is, understanding, defining, and refining algorithms and proposing the most optimal use of the data.

Data science in practice could be viewed as a synergistic combination of data integration, feature engineering, method selection, parameter selection, and eventually learning from outcomes. Each of these aspects stands to merit its own discussion. Briefly, data integration could be viewed as the step wherein you connect to raw data sources and then bring the necessary data together (e.g., in a data lake). Feature engineering has been described as the process of using domain knowledge of the data to create features that make ML algorithms work. It is essentially about creating, selecting, and cleaning features (individual measurable properties or characteristics of a phenomenon being observed). These features are often represented by columns of a giant data table, always with the intent to sharpen predictive accuracy; unrelenting attention to this aspect has an important bearing on outcomes. Method selection and parameter selection are about picking the best options; for example, iterating on different parameters for best results is an essential activity to arrive at the most optimal model. It is common to constantly train, tune, test, validate and rerun models to settle on one that is optimal.

What makes data science an effective weapon is when we can automate many of the functions, such as building and tuning models, especially by users who are not necessarily data science experts. Another example is when a system can automatically perform feature engineering on an input dataset. These abilities should be expected within an enterprise-grade ML engine. The ability to reap the benefits of data science with automation, along with scalability and ease of use, is a key benefit of ML.

So does that mean that ML is all science? Well, here’s where things get interesting. Much of the knowledge gained by the human element is frequently stored in known locations, but also obscure places, often scattered across the organization. Capturing this knowledge within systems, as well as using it as we go through the science of establishing actionable insight, does demand close attention to how well it is brought into the process. This knowledge often has a lot to do with building algorithms that are truly differentiated and geared to uniquely solving a given matter. Tribal knowledge[GB1]  frequently is critical in this regard. As these algorithms get nuanced a little at a time, we observe a phenomenon quite like the perfecting of a painting, one gentle, or firm, brush stroke at a time. It is clear that as much as we’d like to define it as pure science, ML is, in some measure, art as well.

Yes, a Platform Is Still Important

So what of all the ML tools that are threatening to overpower us? No doubt most of them have a defined value proposition. Where things get tricky is the way in which such a tool interacts within an enterprise environment. Where time and context are both of the essence, we cannot afford to lose either when going from one system to another. ML action needs to happen as part of a process, not as a black box sitting outside of it. The organizations that will succeed with ML over the long term are likely going to be the ones that have invested in the platform approach – where their ML capabilities reside within a platform that handles the entire life cycle of the data in question from ingestion to action taken as a result of insights gained. Within such a platform, there will be recognition of an end-to-end framework for data science and the automation of it. Such ML capability within a platform will also be characterized by speed and scale for predictive accuracy, accompanied by an ability to autogenerate documentation and serve up consolidated reports. When such ML capability is found within an AI platform, it provides a tremendous boost of confidence for an organization embarking on a digital transformation journey leveraging AI.

That’s where AI is headed. Are you on board?