newsroom

Is the fail fast methodology relevant in the context of AI application development? Absolutely. Given that enterprise AI computing is still in its infancy, failing fast with quick turnaround times is critical for the development of AI applications. This entails adopting a flexible end-to-end machine learning pipeline for training workloads at the beginning of the application development. Let’s review how this makes sense by looking at four different dimensions of AI development and showing why IBM Watson products and solutions are best suited to this work.

1. AI is an experimental science.

Data scientists experiment with various algorithms and learning paradigms (such as transfer learning) in the process of developing AI models. The algorithms may need to be trained with additional data to address overfitting. Features are dynamically added or removed to improve machine learning accuracies. Hyper parameters are (manually and also automatically) tuned to choose the optimal combination. Hence, the hardware and software setup for your AI applications should facilitate easy experimentation. Watson Machine Learning Accelerator provides a comprehensive set of tools for experimenting with traditional machine learning at scale as well as modern deep learning algorithms.

2. The software ecosystem around AI is rapidly evolving.

While Spark and Hadoop are continually maturing in line with the explosive growth of data, Python bindings are becoming a de facto standard for machine learning libraries and frameworks (PyTorch, Tensorflow, Scikit-Learn and xgboost, to name a few). Choosing software tools that provide the right levels of programming abstractions for common machine leaning operations and distribution of computation can make a huge difference in application development time. Your AI compute setup should allow for hustling with different frameworks and software abstractions. IBM Watson products and solutions provide varied software abstraction levels (Watson APIs, pre-built Watson applications, open source frameworks with add-on development tools and so on) for AI application development.

3. Software engineering practices are not yet established.

Conventional wisdom on development of traditional software applications has been well documented in the form of development lifecycles, design patterns, testing methodologies, software metrics and so forth. In contrast, such mature processes and best practices are yet to be fully developed for production-grade AI applications. Embracing agile development is a key to success in developing enterprise AI applications. IBM Watson products and solutions provide tools for various stages of AI lifecycle development such as data preparation, model development and deployment (Watson Studio, Watson Machine Learning and Watson OpenScale) all under one roof thereby allowing customizable AI workflows.

4. Distributed computing is a way of development.

In machine learning, distributing the computation among hardware resources is more than just a technique that improves the execution time of an algorithm. It can be employed at various stages of application development including hyperparameter tuning, ensembling, data parallelism and model parallelism. In this sense, it’s a way of development that promotes agility in model building from the very beginning. The software and hardware infrastructure should facilitate easy scale up and scale down of resources for parallel execution of an AI workload. Watson Machine Learning Accelerator infuses elasticity and transparent parallel execution into its components, thus shielding the data scientists from implementation complexities through simpler programming constructs.

For all of the aforementioned reasons, applying the “fail fast” methodology makes good sense in AI development. It’s imperative that teams set up an end-to-end machine learning pipeline in the early stages of your AI application development that allows for iterative development and quick changes of course. Click here to read about one such pipeline designed by IBM Systems Lab Services consultants by integrating open source technologies with Watson Machine Learning Accelerator.

The post Fail fast (and better) in AI development with IBM Watson appeared first on IBM IT Infrastructure Blog.