Agile ModelEvery technology organization today seems to practice the agile methodology for software development, or a version of it. Or at least they believe they do. Whether you are new to agile application development or you learned software development decades ago using the waterfall software development methodology, today your work is at least influenced by the agile methodology.

Agile was formally launched in 2001 when 17 technologists drafted the Agile Manifesto. They wrote four major principles for agile project management, to develop better software:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Agile methodologies have long been praised for their ability to increase team collaboration, break down silos, empower decision-making and project management, and more.

With roots in product development, the agile methodology was later adopted in the nineties by software engineers who were experimenting with different approaches to software development. Today, you can find agile ways of working in many industries outside of IT. From agile marketing to agile HR, it seems that the agile methodology has something to offer everyone.

Some argue that the agile methodology can accelerate AI projects through feedback loops that facilitate fast-paced problem-solving. Others maintain that existing agile methodologies encounter significant challenges when faced with the unique lifecycle requirements of AI projects.

Agility doesn't just increase speed, efficiency, and productivity. The real magic is its ability to reduce waste and help teams work more intelligently, focusing keenly on value delivery and business goals. Agility also helps deal with unpredictable problems and impediments by keeping everything visible, keeping everyone involved, and keeping the focus on continuous and incremental improvement.

The most successful applications of agile methodologies rely on small, focused, and self-sufficient teams periodically testing and either rejecting or accepting new systems and processes. Progress is made in short bursts, with an ongoing process of testing and troubleshooting ideas ultimately yielding more successful outcomes. Sometimes, however, larger organizations may struggle to reorganize their traditional structure in a way that minimizes dependencies and optimizes the flow of value of their agile teams. If that is the case, they can find support in commonly accepted good practices for agility at a scale that helps with aligning and synchronizing multiple teams toward the same business goals.

Artificial intelligence (AI) is revolutionizing everything from customer service in banking to data privacy compliance to elevator maintenance. That’s why businesses and public sector organizations around the world have AI programs on their IT agenda. Yet despite the broad interest, only 20 percent of companies have implemented any sort of functioning AI in their business.

Agile is a proven approach for delivering software solutions. With traditional agile methodologies, software teams work in time-boxed iterations to develop and deliver incremental portions of functionality in the form of vertical slices across the entire tech stack. This makes it easier to see progress and validate that the solution is developing as intended. Unfortunately, this approach is less effective for data-science solutions.

One reason traditional Agile methodologies fail in AI projects is that when it comes to advanced analytics products, the development stack often looks more like a pyramid with a broad base supporting fewer user-visible outputs.

Data science requires a detailed examination of available data, thorough analysis of solution alternatives, and repeated hypothesis testing to determine the best approach. As a result, data efforts tend to emphasize research and learning, work that does not naturally fit into a software development time-box.

Agile as traditionally applied to software also makes implicit assumptions about what is known and certain. Two specific ones can cause problems for AI efforts. First, software efforts tend to assume that the right problem to solve has been identified. Second, they also assume that the overall solution design is effective and will result in a cohesive, valuable solution. With AI, we need to validate both assumptions as we progress.