Podcast: The 'muddy waters' of implementing AI, and how manufacturers can avoid them
Key Highlights
- Start AI projects with a specific, measurable business problem rather than chasing the most exciting technology.
- AI solutions must handle messy real-world data while continuously improving data quality through governance and feedback.
- Include operators and industry practitioners in AI teams to identify edge cases, test solutions and build user trust.
- Successful AI adoption starts small, proves value quickly and repeats the process to build momentum and fund future projects.
In this episode of Great Question: A Manufacturing Podcast, Smart Industry's Sarah Mattalian talks with Mike Fedorov, COO and co-CEO of Applied AI Labs, who discuss why AI implementations often fail to deliver expected results. They explore how manufacturers can work with the muddy waters of imperfect data, establish effective governance, and design AI solutions around measurable business outcomes. The conversation concludes with a look at AI as a tool for amplifying human capabilities rather than replacing them.
Below is an excerpt from the podcast:
Sarah Mattalian: Hello everyone, and welcome to another episode of Great Question: A Manufacturing Podcast, this one by Smart Industry. I'm Sarah Mattalian, a staff writer at SI, and I'm joined for this episode by Mike Fedorov, the COO and co-CEO of Applied AI Labs, which helps companies add practical AI to their business processes for fast, tangible improvements.
As manufacturers across verticals adopt AI agents and company functions, many struggle with successfully implementing agents that are both cost effective and functional. Despite implementation challenges, manufacturers and other companies are still pouring record investments into new AI technology. But even with the investments in the technology, most AI remains in experimentation or piloting stages, and failure rates are high. 70% to 85% of gen AI deployments fail to deliver their intended ROI, which is double the already high failure rate of traditional IT projects. Manufacturing companies often struggle with agent implementation due to issues in data collection and governance, lack of connection between needs across company teams, and not properly considering outcomes before implementation.
Mike Fedorov of Applied AI Labs can discuss why companies face these issues and how they can mitigate the risks of AI adoption. First, a little about Mike. He has been working on these very operational issues, digital transformation, automation, and taking advantage of AI for the last 26 years. First, he managed operations for Mars, one of the top consumer goods companies. Then he led transformations for large complex organizations as a director at Accenture. Most recently, Mike has been leading a team that focuses on helping smaller businesses get earlier access to operational technology that historically has only been available for large corporations. I'll let Mike introduce himself.
Mike Federov: Thank you, Sarah. I am really excited to be here and help the audience sort through the noise and figure out how to make AI technology really work so without failing into one of those traps that cause that 80% rate of the failure.
SM: Thanks again for joining today. I wanted to start off by asking about the top reasons why AI pilots fail. Can you outline three main reasons why this happens?
MF: It's quite simple. First, I'd say this is because people are overly excited about the technology and look at that as a shiny thing to apply in the business rather than the other way around. something that the business needs and then you find the solution for. Second, it's about the learning curve for people both on the technology and the business. People often tend to start with the most exciting complex problems, and that's not the best way because complex problems are difficult to solve and oftentimes don't bring success. And then third and probably the most interesting piece is how data fits into that. The thing is, the data is the blood of the artificial intelligence and people make either of two mistakes typically, either assuming that the data is already in a great shape or refusing to move forward until it is in a great shape. So the third pitfall is what I think most of the issues are happening around.
SM: Can you expand on that third pitfall more and that data approach element, and why that can be kind of a creative problem for companies to solve?
MF: Let me explain. Think about which people are typically bringing AI to companies. They are normally very good, bright engineers who know exactly how the technology works. But they are much less aware of how the operations work because they help so many different operations. So they prepare a solution that works awesomely on simple sample data sets used in the tests. And then guess what? It fails. It just falls apart in the real life. So what happens is what works excellent in the test case meets edge cases, missing data, broken data, user mistakes, you name it.
So for example, think about a typical issue that is solved by AI. There is a production schedule to build around the historical times of the changeover between materials. So it's tempting to test this AI engine after training on a few sample setups of the real production data and conclude that it works beautifully. Now how about running with a new material without any history or a setup that has a quality finding last week, which dramatically increased the changeover cleanup times, setup, all that stuff. So this will not work very well. And what if an operator added a zero in the end of the typed in numbers? So without understanding all these cases of how the data goes wrong in the real life, the solution would not understand these cases. It will not work in all the situations. Now what will happen, it will work, still work well in most of the cases.
What do you think is going to be the trust level of the operators for the solution that works in most of the cases? I would argue that it would be exactly the same way as the pilots will trust the aircraft that lands safely most of the times. That is zero trust. So the real production data is messy. It's very far from the idealistic model. So when AI engineers come without the experience in the industry, it happens exactly that.
Think about the elite Olympic athlete goes into special forces and has a mission to swim over a muddy stretch of water. The athlete who is trained to swim really well, really fast in the Olympic pool with the clean water would now meet the mud, the occasional fish, maybe occasional logs. Do you think this mission is going to be a success? I don't think so. So now imagine the same athlete just refusing to swim over the stretch of water because the water doesn't meet Olympic pool standards. So those are two modes of operation. The first mode you dump an Olympic athlete into muddy water with the logs and you get a pure failure. And the second, you don't even start the project because you think that until all the data is cleaned up, you can swim. Neither works to help the mission. So those are two biggest ways how people make a mistake, either requesting that the solution just works, don't mind the data, just make it work. And the second one is how you don't do anything until all the data is cleaned.
SM: So to summarize, having messy data is akin to the logs and the algae and having muddy waters, and companies need an AI solution to be able to act like an athlete that is trained to swim through these muddy waters.
MF: That's exactly the point. And I think this is where you understand it much better than 80% of the companies judging by the results. The real solution is not to do either extreme, the real solution is twofold: first of all, make sure that your special ops person of the mission. knows how to swim in the real water. That is, an AI solution needs to understand and maybe not be as fast and as accurate as the ideal solution in ideal water, but needs to be processing the muddy water, needs to be processing the muddy data, needs to be able to give good enough results in the muddy real world situation. Now the second part of the solution is to set up the governance, the continuous improvement process and the technology to continuously improve the data quality because the second camp, the second school of the data quality first, actually right, the real good results cannot be achieved on the bad data. But that is not the extreme process of don't do anything until you get the water absolutely clean. This is the process where you need to do something, do something that works, and then use that something. to modify people to clean the data to get better results over time.
SM: So what I'm hearing is those “winning” companies have solutions and AI agents that can really, they're able to swim through these waters. And it sounds like they're also kind of designing these solutions with outcomes in mind. So that being said, can you expand more on what those key outcomes are that companies should be considering?
MF: The key outcomes on the data side are simple. That needs to be a solution that allows to work with the data that they have right now, understands the imperfection and can treat it with the right measures. And at the same time, the far further reaching outcome is to get a solution that helps improve the data. The guidelines, the tools, the governance, to get the data to the better shape.
SM: Can you expand a little bit more on the governance aspect and what that looks like?
MF: The biggest part of that, the data needs to be owned. The solution needs to be owned for that sake too. But data is a special part of that. Solution is more on the technology and product side oftentimes, and the data is sitting in the business. So the governance part of that, there needs to be a structure like a person who owns it. or who owns each part of that. That person needs to understand what good looks like and what is the controlled range, like what acceptable looks like, what good looks like, what ideal looks like, would need to have the tools to measure, would need to have the influence to get things better. That is in a nutshell what the governance looks like.
SM: That makes sense. Going back to designing solutions with outcomes in mind, do you have an example of a case study where a company was able to do this successfully?
MF: Sure. One of the keys that I like and keep coming back to is where we implement the demand planning solution for an auto part manufacturer. What we diagnosed as the key problem for them is that without the exact understanding of what would be the needs, the customer needs for each of their products, time phased, location phased, their business was losing money, leaving money on the table. So to get to that exact picture, the best forecast of the SKU by SKU, time bucket by time bucket, The key problem was the historical data cleansing. That is the history of data. It always has special cases. It has the promotions. It has the outages, the out of stocks. So once you get this done, this messy historical data cleansed, the forecasting is a relatively simple case.
But that is exactly where your data is muddy. You need to have a solution that automatically parses through that data. And that is where AI can be a great helper. So in our case, we didn't just ask the model to forecast based something. We explained to the model of how the data is contaminated with the incorrect data, how to understand the outliers, how clean them out, how to ask for additional inputs where the data is not clear to the model. So basically imagine that you've deployed a thousand very smart, talented analysts, diligent people who would be reviewing that data piece by piece. That's exactly how the AI would be able to do this. The outcome for us, for our client was that instead of spending months and months on figuring out the data and tuning the data to get right outcomes, We were able to ask right questions to people on the first day, first day and second day of what are the cases, what has happened in the past, how to train the AI engine. In the end of the first week, we would have the data preliminarily cleansed in fully automatic mode and the forecast will be completed.
So the manufacturing team would be looking at the forecast, the real forecast with their real data starting week two. But by the week four, that data was managed well. It was same data. It was not better data that they started with, but the forecast was looking right, was making sense to all people included. And starting week five, they started seeing the outcome. The needle has moved meaningfully. It was the data that they had using the ability to swim in the muddy water, if you will.
SM: They started, I guess, from the top down where first you considered the needs and did this data forecasting. And then going from there, you explained that to the model. So it was able to apply that information first to the company operations.
MF: That's exactly the point. You got it right. We started with the right initial understanding of what the problem was. We trained the artificial intelligence to solve the right problem. And that's how we got to the results pretty fast.
SM: And you said that happened within six weeks, right?
MF: Within four weeks.
SM: Wow, four weeks. So in a month you were able to mitigate that problem.
MF: Correct. And we actually were able to mitigate it faster. It's just we needed to prove that and the cycle in the demand forecasting is typically a month. So until you get into the second month, you can't really use the new data, the new models, the new data, the new outcomes.
SM: I also wanted to call back to the other two pitfalls other than data that you mentioned at the beginning of the episode. You mentioned that people get kind of overly excited about this new technology and start by addressing the most impressive things instead of the issues that actually need the most attention from the business. You also mentioned that companies often forget about the learning curve for both technology providers and operators and the people using this technology. I can see why these ways of thinking are tempting for companies when implementing these new exciting technologies. But how can they approach AI implementation differently so that there's still that eagerness but with sustainable results?
MF: That is where I would say the first two pitfalls apply, Sarah. First thing is, time and time again, we are seeing people are coming from the wrong perspective. That is, think how news are portraying the AI technologies, the magic of that. It can do everything. It can do your job. It can do my job. It can do everybody's job, president's job, whoever else. So people believe in this and people want to go big, like go big or go home, right? So instead of starting with the simple limited need that they can profitably solve in quick with quick turn around. They look at something really shiny. The moonshot going under the after the moonshots is problematic. It's expensive. It's a lot of it's a lot of risk in it. So and typically it fails to achieve results fast. Besides, yeah, it's not magic as we know. So people have very high expectations. People go big and people see that there is no outcome in the first few months. That's very, very, very disappointing.
So on the second part, if you go after something that you don't fully understand, you cannot quantify the results. You cannot tell whether the solution is good or not. That also leads you to a failure. What works in reality, what I have seen to be successful is that you need to do a few things right. First, the business needs to define a task, maybe a series of tasks that would move the needle quite a bit and that would be relatively simple to address. Imagine getting something successful fast creates the momentum, it builds the trust, and it oftentimes pays for itself before you can even account for it. And of course it pays for the next element of work. So that is a very important piece to start small, to start modest. The second piece is for the implementing teams to absolutely have people on those teams, not just the AI engineers, but also the practitioners. Ideally, not even consultants, ideally people from the industry who understand what can go wrong and what to do about it.
Finally, it's very important. especially when you work on the fast pace and you want to achieve results in weeks rather than months or years. You don't want to create solution then finding the testing that it actually doesn't do what you hope or hoped it was doing. To avoid this, you want to add your real production people, your real business people into the team so that they continuously check on the progress and give feedback. In my practice, what was most successful is when even before engineers deploy something to production, to testing, there is a pilot user who is curious, who is interested, who is excited, who tries that and who gives important feedback of what works, what feels good, what doesn't. Then engineers can actually build a much, much better solution faster. with smaller cycles that would solve that specific problem that we started with. So that works.
SM: Can I ask for specifically the auto parts case study that you kind of mentioned earlier? So you said that it was able to solve this issue within one month, but the AI agent that it used, it didn't just pop up out of nowhere. So how long did the preparation take to kind of get to that point where you could deploy it in one month and be able to solve this issue in one month? What did that timeline look like leading up to that?
MF: That's a great question. There are various cases for us. There are oftentimes cases where we prepare just a week in advance. We use the library of the solutions that we've already prepared However, in that situation for the data cleansing for the demand planning, it was actually about a year to fine tune that set of solutions. So in this case, we didn't use something that was like impromptu build. This is one of the non-industrial cases where you refine, you build, you tune, you apply it to the different industries for quite some time. So in that case, we were using an established solution. from one of our partners.
SM: And is this a model or a method that you think other companies can use and deploy on their own scale?
MF: Absolutely. That is where the providers are existent. My recommendation would be to be very thoughtful about what you can solve yourself and where it would be smarter to call to find the right partners. So think this way, simple cases where it is enough to employ ChatGPT or Gemini tools like that, Copilot, you don't need help for that probably. For the things that are industrially heavy, difficult, you want to have a partner, but you don't want to have a partner, any partner. You want to have somebody who has the right approach, the approach that starts with the real problem that you experience. who can understand you read into what drives your business result, understand in a narrow term of what help you need and provide a result, not bill you for the hours, but to explain you how you achieve what you need and charge for that. This way you would save money because otherwise it's going to take you much longer to get to the result even if eventually you achieve it. And it'll save time, which is also important.
SM: So that consideration allows you to save both time and money in the long run.
MF: Yes, for example, in that example was the automotive parts manufacturer. That would be exactly where a team of talented engineers spent about a year tuning that solution. So it would be good luck kind of exercise for the internal team to build anything close to that in quality. So that is where it was purely justifiable to save time and save money, use the economy of scale, basically get charged for a small fraction of the real cost of that solution and get it within weeks rather than spending another client of mine, a much bigger company, spent years, literally years trying to tune that kind of solution internally.
SM: Thanks for walking me through that. Is there any other advice that you'd like to give to companies when they're implementing AI solutions that we haven't touched on yet?
MF: To be very pragmatic. If one thing stays with our audience after our conversation, AI is a great tool. It's not a silver bullet. Need to be very pragmatic, need to find a small enough, tangible enough problem, solve it, rinse, repeat. That is the best winning approach And, of course, don't wait till the perfect data don't neglect the problem that you don't have perfect data that's a part of the solution that needs to be worked in those two things. is what I'd love people to retain out of our conversation.
SM: So we've talked a lot about the issues and implementation and how companies can avoid those. But is there anything that really excites you about this and the future of AI agents with these companies? What do you look forward to seeing in your work eventually?
MF: Absolutely. Sarah, what we are starting to see right now is that AI, everybody was trying to figure out what exactly it can do, what it cannot do. People were fearing for their jobs, people were fearing for the whole humanity. I think we are hopefully beyond that conversation right now. What I am seeing is that AI is a great complement to human being. That is what I am seeing right now and what I expect to see much more in the near future is that it's not that AI is competing with a person, it's a person with AI is beating handily a person without AI. So I'm looking forward to seeing a lot more collaboration where right AI is amplifying people's abilities.
SM: Well, thank you again for walking me through all of this and explaining how companies can really make their AI agents swim through these muddy waters.
MF: It's my pleasure. It's great to be here in this conversation and looking forward to hearing back of people's feedback.
About the Podcast
Great Question: A Manufacturing Podcast offers news and information for the people who make, store and move things and those who manage and maintain the facilities where that work gets done. Manufacturers from chemical producers to automakers to machine shops can listen for critical insights into the technologies, economic conditions and best practices that can influence how to best run facilities to reach operational excellence.
Listen to another episode and subscribe on your favorite podcast app
About the Author
Sarah Mattalian
Sarah Mattalian is a staff writer for EndeavorB2B's Manufacturing Group.