Is My AI Accurate? Co-founder and CEO of FoxyAI, Vin Vomero, Provides Insight on How to Tell If Your AI is Accurate EnoughAug 05, 2022
By Vin Vomero, Co-founder and CEO of FoxyAI
Artificial intelligence (AI) has become ubiquitous. From Netflix recommendations to smart speakers, it seems like every day brings new innovations from the field of artificial intelligence. However, today, no AI is 100% accurate. In fact, AI may never be perfect. Understanding this is key to successfully deploying any AI in your business. When considering applying AI to your business, how should you think about accuracy?
The first thing to consider is what level of accuracy is “good enough” for your business and use case. If you’re building self-driving cars or AI for medical purposes, lives are at stake, and very high levels of accuracy are necessary. If the business requires a high degree of accuracy, but your models don’t meet that goal, the model may not be useful for the business.
Once you have established the accuracy acceptable to the business, you must look at the current state-of-the-art research. If your use case requires 90% accuracy, but current methods perform at 75%, you may not be able to achieve the required level of performance without significant investments into research and development. If current methods are at or near your required level of accuracy, you may be able to get up and running in a much shorter amount of time.
Many datasets are used as benchmarks to measure the performance of various tasks within the AI community. For example, the current state of the art for the detection of objects in an image, benchmarked on the popular MS-COCO dataset, currently stands at ~90% accuracy. Similarly, the current state of the art for image classification, which is benchmarked on the ImageNet dataset, performs presently at 91% accuracy.
These metrics are helpful in quickly assessing whether or not the current technology performs well enough for your needs.
When possible, you should also benchmark against human performance to help compare error rates for the same task within your organization. Of course, humans make mistakes. If people perform with 90% accuracy and your model performs at 85%, you’re not too far off human-level performance – you should be good to go.
Lastly, you must consider the cost of incremental improvements. If your model currently performs at 85% accuracy, is it worth potentially months of research and data collection to get to 90%? As your model improves, you will see diminishing returns on the time and money invested in increasing accuracy further. This is a question that must be addressed internally based on the business unit’s needs, and time and cost constraints.
Tesla, for example, has been pursuing autonomous driving for years, collecting and annotating mind-boggling amounts of data in the process, and has yet to achieve its goal of fully autonomous driving.
That being said, while accuracy is an important metric to consider, even an “inaccurate” model can transform your business. Consider taking a quick look at AI Accuracy Is Overrated: How Even A “Wrong” Model Can Transform Your Business from Forbes and ask yourself what level of accuracy fits your needs adequately.
Even incremental improvements, especially compounded over time, can have a dramatic impact on your bottom line.