Data science and artificial intelligence are hot media topics. An expert talking about the capabilities of predictive analytics for business on a morning TV show is far from unusual. Articles covering AI or data science in Facebook and LinkedIn appear regularly, if not daily.
Due to a surfeit of information about AI and big data on the Internet, companies can assume that data analysis is the solution for most of their data-related issues.
For instance, we had such a case in our work. An OTA that uses flight information from the Innovata dataset contacted our data science team with a request to analyze it to extract data. Innovata, which was incorporated by the FlightGlobal news and information site, is a leading provider of historical, current, and future schedule data for more than 900 airlines worldwide. Our clients considered working with large datasets a big data problem. While the dataset is indeed large, the problem that the OTA was solving didn’t require data analysis, only data retrieval. Our data science team delegated the project to a software architect instead of a data scientist.
So, before planning to use data science or AI for your business, find out whether it’s the technology you need. You can answer this initial question by isolating and specifying your problem.
In this article, we’ll discuss what factors to consider and steps to take to team up with a competent data science or AI vendor.
1. Match your problem with possible solutions
In general, data science and AI solutions entail gaining valuable insights using available data. Some companies approach data science vendors to build products whose key functionality is centered around machine learning. For instance, this may be an application that transforms speech into text. Other organizations may want to develop a custom analytical and visualization platform to be in control of their operations and make strategic decisions based on the insights.
In the broadest sense, you can apply data science to gain insights about your business and improve your operational efficiency, or you may want to deliver AI-based applications to your end-customers. In the former case, the end-consumer would be your company, in the latter – your customers.
Customer-facing apps and fraud detection
Customer-facing applications powered by machine learning algorithms solve your customers' problems. People may use these products for their daily activities or to do work tasks faster and easier.
These are examples of customer-facing solutions that need data science engagement:
Virtual text and voice assistants (e.g. Mezi travel assistant or Expedia chatbot)
Recommendation engines for eCommerce and over-the-top media service providers (e.g. Amazon and Netflix recommendation systems)
Price prediction engines (e.g. Fareboom or Kayak fare predictions)
Apps for conversion of speech into text (e.g. the IBM Watson Speech to Text or Voice Assistant by QuanticApps)
Sound recognition and analysis applications (e.g. Do I Grind healthcare app)
Image editing applications (e.g. Prisma app)
Image recognition apps and features (e.g. credit card recognition with the Uber app)
Real-time visual and voice translators (e.g. Google Translate app or iTranslate Voice)
Document classification apps (e.g. Knowmail)
There’s also a group of fraud detection products that employs both data science and traditional programming techniques to build.
Business analytics: business intelligence and statistical analytics
Business analytics (BA) is the exploration of data through statistical and operations analysis. The purpose of BA is to monitor business processes and to use insights from data for making informed decisions.
Business analytics techniques can be divided into two groups – business intelligence and statistical analysis. Companies with business intelligence (BI) expertise analyze and report on historical data. Insights into past events allow companies to make strategic decisions regarding current operations and development options. Statistical analytics (SA) allows for digging deeper while exploring a problem. For instance, you can find out why customers prefer booking from OTAs rather than from your hotel site this week or whether or not a particular user buys products after reading an email about current deals.
Business analytics can be used for:
Data management
Dashboards and scorecards development
Big data analysis
Price, sales, or demand forecasting
Client analysis
Sentiment analysis on social media
Risk analysis
Market and customer segmentation
Customer lifetime value prediction
Upsell opportunity analysis, etc.
Business analytics allows for solving problems of various complexity, from simple reporting to advice on measures for risk mitigation or operations improvement. And to address these problems, you may use different types of analytics. There are four analytics types: descriptive, diagnostic, predictive, and prescriptive.
Four types of analytics
Let’s suppose you own a car dealership. According to analysis, sales of new models are lower compared to the same period last year. This is an example of descriptive analytics and its purpose is to provide a performance overview. Descriptive analytics allows for answering the question what happened?
Please note that you don't need data scientists for most cases of descriptive analysis. You can do it by yourself with BI tools.
You can try to dig deeper and understand why people are purchasing fewer cars in your stores. One of the ways is to measure sales records against external data like industry trends or market data. In other words, to perform diagnostic analytics. This analytics type allows for figuring out why something happened.
If you want to estimate how many automobiles will be purchased during the next three months or whether they will be purchased or not, you need predictive analytics. This more complex analytics type uses insights of descriptive and diagnostic analytics to forecast the probability of future events and outcomes. In addition, you can evaluate a moment at which an event might happen. So, predictive analytics allows for determining what is likely to happen next to optimize ongoing operations.
Predictive analytics usually requires machine learning that you can read more about in our article.
If you want to know what measures to take to increase sales, embark on prescriptive analytics. This analytics type entails a set of actions that include evaluation of insights about past events, their cause, and various forecasts on upcoming events. With prescriptive analytics, entrepreneurs get recommendations on actions needed to eliminate future issues or benefit from emerging trends. Prescriptive analytics uses machine learning as well.
Imagine that every event in your business, be it a transaction, a website visit, or staffing changes, was filmed and became a part of an episode in a long series. Business analytics is the tool with which you can rewind the video, spot some facts, analyze them, and draw conclusions. But that’s not all you can do. BA allows you to understand the cause of events, predict what can happen next, and make an optimal plan to ensure that yours is a success story only.
2. Consider off-the-shelf products
Make sure you’ve analyzed off-the-shelf solutions before you start looking for a team. Such websites as KDnuggets and PCMag have listings of analytics solutions and SaaS companies. Some of them (e.g. Capterra) even specialize in software research. We’ll talk more about sources you can check out to find vendors or products below. Also, if you look for customer insights and use common CRM software to collect and maintain customer interaction data, find out whether your vendor provides modules that would address your problem.
However, off-the-shelf solutions may not support the functionality you need. Let’s assume, for example, to adjust your marketing campaign you need to analyze reviews that customers share on social media. As your data analytics platform doesn’t support this feature, you need the one that’s customized to analyze feedback in your domain. These are sentences, words, and word combinations customers may frequently use when discussing your services. For instance, reservation, room service, canceled, room key, or breakfast included if you run a hotel business. In this case, you may hire a team of data science specialists to build tailor-made software.
3. Study company listings
Now it’s time to find a consultant who will develop a solution specifically for you. We’ve picked several websites that may help you fill the workforce gap with qualified data scientists.
KDnuggets: listings of DS consulting, product companies, and analytics solutions
Besides news and articles, KDnuggets has publications that may be useful for businesses that want to leverage DS. The website has the list of data science consulting companies with brief descriptions of their expertise and office location. You can also find listings with product companies and analytics solutions clustered by technology and industries on the same page.
KDnuggets provides a list of consulting companies with brief notes about each of them. Source: KDnuggets
Experfy: a marketplace to find experts for big data, analytics, and BI projects
Experfy is a Harvard-based data science consulting marketplace where clients from various industries can partner with experts for short- and long-term projects. The platform allows clients to hire external teams or augment existing ones. While most experts are freelancers, clients can negotiate with them to work on-site if needed.
Users post their projects according to their practice area or technology, specifying goals and requirements. Experfy then solicits bids on clients’ behalf, narrows down potential contractors, and clients start getting expert proposals. Collaboration with freelance specialists is done in the Experfy Project Room – a private work environment. The platform deals with contacts and charges a 20 percent fee on each transaction.
Сlutch: verified reviews to take data-driven hiring decisions
Clutch is an independent B2B research company based in Washington DC. Its goal is to assist businesses in making informed hiring decisions and allow service providers to advertise what they do. Clutch surveys clients about their collaboration experience with companies registered on the website. Reviews are located on a company profile and include some information about the commentator and the project, as well as a feedback summary. A full review form is also available. Clients score providers on schedule, quality, cost, and willingness to refer, helping potential clients know the overall service quality.
Clutch has a list of the best big data analytics companies. The list includes more than 600 companies and constantly updates. You can filter providers by Clutch rank, number of reviews, company name, and review rating.
4. Look at the vendor portfolio: case studies and references
Once you have a vendor shortlist, start learning about the expertise of each of the potential consultants. Industries for which a company builds solutions must be the first thing to look at when researching its website. A data science consultancy company with a domain knowledge goes beyond delivering a solution: Specialists can consult you regarding the product development and usually spend less time studying your problem.
Case studies. Each company strives to prove it has sufficient expertise in its field. However, nicely written testimonials and client logos on the main site page don’t add as much credibility to a company as related case studies. A case study allows for learning about a client’s background, the technology and methods a consultancy company applied to solve a specific problem, and of course the outcomes. Level of analytics used in case studies is another source of guidance for a consultant search and selection.
Besides reading case studies, consider contacting previous clients or at least visiting their websites to evaluate a vendor’s competency. You can ask companies from your shortlist to provide client contacts for references.
Other marketing materials. News, press releases, and blog articles can tell much about a specialists’ proficiency, a company business strategy, and reputation. What events did a company participate in or hold? How many times has media or other industry players mentioned this company in their publications and in what context? Also, check corporate pages on social media to get a fuller picture of a potential partner’s expertise.
5. Interview a consulting company: evaluation phase
Finally, you have some companies on the list you want to contact. That means that the time has come for the most important part of the consultant search journey – an evaluation phase. Ideally, this phase is all about a meaningful dialog between consultant and potential client.
Problem exploration
First things first, data science experts must define whether it’s feasible to solve your problem with data science or AI. You must be ready to describe your problem in detail and provide experts with operational data. The quality and amount of data are also crucial because an algorithm performance depends directly on them.
Lack of a clear question to answer (30.4 percent) and data either unavailable or difficult to access (30.2 percent) are other difficulties that experts wished they didn’t know about.
The top 15 answers about difficulties faced at work from data scientists. Source: Kaggle
Scope of work definition and preliminary results
A reliable consultant always takes time to verify whether a client has enough high-quality data before describing a scope of work and negotiating contract terms. Experts generally study a problem, research scientific articles, feed client data to algorithms that can be potentially used for a project, and present preliminary results – the estimated accuracy of an algorithm for a solution.
For instance, Fareboom CEO Marko Cadez approached our data science team with the goal upgrading existing travel booking engine with a fare-price prediction feature. Fareboom is an airline ticket search and booking site where customers can find the cheapest travel deals. Our team researched nearly 1 billion stored fares (which took them about 6 months) before making any assumptions regarding prediction confidence. Eventually, the team presented models that showed 80 percent prediction accuracy when tested on real data. It's worth noting that not every data science-related project takes so long in the initial stages. But you must be ready for long-term cooperation as DS projects require time.
Return on investment (ROI) estimation
Finding a solution to a challenge isn’t enough. The investment in the solution must pay off within a specific amount of time. So, one of the key tasks for a vendor team is to calculate the difference between a solution cost and return on investment.
For example, your company loses a significant amount of money due to fraud. The simplest way to eliminate nearly 60 percent of suspicious traffic is to buy a list of IPs from which fraudulent activity was detected. An off-the-shelf software would block another part of potentially fraudulent transactions.
But if you must achieve 95-100 percent fraud detection accuracy (which isn’t possible with most commercial solutions), then hiring a data science team to develop a model is reasonable. The higher the detection accuracy desired, the bigger the investment required. If you spend at least $250,000 on ML solutions every year, then allocating $50,000 on a custom solution is a grounded decision. Medium and small businesses must think twice before developing a complex solution that takes more than a year to pay off. It’s important to evaluate whether a sophisticated ML solution works well for an organization at its current phase of development.
Consider hiring data science consultant teams that honestly evaluate solution options, their cost, and the value of these solutions in terms of ROI.
Final word
The abundance of data science and AI consultants won't confuse you if you have a clear project goal and at least a basic understanding of what kind of expertise you need to achieve it. At the same time, a vendor must research your problem to determine whether it can be solved by means of DS or AI. If it seems appropriate to leverage these technologies, then a consultancy must provide you with a scope of work, present preliminary results, and evaluate project cost and return on investment.
These are additional factors to consider before making a final choice:
Data science team training. Find out whether a vendor provides knowledge sharing if you plan to form a data science team in the future.
Other expertise. A company that provides related services besides modeling can help you define a product vision.
Cooperation approach. Discuss partnership details with a potential consultant. For instance, teams following the Lean project management framework will require you or your company representative spend the entire length of the project with the development team.
