Business Problem

An organization’s personnel are key to its success, but the right people can be hard to find. SHRM (the Society for Human Resource Management) estimated in 2017 that between applicants, referrals, and agencies, there are 100 applicants to every hire.

Recruiters dealing with high numbers of applicants are forced to process them extremely quickly—the notorious “six seconds” per resume rule—rather than spending time going deeper with the best candidates and using their time to craft a compelling value proposition for the candidate. Automated screenings can also give applicants results more quickly and dramatically speed up the hiring process.

Intelligent Solution

With AI, organizations can identify candidates who have the right background and credentials to be successful in the role. The explainable insights from AI models (e.g., the relative importance of education vs job experience for new entry-level hires) can provide valuable guidance for recruiters and hiring managers. Prediction Explanations (e.g., individual callouts highlighting what makes someone a particularly strong candidate) could also be used to inform a new hire’s onboarding process to proactively address any relative weaknesses.

Finally and most importantly, models are explainable, consistent, and can be documented to ensure compliance with regulatory guidelines and ensure fairness to applicants.

IMPORTANT: Many countries have laws in place to protect employees from discrimination in regards to hiring or employment decisions. Besides that, fairness is the right thing to do. It is incredibly important that you work closely and proactively with your organization’s HR and Legal and Compliance teams to ensure that the models you build will pass legal and ethical scrutiny before models are put into production (View the business implementation tab to learn more about this use case and Trusted AI).

Value Estimation

How would I measure ROI for my use case? 

Most HR departments track average cost per hire. This metric combines both internal and external factors (recruiter time, agency fees, recruiting bonuses, employee travel, etc.) and represents the total amount of money and time spent to bring a new hire into the organization. 

SHRM’s 2016 Human Capital Report benchmarked average cost per hire at $4,179. (This will vary by industry and job role, e.g., entry level roles will be lower.) If a machine learning algorithm can reduce the total candidate pool by 30% at the beginning of the hiring process, that can save recruiters time and dramatically reduce cost per hire. A 10% reduction in total cost per hire by reducing the demands on recruiters’ time would equate to over $400 saved per hire brought into the organization.

Problem Framing

A typical target variable for this use case is to predict whether an applicant will pass a recruiter screen, which is a binary classification problem. This prediction is usually a preliminary review done by the recruiter before passing an applicant to the hiring manager for consideration. 

However, defining a target can become complex and will need to be adapted to your process as it depends on the data your organization may or may not have. While many organizations ultimately want to predict a hire decision or even on-the-job performance, there may be data limitations based on how many people were actually hired into that role.

The target, i.e., the “end result” you are trying to predict, will define what features are included in the model; if the goal is to predict which new applicants will get passed by a recruiter to a hiring manager, then we cannot use hiring manager feedback in a model because, in practice, that feedback won’t be available yet. If the target is instead a hiring decision, then the model will do best when hiring manager feedback is included. The model should be trained on the available data at the time the decision is made.

These are some of the recommended features needed to build the model; you can add or remove features based on the nature of the data available and the requirements of the model you are trying to build. 

• Numeric and categorical features from a structured application (e.g., previous job history, employers, education credentials)
• Resume data, if available
• Source of application
• If required, external tools or resume parsers can be used to do pre-processing and provide additional structure to raw applicant data.

These datasets usually come from Greenhouse or a similar ATS (Applicant Tracking System). For jobs in which candidates don’t generally provide a resume, any kind of completed job application can be used provided it’s in machine-readable format (e.g., avoid scanned PDFs).

Sample Feature Set

Feature Name	Data Type	Description	Data Source	Example
Pass_Screen	Binary (Target)	Whether the applicant passes the hiring manager screen for a given role	ATS	True
Application Source	Categorical	Source of the application	ATS	Employee referral
Highest degree attained	Categorical	Highest educational credential	ATS	2-year college degree
Previous employers	Text	List of previous employers	ATS	Billy Jo’s Pizza
Educational studies	Text or Categorical	Dropdown or user-entered text describing educational study	ATS	Business Management
Resume	Text	Raw resume text (if available)	ATS (may need to be converted from PDF)
Questions asked on a job page	Numeric or Categorical	“How many years experience do you have working directly with customers?”	ATS
Job description	Text	Description of the position being hired for	Job postings

Data Preparation 

To prepare the data, applicant data from an ATS is converted to machine readable as needed (e.g., text fields are extracted from a PDF document). Each row of the training data represents an application rather than an applicant, as applicants may apply to different positions or to the same position multiple times. Any external data sources are considered and added in as new features.

For an applicant scoring model to be accurate, it should be specific. Similar roles can be grouped together, but fundamentally different roles should be trained with different models. This is where automation and iteration are helpful. For instance, a model trained on hires within a specific geography might reveal more concrete insights (e.g., a certain university is a good feeding program for new Analysts) than a national model. 

We should also be careful to exclude people from our training data who “failed” the recruiter screen but were actually qualified. Recruiters may decide not to interview applicants for a variety of reasons unrelated to their qualifications, including because the candidates themselves expressed that they weren’t interested. This data can usually be found in an Applicant Tracking System the (ATS). 

Model Training

DataRobot Automated Machine Learning automates many parts of the modeling pipeline. Instead of hand-coding and manually testing dozens of models to find the one that best fits your needs, DataRobot automatically runs dozens of models and finds the most accurate one for you, all in a matter of minutes. In addition to training the models, DataRobot automates other steps in the modeling process such as processing and partitioning the dataset.

While we will jump straight to deploying the model, you can take a look here to see how DataRobot works from start to finish and to understand the data science methodologies embedded in its automation. 

A few key modeling decisions for this use case:

• Partitioning: Hiring practices change over time in response to both the macroeconomic environment and organizational initiatives / hiring practices. An OTV (out of time validation) partitioning scheme will evaluate model performance on the most recent data and give a more accurate benchmark to how well the model will perform when deployed.
• Setting a threshold:  If the model is to be used as a pass/fail screen, explore the false positive and negative rates at different thresholds. Ultimately, organizational demand will also help determine the threshold. For example, if the hiring pipeline is sparse, the needs of the organization might necessitate a lower threshold (e.g., more candidates passed) than the optimal one determined in training. 
• Accuracy metrics: If the model is being used to stack-rank applicants, consider using AUC in addition to LogLoss as a measure of performance for binary classification. 

Decision Environment 

After you finalize a model, DataRobot makes it easy to deploy the model into your desired decision environment. Decision environments are the methods by which predictions will ultimately be used for decision making.

Decision Maturity 

Automation | Augmentation | Blend 

There are many ways to implement a hiring model in practice. Some organizations use a hiring model as a pass/fail screening tool to cut down on the number of applications that recruiters are required to read and review. This has the advantage of giving candidates an answer more quickly.

Other organizations use the score as a way to stack-rank applicants, allowing recruiters to focus on the most promising candidates first. The most sophisticated is a blended approach: set a relatively low pass/fail barrier so that the “automatic no” values are removed from the pipeline up front. From there, provide recruiters the scores and the Prediction Explanations to help them make better decisions faster. 

Model Deployment

All new applicants to a role should be scored on a batch basis (e.g., one batch request per hour). Predictions and Prediction Explanations should be returned and saved in the database underlying the Applicant Tracking System. 

Decision Stakeholders

• Decision Executors: Recruiters are the most direct consumers and will use the predictions on a daily or weekly basis.
• Decision Managers: Hiring managers and ultimately Chief Human Resources Officers and their teams are responsible for making sure that decisions are being made correctly.
• Decision Authors: Data scientists, HR analysts, and industrial/organizational psychologists are all well-positioned to build the models in DataRobot. IT Support or vendors can be brought in if there are particular data processing challenges (e.g., PDFs).

Decision Process

Here are some examples of decisions you can take using the predictions generated from the model.

• Remove candidates who don’t meet minimum qualifications from the hiring pipeline.
• Review the most promising candidates first.
• Accelerate the review process for recruiters and hiring managers by leveraging explainable insights to highlight each candidate’s comparative strengths and weaknesses.

Model Monitoring 

Models should be retrained when data drift tracking shows significant deviations between the scoring and training data. In addition, if there are significant changes to the role (e.g., a change in the requirements of a position), the model will need to be refreshed. In that case, teams may have to manually rescore historical applicants against the updated requirements before model retraining can occur. 

Finally, think carefully about how to evaluate model accuracy. If a model imposes a pass/fail requirement but failing applicants are never evaluated by the recruiters, then we will track False Positives (applicants predicted to pass who did not) but not False Negatives (applicants rejected who would have passed). In a blended scenario (stack-ranking + scores), the model is directly influencing the recruiters’ decision making, which would tend to make the model seem more accurate than it is. 

The best way to evaluate accuracy is to have recruiters score a certain number of applicants independently and evaluate the model accuracy based on those cases.

Trusted AI 

In addition to traditional risk analysis, AI Trust is essential for this use case. 

Bias & Fairness: HR decision makers need to be aware of the risks that come with automating decision making within HR. Specifically, models trained on historically biased hiring practices can learn and reflect those same biases. It is incredibly important to make sure that your organization involves the right decision makers and content experts when building models to ensure that they remain fair.

However, there is also opportunity here. Upturn (a think tank focused on justice in technology) published guidelines for ethical AI in hiring. They note both the risks and opportunities of using AI in this space, suggesting that “with more deliberation, transparency, and oversight, some new hiring technologies might be poised to help improve on [our current ethical] baseline.”

The key, they argue, is explainability. Machine learning in this space must be transparent, documented, and explainable. Using the suite of explainability tools in DataRobot, non-data scientist HR teams can understand:

• what features are important in a model
• model performance for key segments or demographic groups
• how the models are actually working in practice
• individual-specific explanations of *why* a model is returning the score it does 

This is particularly important for free text fields, where it is essential to understand and actively control what words and phrases the model is allowed to learn from. Importantly, these models are not “black boxes;” rather, they are fully transparent and controlled by the organization.

In addition to explainability, bias testing should be part of model evaluation. One bias test that may be appropriate is statistical parity. With statistical parity, your goal is to measure if different demographic groups have an equal probability of achieving a favorable outcome. In this case, that would mean testing whether protected groups (e.g., race, gender, ethnicity) pass the recruiting screen at equivalent rates. In US law, the four-fifths rule is generally used to determine if a personnel process demonstrates adverse impact. A selection rate for a group less than four-fifths (80%) of the rate for another comparable group is evidence of adverse impact.

Note: Leaders interested in ethics and bias should also consider attending DataRobot’s course on Ethical AI, which teaches executives how to identify ethical issues within machine learning and develop an ethics policy for AI. 

Business Problem

To operate, Federally Qualified Health Centers (FQHC) rely on funding through programs such as Medicare and Medicaid. They are required to provide medical services on a sliding scale and are often governed by a board that includes patients. Success of an FQHC is measured in part on the satisfaction of patients having received care; this feedback plays a direct role in a center’s ability to receive funding and continue serving the local community.

In addition to understanding how hospital operations have led to poor satisfaction historically, it’s also important to have the ability to flag potential risks in real time. This can provide insight into which patients are potentially not having a great experience and can be used by hospital administration to intervene, talk to the patient, and rectify any issues. Increased patient satisfaction is a positive outcome for the community that the FQHC serves, and ensures that the provider can continue to operate to its fullest.

Intelligent Solution

With AI, hospital administrators can, in real time, understand which patients are likely to leave the hospital with a bad experience. Knowing which patients may be unsatisfied with their care, coupled with the primary reasons why, provides useful information to hospital administrators who can seek out patients, discuss the quality of their stay, and take steps to improve satisfaction. AI informs hospital administrators with the information they need to increase patient satisfaction scores: an outcome that serves the community and ensures that the provider can continue to operate.

There are two primary parts involved in this particular solution:

• Predict which patients are likely to respond to a survey request.
• Predict if that patient will respond with a positive or negative review of the care.

Value Estimation

How would I measure ROI for my use case? 

The value of increased patient satisfaction comes in two primary ways.  First, patient satisfaction is a key indicator of how well an FQHC is meeting the healthcare needs of the community that it serves.  Therefore, taking steps in increasing satisfaction is a positive outcome for the surrounding community.  Additionally, a one percent increase in satisfaction scores can substantially impact that funding that an FQHC receives, allowing it to expand services, hire more clinicians and continue providing quality care for those that need it most.

Problem Framing 

The target variable for this use case should be aligned to the survey output by which the provider is measured.  In this case, we’ll consider the Press Ganey patient satisfaction survey that buckets responses in three categories: top, middle, and bottom box, with top box being the best and bottom box being the worst.  Therefore, the target will be if the patient’s survey response was in the top, middle, or bottom box.  Furthermore, this means that we can frame this problem in two ways: 

• Multiclass with three classes: top, middle, and bottom box.  This will be the approach we’ll consider during this tutorial.
• It may also make sense to have three different binary models, one for each box score.

Additionally, it might be useful to also develop an initial model that predicts the likelihood of a response to a survey request. In this case, the target variable can be whether or not patients responded to surveys historically. 

Key features that are important in predicting patient satisfaction are listed below. They encompass information about the patient, their stay, diagnosis, and the interaction with clinicians.

• Demographic information
• Clinical data including diagnoses codes
• Clinician notes
• Clinician performance data and demographics
• Patient history information such as number of previous visits, recency, and frequency of visits and reasons
• Patient location information such as distance to provider

Beyond the feature categories listed above, we suggest incorporating additional data your organization may collect that could be relevant to readmissions. As you will see later, DataRobot is able to quickly differentiate important vs. unimportant features. 

Many of these features are generally stored across proprietary data sources available in an EMR system: Patient Data, Diagnosis Data, Clinician notes, Admissions Data. Examples of EMR systems are Epic and Cerner. 

Sample Feature List

Feature Name	Data Type	Description	Data Source	Example
Response	Multiclass (target)	Top, middle or bottom box as 3,2 and 1, respectively	Provided by survey vendor	3
Age	Numeric	Patient age group	Patient Data	40
Clinical Notes	Text	Notes from nurse or doctor	Clinical Data
Number of past visits	Numeric	Count of the prior visits within a specific period.  Could also include all prior visits.	Patient Data	10
Distance	Numeric	Distance in miles between home and provider location	Patient Data	20
Diagnosis Code	Text (alpha numeric)	Code indicating the diagnosis upon arrival	Clinical Data	E10.9

Model Training

DataRobot Automated Machine Learning automates many parts of the modeling pipeline. Instead of hand-coding and manually testing dozens of models to find the one that best fits your needs, DataRobot automatically runs dozens of models and finds the most accurate one for you, all in a matter of minutes. In addition to training the models, DataRobot automates other steps in the modeling process such as processing and partitioning the dataset.

Take a look here to see how to use DataRobot from start to finish and how to understand the data science methodologies embedded in its automation.

There are a couple of key modeling decisions for this use case:

• Partitioning: It is possible that care procedures may change over time due to organizational or personnel changes.  Therefore, an OTV (out of time validation) partitioning scheme would be a good choice.  This approach will evaluate model performance on the most recent encounters and give a more accurate benchmark of how well the model will perform when deployed.
• Accuracy metrics: LogLoss

Decision Flow

After you are able to find the right model that best learns patterns in your data to predict patient satisfaction, DataRobot makes it easy to deploy the model into your desired decision environment. Decision environments are the ways in which the predictions generated by the model will be consumed by the appropriate stakeholders in your organization, and how these stakeholders will make decisions using the predictions to impact the overall process. 

Decision Maturity 

Automation | Augmentation | Blend

In practice, the output of these models can be consumed by a team focused on ensuring care satisfaction during a patient’s stay. A daily report or dashboard of patients and their predicted satisfaction scores can provide a guide for senior hospital administration and an understanding, in real time, of which patients are potentially dissatisfied with their care.  Furthermore, leveraging Prediction Explanations from DataRobot can be a useful way to understand the primary drivers of the dissatisfaction and allow the decision maker to address those issues directly.   

Model Deployment

All new applicants should be scored on a daily batch basis. Predictions and Prediction Explanations should be returned and saved in the database, which can be integrated into a BI dashboard or tool available to the hospital administrators. This allows them to continuously monitor which patients are likely to be dissatisfied with their stay. 

Decision Stakeholders

Decision Executors

Hospital administrators focused on patient experience. They are often under the organization of the Chief Patient Experience Officer of a provider or hospital.

Decision Managers

The Chief Patient Experience Officer is ultimately responsible for executing the strategy to improve patient experience and responsible for the use of the model output.

Decision Authors

Analytics professionals and data scientists with strong understanding of the patient data and processes are best positioned to develop these models as well as develop a meaningful representation of the output in the form of dashboards or reports. Data engineers and IT support is needed to ensure that the stakeholders can receive timely predictions reliably since these will require daily action from hospital administration.

Decision Process

• Coaching staff and clinicians are provided with information about potential issues.
• Prediction Explanations can guide staff on the top drivers for dissatisfaction.
• Staff can also speak directly to patients who are predicted to score low to understand what issues they may have regarding their experience and addressing those issues directly. 

Model Monitoring

Models should be retrained when data drift tracking shows significant deviations between the scoring and training data. 

Business Problem

A critical component of any supply chain network is to prevent parts shortages, especially when they occur at the last minute. Parts shortages not only lead to underutilized machines and transportation, but also cause a domino effect of late deliveries through the entire network. In addition, the discrepancies between the forecasted and actual number of parts that arrive on time prevent supply chain managers from optimizing their materials plans.

Parts shortages are often caused by delays in their shipment. To mitigate the impact delays will have on their supply chain, manufacturers adopt approaches such as holding excess inventory, optimizing product designs for more standardization, and moving away from single-sourcing strategies. However, most of these approaches add up to unnecessary costs for parts, storage, and logistics.

In many cases, late shipments persist until supply chain managers can evaluate root cause and then implement short term and long term adjustments that prevent them from occurring in the future. Unfortunately, supply chain managers have been unable to efficiently analyze historical data available in MRP systems because of the time and resources required.

Intelligent Solution

AI helps supply chain managers reduce parts shortages by predicting the occurrence of late shipments, giving them time to intervene. By learning from past cases of late shipments and their associated features, AI applies these patterns to future shipments to predict the likelihood that those shipments will also be delayed. Unlike complex MRP systems, AI provides supply chain managers with the statistical reasons behind each late shipment in an intuitive but scientific way. For example, when AI notifies supply chain managers of a late shipment, it will also explain why, offering reasons such as the shipment’s vendor, mode of transportation, or country.

Then, using this information, supply chain managers can apply both short term and long term solutions to preventing late shipments. In the short term, based on their unique characteristics, shipment delays can be prevented by adjusting their transportation or delivery routes. In the long term, supply chain managers can conduct aggregated root-cause analyses to discover and solve the systematic causes of delays. They can use this information to make strategic decisions, such as choosing vendors located in more accessible geographies or reorganizing their shipment schedules and quantities.

Value Estimation

How would I measure ROI for my use case? 

The ROI for implementing this solution can be estimated by considering the following factors: 

1. Starting with the manufacturing company and production line stoppage, the cycle time of the production process can be used to understand how much of the production loss relates to part shortages. For example, if the cycle time (time taken to complete one part) is 60 seconds and each day 15 minutes of production are lost to part shortages, then total production loss is equivalent to 15 products, which can be translated to loss in profit of 15 products in a day. A similar calculation can be used to estimate annual loss due to part shortage.
2. For a logistic provider, predicting part shortages early can increase savings in terms of reduced inventory. This can be roughly measured by capturing the difference in maintaining parts stock before and after implementation of the AI solution. The difference in stock when multiplied with holding and inventory cost per unit gives the overall ROI. Furthermore, in cases when the demand for parts is left unfulfilled (because of part shortages), the opportunity cost related to the unsatisfied demand would directly result in loss of respective business opportunity.

About the Data

For illustrative purposes, we use a sample dataset provided by the President’s Emergency plan for AIDS relief (PEPFAR), which is publicly available on Kaggle. This dataset provides supply chain health commodity shipment and pricing data. Specifically, the dataset identifies Antiretroviral (ARV) and HIV lab shipments to supported countries. In addition, the dataset provides the commodity pricing and associated supply chain expenses necessary to move the commodities to other countries for use. We use this dataset to represent how a manufacturing or logistics company can leverage AI models to improve their decision making.

Problem Framing

The target variable for this use case is whether or not the shipment would be delayed (Binary; True or False, 1 or 0, etc.). This choice in target (Late_delivery) makes this a binary classification problem. The distribution of the target variable is imbalanced, with 11.4% being 1 (late delivery) and 88.6% being 0 (on time delivery). (See here for more information about imbalanced data in machine learning.)

The features below represent some of the factors that are important in predicting delays. The feature list encompasses all of the information in each purchase order sent to the vendor, which would eventually be used to make predictions of delays when new purchase orders are raised.

Beyond the features listed below, we suggest incorporating any additional data your organization may collect that could be relevant to delays. As you will see later, DataRobot is able to quickly differentiate important/unimportant features. 

These features are generally stored across proprietary data sources available in the ERP systems of the organization. 

Sample Feature List

Feature Name	Data Type	Description	Data Source	Example
Supplier name	Categorical	Name of the vendor who would be shipping the delivery	Purchase order	Ranbaxy, Sun Pharma etc.
Part description	Text	The details of the part/item that is being shipped	Purchase order	30mg HIV test kit, 600mg Lamivudine capsules
Order quantity	Numeric	The amount of item that was ordered	Purchase order	1000, 300 etc.
Line item value	Numeric	The unit price of the line item ordered	Purchase order	0.39, 1.33
Scheduled delivery date	Date	The date at which the order is scheduled to be delivered	Purchase order	2-Jun-06
Delivery recorded date	Date	The date at which the order was eventually delivered	ERP system	2-Dec-06
Manufacturing site	Categorical	The site of the vendor where the manufacturing was done since the same vendor can ship parts from different sites	Invoice	Sun Pharma, India
Product Group	Categorical	The category of the product that is ordered	Purchase order	HRDT, ARV
Mode of delivery	Categorical	The mode of transport for part delivery	Invoice	Air, Truck
Late Delivery	Target (Binary)	Whether the delivery was late or on-time	ERP System, Purchase Order	0 or 1

Data Preparation 

The dataset contains historical information on procurement transactions. Each row of analysis in the dataset is an individual order that is placed and whose delivery needs to be predicted. Every order has a scheduled delivery date and actual delivery date, and the difference between these were used to define the target variable (Late_delivery). If the delivery date surpassed the scheduled date, then the target variable had a value 1, else 0. Overall, the dataset contains about 10,320 rows and 26 features, including the target variable. 

Model Training

DataRobot Machine Learning automates many parts of the modeling pipeline. Instead of hand-coding and manually testing dozens of models to find the one that best fits your needs, DataRobot automatically runs dozens of models and finds the most accurate one for you, all in a matter of minutes. In addition to training the models, DataRobot automates other steps in the modeling process such as processing and partitioning the dataset.

While we will jump straight to interpreting the model results, you can take a look here to see how DataRobot works from start to finish, and to understand the data science methodologies embedded in its automation. 

Something to highlight is, since we are dealing with an imbalanced dataset, DataRobot automatically recommends using LogLoss as the optimization metric to identify the most accurate model, being that it is an error metric which penalizes wrong predictions. 

For this dataset, DataRobot found the most accurate model to be Extreme Gradient Boosting Tree Classifier with unsupervised learning features using open source XGboost library.

Interpret Results

To give transparency on how the model works, DataRobot provides both global and local levels of model explanations. In broad terms, the model can be understood by looking at the Feature Impact graph, which shows the relative importance of the features in the dataset in relation to the selected target variable. The technique adopted by DataRobot to build this plot is called Permutation Importance. 

As you can see, the model identified Pack Price, Country, Vendor, Vendor INCO Term, and Line item Insurance as some of the most critical factors affecting delays in the parts shipments. 

Moving to the local view of explainability, DataRobot also provides Prediction Explanations that enable you to understand the top 10 key drivers for each prediction generated. This offers you the granularity you need to tailor your actions to the unique characteristics behind each part shortage. 

For example, if a particular country is a top reason for a shipment delay, such as Nigeria or South Africa, you can take actions by reaching out to vendors in these countries and closely monitoring the shipment delivery across these routes.

Similarly, if there are certain vendors that are amongst the top reasons for delays, you can reach out to these vendors upfront and take corrective actions to avoid any delayed shipments which would affect the supply chain network. These insights help businesses make data-driven decisions to improve the supply chain process by incorporating new rules or alternative procurement sources.

For text variables, such as Part description (included in the dataset), we can look at Word Clouds to discover the words or phrases that are highly associated with delayed shipments. Text features are generally the most challenging and time consuming to build models for, but with DataRobot each individual text column is automatically fitted as an individual classifier and is directly preprocessed with NLP techniques (tf-idf, n grams, etc.) In this case, we can see that the items described as nevirapine 10 mg are more likely to get delayed in comparison to other items.

Evaluate Accuracy

To evaluate the performance of the model, DataRobot by default ran five-fold cross validation and the resulting AUC score (for ROC Curve) was around 0.82. Since the AUC score on the holdout set (unseen data) was also around 0.82, we can be reassured that the model is generalizing well and is not overfitting. The reason we look at the AUC score for evaluating the model is because AUC ranks the output (i.e., the probability of delayed shipment) instead of looking at actual values. The Lift Chart below shows how the predicted values (blue line) compared to actual values (red line) when the data is sorted by predicted values. We see that the model has slight under-predictions for the orders which are more likely to get delayed. But overall, the model does perform well. Furthermore, depending on the problem being solved, you can review the confusion matrix for the selected model and, if required, adjust the prediction threshold to optimize for precision and recall. 

Decision Environment 

After the right model has been chosen, DataRobot makes it easy to deploy the model into your desired decision environment. Decision environments are the ways in which the predictions generated by the model will be consumed by the appropriate stakeholders in your organization, and how these stakeholders will make decisions using the predictions to impact the overall process. 

Decision Maturity 

Automation | Augmentation | Blend

The predictions from this use case can augment the decisions of the supply chain managers as they foresee any upcoming delays in logistics. It acts as an intelligent machine that, combined with the decisions of the managers, help improve your entire supply chain network. 

Model Deployment 

The model can be deployed using the DataRobot Prediction API. A REST API endpoint which would be used to bounce back predictions in near real time when new scoring data from new orders are received. 

Once the model has been deployed (in whatever way the organization decides), the predictions can be consumed in several ways. For example, a front-end application that acts as the supply chain’s reporting tool can be used to deliver new scoring data as an input to the model, which then bounces back predictions and Prediction Explanations in real time.

Decision Stakeholders

The predictions and Prediction Explanations would be used by supply chain managers or logistic analysts to help them understand the critical factors or bottlenecks in the supply chain.

Decision Executors

Decision executors are the supply chain managers and procurement teams who are empowered with the information they need to ensure that the supply chain network is free from bottlenecks. These personnel have strong relationships with vendors and the ability to take corrective action using the model’s predictions.

Decision Managers

Decision managers are the executive stakeholders such as the Head of Vendor Development who manage large scale partnerships with key vendors. Based on the overall results, these stakeholders can perform quarterly reviews of the health of their vendor relationships to make strategic decisions on long-term investments and business partnerships.

Decision Authors

Decision authors are the business analysts or data scientists who would build this decision environment. These analysts could be the engineers/analysts from the supply chain, engineering, or vendor development teams in the organization who usually work in collaboration with the supply chain managers and their teams.

Decision Process

The decisions that the managers and executive stakeholders would take based on the predictions and Prediction Explanations for identifying potential bottlenecks would be reaching out and collaborating with appropriate vendor teams in the supply chain network based on data-driven insights. The decisions could be both long- and short-term based on the severity of the impact of shortages on the business.

Model Monitoring 

One of the most critical components in implementing an AI is having the ability to track the performance of the model for data drift and accuracy. With DataRobot MLOps, you can deploy, monitor and manage all models across the organization through a centralized platform. Tracking model health is very important for proper model lifecycle management, similar to product lifecycle management. 

Implementation Risks

One of the major risks in implementing this solution in the real world is adoption at the ground level. Having strong and transparent relationships with vendors is also critical in taking corrective action. The risk is that vendors may not be ready to adopt a data-driven strategy and trust the model results.