Prerequisites<\/h2>\n
To try out the solution in your own account, make sure that you have the following in place:<\/p>\n
In this solution, we focus on SageMaker components. We use SageMaker training jobs to train the churn prediction model and a SageMaker endpoint to deploy the model. We use Amazon Simple Storage Service<\/a> (Amazon S3) to store the training data and model artifacts, and Amazon CloudWatch<\/a> to log training and endpoint outputs. The following figure illustrates the architecture for the solution.<\/p>\n In this post, we use a mobile operator\u2019s historical records of which customers ended up churning and which continued using the service. The data also includes transcriptions of the latest phone call conversations between the customer and the agent (which could also be the streaming transcription as the call is happening). We can use this historical information to train an ML classifier model, which we can then use to predict the probability of customer churn based on the customer\u2019s profile information and the content of the phone call transcription. We create a SageMaker endpoint to make real-time predictions using the model and provide more insight to customer service agents as they handle customer phone calls.<\/p>\n The dataset we use is synthetically generated and available under the CC BY 4.0 license. The data used to generate the numerical and categorical features is based on the public dataset KDD Cup 2009: Customer relationship prediction<\/a>. We have generated over 50,000 samples and randomly split the data into 45,000 samples for training and 5,000 samples for testing. In addition, the phone conversation transcripts were synthetically generated using the GPT2 (Generative Pre-trained Transformer 2<\/a>) algorithm. The data is hosted on Amazon S3.<\/p>\n More details on customer churn classification models using similar data, and also step-by-step instructions on how to build a binary classifier model using similar data, can be found in the blog post Predicting Customer Churn with Amazon Machine Learning<\/a>. That post is focused more on binary classification using the tabular data. This blog post approaches this problem from a different perspective, and brings in natural language processing (NLP) by processing the context of agent-customer phone conversations.<\/p>\n The following are the attributes (features) of the customer profiles dataset:<\/p>\n The last attribute, We don\u2019t go over exploratory data analysis in this post. For more details, see Predicting Customer Churn with Amazon Machine Learning<\/a> and the Customer Churn Prediction with XGBoost<\/a> sample notebook.<\/p>\n The training script is developed to allow the ML practitioner to pick and choose the features used in training. For example, we don\u2019t use all the features in training. We focus more on the maturity of the customer\u2019s account, number of times the customer has contacted customer service, type of plan they have, and transcription of the latest phone call. You can use additional features in training by including the list in the hyperparameters, as we show in the next section.<\/p>\n The transcription of customer-agent phone call in the For this post, we use the SageMaker PyTorch Estimator<\/a> to build a SageMaker estimator using an Amazon-built Docker container that runs functions defined in the supplied Also, visit Available Deep Learning Containers Images<\/a> on GitHub to get a list of supported PyTorch versions. At the time of this writing, the latest version available is PyTorch 1.8.1 with Python version 3.6. You can update the framework version to the latest supported version by changing the The hyperparameters dictionary defines which features we want to use for training and also the number of trees in the forest ( If you launched the SageMaker JumpStart solution in your account, the custom scripts are available in your Studio files. We use the The purpose of the script presented in this post is to provide an example of how you can develop your own custom feature transformation pipeline. You can apply other transformations to the data based on your specific use case and the nature of your dataset, and make it as complex or as simple as you want. For example, depending on the nature of your dataset and the results of the exploratory data analysis, you may want to consider normalization, log transformation, or dropping records with null values. For a more complete list of feature transformation techniques, visit SKLearn Dataset Transformations<\/a>.<\/p>\n The following code snippet shows you how to instantiate these transformers for numerical and categorical features, and how to apply them to your dataset. More details on how these are done in the training script is available in the Now let\u2019s focus on the textual data. We use Hugging Face sentence transformers<\/a>, which you can use for sentence embedding generation. They come with pre-trained models that you can use out of the box based on your use case. In this post, we use the bert-base-nli-cls-token <\/a>model, which is described in Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks<\/a>.<\/p>\n Recently, SageMaker introduced new Hugging Face Deep Learning Containers (DLCs)<\/a> that enable you to train, fine-tune, and run inference using Hugging Face models for NLP on SageMaker. In this post, we use the PyTorch container and a custom training script. For this purpose, in our training script, we define a Now that the dataset is processed and ready to be consumed by an ML model, we can train any classifier model to predict if a customer will churn or not. In addition to predicting the class (0\/1 or true\/false) for customer churn, these models also generate the probability of each class, meaning the probability of a customer churning. This is particularly useful for customer service teams for strategizing the incentives or upgrades they can offer to the customer based on how likely the customer is to cancel the service or subscription. In this post, we use the SKLearn RandomForestClassifier<\/a> model. You can choose from many hyperparameters for this model and also optimize the hyperparameters for a more accurate model prediction by using strategies like grid search, random search, and Bayesian search. SageMaker automatic hyperparameter tuning <\/a>can be a powerful tool for this purpose.<\/p>\n Training the model in When you\u2019re comfortable with the performance of your model, you can move to the next step, which is deploying your model for real-time inference.<\/p>\n When the training is complete, you can deploy the model as a SageMaker hosted endpoint<\/a> for real-time inference, or use the model for offline batch inference, using SageMaker batch transform<\/a>. The task of performing inference (either real time or batch) is handled by four main functions in the custom script:<\/p>\n The following diagram illustrates this process.<\/p>\n The following script is a snippet of the To deploy the model, when the training is complete, we use the This deploys the model as an endpoint predictor. After deployment is complete, we can use that to make predictions on sample data. Let\u2019s determine the probability of churn for a hypothetical customer:<\/p>\n In this case, the probability of churn is about 31%. For the same customer, we change the transcript to \u201cI have been using your service for 6 months and I am disappointed in your customer service.\u201d The probability of churn increases to over 46%. This demonstrates that a change in the customer\u2019s sentiment affects the probability of churn.<\/p>\n To clean up the resources and stop incurring charges in your account, you can delete the endpoint:<\/p>\n As we explained earlier, you can use additional features in training and also incorporate more feature transformers in the feature engineering pipeline, which can help improve model performance.<\/p>\n In addition, now that you have a working endpoint that is performing real-time inference, you can use it for your applications or website. However, your SageMaker endpoint is still not public facing, so you need to build an API Gateway to allow external traffic to your SageMaker endpoint. Amazon API Gateway<\/a> is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. You can use API Gateway to present an external-facing, single point of entry for SageMaker endpoints, and provide security, throttling, authentication, firewall as provided by AWS WAF<\/a>, and more. With API Gateway mapping templates, you can invoke your SageMaker endpoint with a REST API request and receive an API response back without needing any intermediate AWS Lambda<\/a> functions, thereby improving the performance and cost-effectiveness of your applications.<\/p>\n To create an API Gateway and use it to perform real-time inference with your SageMaker endpoint (see the following architecture), you can follow the instructions outlined in Creating a machine learning-powered REST API with Amazon API Gateway mapping templates and Amazon SageMaker<\/a>.<\/p>\n In addition, you can use Amazon Transcribe<\/a> to generate transcriptions of recorded customer-agent conversations and use them for training purposes, and also use Amazon Transcribe streaming<\/a> to send the conversation audio stream and receive a stream of text in real time. You can use this text stream to add a real-time speech-to-text capability to your applications and also send that text to the endpoint and provide customer churn insights to your customer service agents in real time.<\/p>\n In this post, we explained an end-to-end solution for creating a customer churn prediction model based on customer profiles and customer-agent call transcriptions. The solution included training a PyTorch model with a custom script and creating an endpoint for real-time model hosting. We also explained how you can create a public-facing API Gateway that can be securely used in your mobile applications or website. In addition, we explained how you can use Amazon Transcribe for batch or real-time transcription of customer-agent conversations, which you can use for training of your model or real-time inference.<\/p>\n For more SageMaker examples, visit the Amazon SageMaker Examples<\/a> GitHub repo. For more PyTorch BYO script examples, visit the following GitHub repository<\/a>. For more SageMaker Python examples for MXNet, TensorFlow, and PyTorch, visit the Amazon SageMaker Pre-Built Framework Containers and the Python SDK<\/a> GitHub repo. Additional information about SageMaker is available in the technical documentation.<\/a><\/p>\n![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker.png\"){kind=spacer}
<\/a>The JumpStart solution launch creates the resources properly set up and configured to successfully run the solution.<\/p>\nArchitecture overview<\/h2>\n
![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker-1.png\"){kind=spacer}
<\/a><\/p>\nExploring the data<\/h2>\n
\n
Y<\/code>, is known as the target feature<\/em>, or the feature we want the ML model to predict. Because the target feature is binary (true\/false), the type of modeling is a binary classification model. The model we train later in this post predicts the likelihood of churn as well.<\/p>\ntext<\/code> column is synthetic text generated by ML models using the GPT2 algorithm. Its purpose is to show how you can apply this solution to real-world customer service phone conversations. GPT2 is an unsupervised transformer language model developed by OpenAI<\/a>. It\u2019s a powerful generative NLP model that excels in processing long-range dependencies, and is pre-trained on a diverse corpus of text. For more details on how to generate text using GPT2, see Experimenting with GPT-2 XL machine learning model package on Amazon SageMaker<\/a> and the Creative Writing using GPT2 Text Generation<\/a> example notebook.<\/p>\nTrain the model<\/h2>\n
entry_point<\/code> Python script within a SageMaker training job. The training job is started by calling .fit()<\/code> on this estimator. Later, we deploy the model by calling the .deploy()<\/code> method on the estimator. Visit Amazon SageMaker Python SDK<\/a> technical documentation for more details on preparing PyTorch scripts for SageMaker training and using the PyTorch Estimator.<\/p>\nframework_version<\/code> parameter in the PyTorch Estimator. You can also use SageMaker utility API image URIs<\/a> to get the latest list of supported versions.<\/p>\nn-estimators<\/code>) for the model. You can add any other hyperparameters for the RandomForestClassifier<\/a>; however, you also need revise your custom training script to receive these parameters in the form of arguments (using the argparse library<\/a>) and add them to your model. See the following code:<\/p>\nhyperparameters = { \"n-estimators\": 100, \"numerical-feature-names\": \"CustServ Calls,Account Length\", \"categorical-feature-names\": \"plan,limit\", \"textual-feature-names\": \"text\", \"label-name\": \"y\"\n} estimator = PyTorch( framework_version='1.8.1', py_version='py3', entry_point='entry_point.py', source_dir='path\/to\/source\/directory', hyperparameters=hyperparameters, role=iam_role, instance_count=1, instance_type='ml.p3.2xlarge', output_path='s3:\/\/path\/to\/output\/location', code_location='s3:\/\/path\/to\/code\/location', base_job_name=base_job_name, sagemaker_session=sagemaker_session, train_volume_size=30\n)<\/code><\/pre>\n<\/p><\/div>\nentry_point.py<\/code> script. This script receives a list of numerical features, categorical features, textual features, and the target label, and trains a SKLearn RandomForestClassifier <\/a>on the data. However, the key here is processing the features before using them in the classifier, especially the call transcription. The following figure shows this process, which applies imputing to numerical features and replaces missing values with mean, one-hot encoding to categorical features, and embeds transformers to textual features.<\/p>\n![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker-2.png\"){kind=spacer}
<\/a><\/p>\nentry_point.py<\/code> script that is launched in your files by the JumpStart solution.<\/p>\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import OneHotEncoder # Instantiate transformers\nnumerical_transformer = SimpleImputer(missing_values=np.nan, strategy='mean', add_indicator=True)\ncategorical_transformer = OneHotEncoder(handle_unknown=\"ignore\") # Train transformers on data, and store transformers for future use by predict function\nnumerical_transformer.fit(numerical_features)\njoblib.dump(numerical_transformer, Path(args.model_dir, \"numerical_transformer.joblib\")) categorical_transformer.fit(categorical_features)\njoblib.dump(categorical_transformer, Path(args.model_dir, \"categorical_transformer.joblib\")) # transform the data\nnumerical_features = numerical_transformer.transform(numerical_features)\ncategorical_features = categorical_transformer.transform(categorical_features)<\/code><\/pre>\n<\/p><\/div>\nBertEncoder<\/code> class based on Hugging Face SentenceTransformer<\/code> and define the pre-trained model as bert-base-nli-cls-token<\/code>, as shown in the following code. The reason for this is to be able to apply the transformer to the dataset in the same way as the other dataset transformers, with the applying .transform()<\/code> method. The benefit of using Hugging Face pre-trained models is that you don\u2019t need to do additional training to be able to use the model. However, you can still fine-tune the models with custom data, as described in Fine-tuning a pretrained model<\/a>.<\/p>\nfrom sentence_transformers import SentenceTransformer # Define a class for BertEncoder\nclass BertEncoder(BaseEstimator, TransformerMixin): def __init__(self, model_name='bert-base-nli-cls-token'): self.model = SentenceTransformer(model_name) self.model.parallel_tokenization = False def fit(self, X, y=None): return self def transform(self, X): output = [] for sample in X: encodings = self.model.encode(sample) output.append(encodings) return output # Instantiate the class textual_transformer = BertEncoder() # Apply the transformation to textual features\ntextual_features = textual_transformer.transform(textual_features)<\/code><\/pre>\n<\/p><\/div>\nentry_point.py<\/code> is handled by the train_fn()<\/code> function in the custom script. This function is called when the .fit()<\/code> method is applied to the estimator. This function also stores the trained model and trained data transformers on Amazon S3. These files are used later by model_fn()<\/code> to load the model for inference purposes.<\/p>\ntrain_fn()<\/code> also includes evaluation of the trained model, and provides accuracy scores for the model for both train and test datasets. This helps you better evaluate model performance. Because this is a classification problem, we recommend including other metrics in your evaluation script, for example F1 score<\/a>, ROC AUC score<\/a>, and recall score<\/a>, the same way we added accuracy scores. These are printed as the training progresses. Because we\u2019re using synthetic data for training the model in this example notebook, especially for the agent-customer call transcription, we\u2019re not expecting to see high-performing models with regards to classification metrics, and therefore we\u2019re not focusing on these metrics in this example. However, when you use your own data, you should consider how each classification metric could impact the applicability of the model to your use case. Training this model on 45,000 samples on an ml.p3.2xlarge instance takes about 30 minutes.<\/p>\nestimator.fit({ 'train': 's3:\/\/path\/to\/your\/train.jsonl')), 'test': 's3:\/\/path\/to\/your\/test.jsonl'))\n})<\/code><\/pre>\n<\/p><\/div>\nDeploy the model<\/h2>\n
\n
input_fn()<\/code> processes the input data<\/li>\nmodel_fn()<\/code> loads the trained model artifacts from Amazon S3<\/li>\npredict_fn()<\/code> makes predictions<\/li>\noutput_fn()<\/code> prepares the model output<\/li>\n<\/ul>\n![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker-3.png\"){kind=spacer}
<\/a><\/p>\nentry_point.py<\/code> script, and shows how the four functions work together to perform inference:<\/p>\n# Model function to load the trained model and trained transformers from S3\ndef model_fn(model_dir): print('loading feature_names') numerical_feature_names, categorical_feature_names, textual_feature_names = load_feature_names(Path(model_dir, \"feature_names.json\")) print('loading numerical_transformer') numerical_transformer = joblib.load(Path(model_dir, \"numerical_transformer.joblib\")) print('loading categorical_transformer') categorical_transformer = joblib.load(Path(model_dir, \"categorical_transformer.joblib\")) print('loading textual_transformer') textual_transformer = BertEncoder() classifier = joblib.load(Path(model_dir, \"classifier.joblib\")) model_assets = { 'numerical_feature_names': numerical_feature_names, 'numerical_transformer': numerical_transformer, 'categorical_feature_names': categorical_feature_names, 'categorical_transformer': categorical_transformer, 'textual_feature_names': textual_feature_names, 'textual_transformer': textual_transformer, 'classifier': classifier } return model_assets # Input Preparation Function to receive the request body and ensure proper format\ndef input_fn(request_body_str, request_content_type): assert ( request_content_type == \"application\/json\" ), \"content_type must be 'application\/json'\" request_body = json.loads(request_body_str) return request_body # Predict function to make inference\ndef predict_fn(request, model_assets): print('making batch') request = [request] print('extracting features') numerical_features, categorical_features, textual_features = extract_features( request, model_assets['numerical_feature_names'], model_assets['categorical_feature_names'], model_assets['textual_feature_names'] ) print('transforming numerical_features') numerical_features = model_assets['numerical_transformer'].transform(numerical_features) print('transforming categorical_features') categorical_features = model_assets['categorical_transformer'].transform(categorical_features) print('transforming textual_features') textual_features = model_assets['textual_transformer'].transform(textual_features) # Concatenate Features print('concatenating features') categorical_features = categorical_features.toarray() textual_features = np.array(textual_features) textual_features = textual_features.reshape(textual_features.shape[0], -1) features = np.concatenate([ numerical_features, categorical_features, textual_features ], axis=1) print('predicting using model') prediction = model_assets['classifier'].predict_proba(features) probability = prediction[0][1].tolist() output = { 'probability': probability } return output # Output function to prepare the output\ndef output_fn(prediction, response_content_type): assert ( response_content_type == \"application\/json\" ), \"accept must be 'application\/json'\" response_body_str = json.dumps(prediction) return response_body_str<\/code><\/pre>\n<\/p><\/div>\n.deploy()<\/code> method on the estimator and define the number and type of instances we want to attach to the endpoint, and SageMaker manages the infrastructure on your behalf. When calling the endpoint from the notebook, we use a SageMaker SDK predictor<\/a>. The predictor sends data to an endpoint (as part of a request), and interprets the response. See the following code:<\/p>\n# Deploy the predictor\npredictor = estimator.deploy( endpoint_name=endpoint_name, instance_type='ml.p3.2xlarge', initial_instance_count=1\n) predictor.serializer = JSONSerializer()\npredictor.deserializer = JSONDeserializer()<\/code><\/pre>\n<\/p><\/div>\ndata = { \"CustServ Calls\": 10.0, \"Account Length\": 66, \"plan\": \"B\", \"limit\": \"limited\", 'text': \"Well, I've been dealing with TelCom for three months now and I am quite happy with your service\"} response = predictor.predict(data=data) print(\"{:.2%} probability of churn\".format(response['probability']))<\/code><\/pre>\n<\/p><\/div>\nClean up<\/h2>\n
predictor.delete_endpoint()<\/code><\/pre>\n<\/p><\/div>\nExtensions<\/h2>\n
![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker-4.png\"){kind=spacer}
<\/a><\/p>\nConclusions<\/h2>\n
\nAbout the Author<\/h3>\n
![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker.jpg\"){kind=spacer}
<\/a>Nick Minaie<\/strong> is an Sr AI\/ML Specialist Solutions Architect with AWS, helping customers on their journey to well-architected machine learning solutions at scale. In his spare time, Nick enjoys family time, abstract painting, and exploring nature.<\/p>\n![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker-5.png\"){kind=spacer}
<\/a> Ehsan M. Kermani<\/strong> is a Machine Learning Engineer in the AWS ML Automation Services group. He helps customers through their MLOps journey by providing his expertise in Software Engineering best practices to solve customers\u2019 end-to-end Machine Learning tasks from infrastructure to deployment.<\/p>\n
![](\"https:\/\/wordpress-1016567-4521551.cloudwaysapps.com\/wp-content\/uploads\/2021\/08\/analyze-customer-churn-probability-using-call-transcription-and-customer-profiles-with-amazon-sagemaker-1.jpg\"){kind=spacer}
<\/a>Dr. Li Zhang<\/strong> is a Principal Product Manager-Technical for Amazon SageMaker JumpStart and Amazon SageMaker built-in algorithms, a service that helps data scientists and machine learning practitioners get started with training and deploying their models, and uses reinforcement learning with Amazon SageMaker. His past work as a principal research staff member and master inventor at IBM Research has won the test of time paper award at IEEE INFOCOM.<\/p>\n