In this tutorial, we will cover a simple and fast way to deploy a PyTorch model at scale without a docker, and Kubernetes setup.

I've trained my model. What's next?

Have you ever wanted to deploy a Pytorch model in order to run it on a website or some mobile app?

In the past, I was exactly in your situation trying different tools, resources. Hundreds of hours spent on building a simple one-page demo app that takes an image as an input and returns one word: Cat. I've experienced that the main focus of most Machine Learning Deployment tutorials ends up with building a simple flask app, creating a docker, and setting up the cloud infrastructure. However, it is still time-consuming and difficult. There are many problems with ai model deployment.

Since that time I've managed to find a way to solve that problem in just three easy steps.

Let's have a look and deploy a PyTorch model (Check also How to deploy keras model).

Step 1: Develop a model

In the first step, we need to have a trained model. For this purpose, we will use a pre-trained PyTorch YoloV5.

For those who don’t know what Yolo is a real-time object detection framework and stands for You Only Look Once, which means that the image passes only once through the Fully Convolutional Neural Network. Comparing to previous versions the V5 is much faster and much smaller (only 27Mb).

deploy pytorch model
Input of the model (on the left) and output (on the right)

In order to run the algorithm locally, you can copy and paste a script from PyTorch Hub. The script will automatically download a pre-trained model and sample images to inference a model.

import cv2
import torch
from PIL import Image

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True).fuse().autoshape() # for PIL/cv2/np inputs and NMS

# Images
for f in ['zidane.jpg', 'bus.jpg']: # download 2 images
print(f'Downloading {f}...')
torch.hub.download_url_to_file('https://github.com/ultralytics/yolov5/releases/download/v1.0/' + f, f)
img1 = Image.open('zidane.jpg') # PIL image
img2 = cv2.imread('bus.jpg')[:, :, ::-1] # OpenCV image (BGR to RGB)
imgs = [img1, img2] # batched list of images

# Inference
results = model(imgs, size=640) # includes NMS

# Results
results.show() # .show() results, .save() jpgs, or .print() to scree

Step 2: Deploy a model

We have our basic script with the model, so now we can deploy it in the cloud.

For this purpose, we will use the Syndicai Platform which helps developers deliver AI in production effortlessly without setting up an infrastructure. No config needed, no deep tech knowledge required. Syndicai takes care of security, scalability, monitoring, and more. You just need to prepare a model and connect your git repository. The model will be deployed automatically.

deploy a pytorch model
AI Model Deployment: Traditional Approach vs Syndicai

Prepare a model

During the model preparation process you need to ensure the following files are in the git repository:

1st file: requirements.txt - file with all libraries and frameworks needed to recreate model's environment.

opencv-python==4.2.0.34
pillow
pyyaml
tqdm
torch
matplotlib
torchvision
scipy

2nd file: syndicai.py - main file with the PythonPredictor python class responsible for model prediction.

import torch
from PIL import Image
from helpers import draw_box, url_to_img, img_to_bytes


class PythonPredictor:

def __init__(self, config):
""" Download pretrained model. """
self.model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True).autoshape()

def predict(self, payload):
""" Run a model based on url input. """

# Inference
img = url_to_img(payload["url"])
results = self.model(img)

# Draw boxes
boxes = results.xyxy[0].numpy()
box_img = draw_box(img, boxes)

# Save image
#box_img.save("sample_data/output.png", "PNG")

return img_to_bytes(box_img)

Those files are necessary to correctly recreate your environment, build the infrastructure, and finally run the model in the form of a webservice. You can check with the Syndicai model repository.

Connect a repository

When the model is ready it's time for deployment. The only thing that you need to do at this stage is to log in to Syndicai Platform, click New Model on the Overview page, and follow the steps in the form. After the model is deployed the whole magic happens in order to give you the model API in the Integrate section.

syndicai platform add model
Deploy a Model by connecting the repository.

For more information about the model preparation or deployment process go to Syndicai Docs.

Step 3: Integrate

Great!

We have our scalable API so theoretically the job is done. You can run a model on the Platform or integrate the API with your app.

In order to make an inference the model go to Syndicai platform and paste a sample input script in the Run a model section

{
"url": "https://images.pexels.com/photos/2083866/pexels-photo-2083866.jpeg?auto=compress&cs=tinysrgb&h=750&w=1260"
}

Remember that your model needs to be Active in order to work!

run a pytorch model
Run a model on the platform

API of the model can be integrated with some React app or a website in order to build a Showcase.

syndicai showcase pytorch model
Integrate a model with Syndicai Showcase page.

Fork the repository with the Showcase and explore yolov5 implementation on your own.

Summary

The main goal of that tutorial was to easily deploy the PyTorch model in production. No matter what is your specialty, whether you are a Data Scientist, AI Researcher,or a Machine Learning Engineer. Now you are able to deliver an AI model without any configuration and infrastructure setup.

You can also explore How to deploy Keras model.

* * *

If you found that material helpful, have some comments, or want to share some ideas for the next one - don't hesitate to drop us a line via slack or mail. We would love to hear your feedback!

You might like these

Deploy ML with FastAPI, Docker, and Cloud Run

Tutorial
June 20, 2021
by
Marcin Laskowski

A quick guide on how to deploy depth estimator at scale.

Tutorial
June 20, 2021
by
Marcin Laskowski