[Milestone 그 이후] Hosting the Docker container on Heroku for TFserving

we built deep learning model classifies pills images, and then deploy the model using TFserving with Docker. ([Milestone 그 이후] Tensorflow Model Serving)

In this post, Our main focus will be on model deployment on Heroku using TFserving with Docker.

 

Why Docker ?

Because Docker containers encapsulate everything an application needs to run (and only those things), they allow applications to be shuttled easily between environments and One of the easiest ways to get started using TensorFlow Serving is with Docker.

 

what is TensorFlow serving?

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs.

So in this step we will use TensorFlow serving with docker.

 

1. Make a directory for our app that will contain a dockerfile and the SavedModel in that way

medisharp-tf/
├── Dockerfile
└── models/
    └── model/
        └── 1/
            ├── assets/
            ├── saved_model.pb
            └── variables/
                ├── variables.data-00000-of-00002
                ├── variables.data-00001-of-00002
                └── variables.index

2. Setting Dockerfile

Tensorflow의 serving에 대한 공식 Github repo에 docker를 이용한 TFserving의 Dockerfile을 제공해준다.

# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

ARG TF_SERVING_VERSION=latest
ARG TF_SERVING_BUILD_IMAGE=tensorflow/serving:${TF_SERVING_VERSION}-devel

FROM ${TF_SERVING_BUILD_IMAGE} as build_image #baseimage로 tensorflow/serving
FROM ubuntu:18.04

ARG TF_SERVING_VERSION_GIT_BRANCH=master
ARG TF_SERVING_VERSION_GIT_COMMIT=head

LABEL maintainer="gvasudevan@google.com"
LABEL tensorflow_serving_github_branchtag=${TF_SERVING_VERSION_GIT_BRANCH}
LABEL tensorflow_serving_github_commit=${TF_SERVING_VERSION_GIT_COMMIT}

RUN apt-get update && apt-get install -y --no-install-recommends \
        ca-certificates \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install TF Serving pkg
COPY --from=build_image /usr/local/bin/tensorflow_model_server /usr/bin/tensorflow_model_server

# Expose ports
# gRPC
EXPOSE 8500

# REST
EXPOSE 8501

# Set where models should be stored in the container
ENV MODEL_BASE_PATH=/models
RUN mkdir -p ${MODEL_BASE_PATH}

# The only required piece is the model name in order to differentiate endpoints
ENV MODEL_NAME=model

# Create a script that runs the model server so we can use environment variables
# while also passing in arguments from the docker command line
RUN echo '#!/bin/bash \n\n\
tensorflow_model_server --port=8500 --rest_api_port=8501 \
--model_name=${MODEL_NAME} --model_base_path=${MODEL_BASE_PATH}/${MODEL_NAME} \
"$@"' > /usr/bin/tf_serving_entrypoint.sh \
&& chmod +x /usr/bin/tf_serving_entrypoint.sh

ENTRYPOINT ["/usr/bin/tf_serving_entrypoint.sh"]

3. 도커 이미지 생성 후 도커 컨테이너 실행하기

$ docker build -t medisharp-tf .

$ docker run -p 8501:8501 -e PORT=8501 -t medisharp-tf

 

-p 8501:8501: 도커 엔진이 호스트 시스템의 TCP포트 8501번을 컨테이너의 TCP 8501번으로 포워딩한다. 기본적으로 TF서빙은 이 포트를 사용해 REST API를 제공한다. 

-e PORT=8501: 컨테이너 8501 포트만 개방한다. 포트만 개방하는게 무슨 의미가 있느냐고 생각할 수도 있지만 상황에 따라서 쓰일 때가 있으니 열어둔다. 예를 들면 포트를 개방만 하면 다른 컨테이너에서 접근할 수 있으니 컨테이너 끼리 연결할 때에 필요하다.

 

4. Serving the saved model locally using TensorFlow serving and Docker.

여기까지 하면 [Milestone 그 이후] Tensorflow Model Serving여기서 postman으로 했던 것 처럼 

SERVER_URL = 'http://localhost:8501/v1/models/model:predict'

response = requests.post(SERVER_URL, data=input_data_json)
response.raise_for_status() #에러 발생시 예외 발생시킴
response = response.json() 
print("response is: ", response)

이러한 SERVER_URL로 HTTP POST request로 이미지를 TF서빙에 전송하고 결과를 응답받을 수 있다. 

5. Hosting the docker container on Heroku.

이제는 로컬환경이 아니라 배포(운영) 환경에서 Docker을 이용해서 TFserving을 해보고자 한다. 

docker container을 Heroku에서 hosting하려면 heroku container를 사용한다. 

1) Login in to heroku container

$ heroku container:login

2) Create an app on heroku

$ heroku create medisharp-tf

3) Push and release the docker image to heroku

$ heroku container:push web -a medisharp-tf

$ heroku container:release web -a medisharp-tf

Releasing images web to medisharp-tf... done 라는 문구가 보이면서 성공

 

이제는 SERVER_URL을 'http://localhost:8501/v1/models/model:predict'가 아닌 'http://medisharp-tf.herokuapp.com/v1/models/model:predict'으로 해줄 수 있다.

SERVER_URL = 'http://medisharp-tf.herokuapp.com/v1/models/model:predict'
#output
response = requests.post(SERVER_URL, data=input_data_json)
response.raise_for_status() #에러 발생시 예외 발생시킴
response = response.json() #응답은 "predictions" 키 하나를 가진 딕셔너리로, 이 키에 해당하는 값은 예측의 리스트

이렇게 Heroku에서 Docker을 사용한 TFserving로 모델을 배포해보았다. 
이제는 서버 코드를 무겁게 하는 저장된 모델이 없어도 되며, 매번 모델을 load하지 않아도 된다.

 

AWS lambda를 이용해서 모델을 배포해보는 것도 있던데, 그것도 해보고싶다!

 

reference

TensorFlow Serving with Docker

Training and Deploying A Deep Learning Model in Keras MobileNet V2 and Heroku: A Step-by-Step…

From building to deploying deep learning models using TensorFlow , Docker and Heroku