Integrate with BentoML

Build an ML model into a Docker image with BentoML and serve it as a custom service in DeepFellow.

BentoML streamlines machine learning (ML) deployment by containerizing it, allowing for flexible API specifications. Integrating it with DeepFellow is relatively straightforward due to BentoML's ability to create a Docker image. This enables fully customized ML serving, API design, and data privacy.

To integrate BentoML with DeepFellow, first build a Docker image with BentoML, then install that image as a model in the DeepFellow Custom Service. If you already have an image, skip to Install on DeepFellow Infra.

Integration Guide

The following setup guide is based on the one from BentoML: https://docs.bentoml.com/en/latest/get-started/hello-world.html

1. Create a BentoML Service

Create a file services/bentoml-example/service.py to define a BentoML service:

from __future__ import annotations
import bentoml

with bentoml.importing():
    from transformers import pipeline


EXAMPLE_INPUT = "Breaking News: In an astonishing turn of events, the small \
town of Willow Creek has been taken by storm as local resident Jerry Thompson's cat, \
Whiskers, performed what witnesses are calling a 'miraculous and gravity-defying leap.' \
Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped \
a record-breaking 20 feet into the air to catch a fly. The event, which took \
place in Thompson's backyard, is now being investigated by scientists for potential \
breaches in the laws of physics. Local authorities are considering a town festival \
to celebrate what is being hailed as 'The Leap of the Century."


my_image = bentoml.images.Image(python_version="3.11") \
        .python_packages("torch", "transformers")

@bentoml.service(
    image=my_image,
    resources={"cpu": "2"},
    traffic={"timeout": 30},
    envs=[
        {"name": "HF_TOKEN"},  # You can omit value and set it when deploying the Service
        {"name": "DB_HOST", "value": "localhost"}
    ]
)
class Summarization:
    # Define the Hugging Face model as a class variable
    model_path = bentoml.models.HuggingFaceModel("sshleifer/distilbart-cnn-12-6")

    def __init__(self) -> None:
        # Load model into pipeline
        self.pipeline = pipeline('summarization', model=self.model_path)

    @bentoml.api
    def summarize(self, text: str = EXAMPLE_INPUT) -> str:
        result = self.pipeline(text)
        return f"Hello world! Here's your summary: {result[0]['summary_text']}"

The @bentoml.api decorator on summarize exposes the model at the /summarize HTTP path. BentoML serves the container on port 3000 by default.

2. Build the Image

Install the required commands and build the image:

pip install pipx
pipx install bentoml[all]
bentoml build
bentoml containerize summarization:7dsnrfcvtk4b3hdl

The bentoml containerize command produces a Docker image tagged summarization:7dsnrfcvtk4b3hdl. Use that tag in the next step.

3. Install on DeepFellow Infra

The DeepFellow Custom Service runs any Docker image as a containerized model and exposes it at the /custom/{prefix}/{path} endpoint. Install the BentoML image through the DeepFellow Infra Web Panel:

Open DeepFellow Infra Web Panel and go to Services → custom → Install.

Click custom → Add custom model.

Fill in the form: Model ID bentoml-example, Model endpoint /summarize, Docker image summarization:7dsnrfcvtk4b3hdl, Docker image port 3000.

Click Add, then Install.

Enter bentoml-example as the endpoint prefix when prompted.

Wait for the model to appear in the list with a green Installed label.

DeepFellow Infra assigns each container an internal port automatically. You do not set the exposed port yourself, so you reach the model through the /custom/{prefix} endpoint rather than the container port.

For a full walkthrough of adding custom models, including using images from Docker Hub or GitHub Container Registry, see Custom Service.

4. Call the Service

Once installed, the summarization API is reachable through the DeepFellow Infra /custom/{prefix}/ endpoint. DeepFellow forwards the /summarize path to the container:

curl -X POST 'http://DEEPFELLOW_INFRA_HOST/custom/bentoml-example/summarize' \
  -H 'Authorization: Bearer DEEPFELLOW_INFRA_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Breaking News: In an astonishing turn of events, the small town of Willow Creek has been taken by storm as local resident Jerry Thompson'\''s cat, Whiskers, performed what witnesses are calling a '\''miraculous and gravity-defying leap.'\'' Eyewitnesses report that Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly. The event, which took place in Thompson'\''s backyard, is now being investigated by scientists for potential breaches in the laws of physics. Local authorities are considering a town festival to celebrate what is being hailed as '\''The Leap of the Century."
  }'

Expected response:

Hello world! Here's your summary: Whiskers, an otherwise unremarkable tabby cat, jumped a record-breaking 20 feet into the air to catch a fly . The event is now being investigated by scientists for potential breaches in the laws of physics . Local authorities considering a town festival to celebrate what is being hailed as 'The Leap of the Century'

To expose the model to an LLM as a tool, see Using Tools.

Integration Guide

1. Create a BentoML Service

2. Build the Image

3. Install on DeepFellow Infra

4. Call the Service

On this page