AI-Server

A server that can serve AI models with an API and an authentication system

Usage

The ai-server project require a conda environement. You can use Docker to deploy it easily.

Docker

You can easily use docker-compose to run the project. Simply go into the project directory and run :

docker compose run -d

Environment Variables

The project use special environement variables :

Name	Description
MODEL_DIRECTORY	the directory where the models are stored

Volumes

The project might store data inside multiples volumes :

Path	Type	Required	Description
/models/	Volume	True	The place where the models are stored
/root/.huggingface/hub	Bind	False	The place where the internal models are cached. Avoid redownloading huge amount of data at every inference

Models

A model is an object that can be loaded and do inference.

It is stored inside a directory and must always contain a config.json file.

Configuration

This is a json-structured file with basic information about the model.

It describe :

• its type (see below)
• its tags
• its interface
• its inputs
• its output mimetype

And other properties depending on the model type.

Types

There is for now only a single type of Model : the Python model

Python Model

A python model is isolated in a conda environement.

To be considered a Python model, you need theses three files :

File	Description
config.json	The configuration file
env	The conda virtual environment
model.py	The model file

Configuration

Additionnal fields might be found in the configuration :

...

Virtual Environment

You can create a conda virtual environement it with :

conda create --prefix ./env/ python=<version>

You can install your requirements inside with pip3 or conda.

Internal Model

You need to create a model.py file containing a class named Model. This class must be exposed with the Pyro4 library for inter-operability with our main environement.

Here is an example of an internal model :

import Pyro4

import torch
import transformers


@Pyro4.expose
class Model:
    def __init__(self):
        # load the model
        self.pipe = transformers.pipeline(...)

    def infer(self, messages: list[dict]) -> bytes:
        with torch.no_grad():
            outputs = self.pipe(messages)

        return outputs[0]["generated_text"][-1]