Open Source in Finance Forum New York 2024

Large financial institutions typically deploy their IT services across a number of private and public cloud providers. This asks for a platform-agnostic design of software applications to minimize the amount of rework required to migrate services between platforms. This applies to AI/ML models deployed for inference as much as any other software. In this presentation, we present an innovative framework ("inference-server") to streamline the deployment of an AI/ML model to the cloud. The framework is published by J.P. Morgan Chase at https://github.com/jpmorganchase/inference-server The framework abstracts the cloud infrastructure and connectivity aspects through a simple plugin system. This requires an ML software engineer to define an ML model as 2 functions only: 1) to load the model in memory and 2) to invoke that loaded model for a given set of input features. These 2 functions are then added to a Docker container image along the inference-server framework. This approach leads to a very modular design where an AI/ML model can be trivially tested in a development environment. If required the model could be deployed and accessed from any cloud computing platform.