HOSTING MACHINE LEARNING MODELS

Techniques for hosting machine learning models are described. In some instances, a method of receiving a request to perform an inference using a particular machine learning model; determining a group of hosts to route the request to, the group of hosts to host a plurality of machine learning models including the particular machine learning model; determining a path to the determined group of hosts; determining a particular host of the group of hosts to perform an analysis of the request based on the determined path, the particular host having the particular machine learning model in memory; routing the request to the particular host of the group of hosts; performing inference on the request using the particular host; and providing a result of the inference to a requester is performed. 描述了用于托管机器学习模型的技术。在一些情况下，执行了一种方法：接收使用特定机器学习模型执行推理的请求；确定要将所述请求路由到的主机组，所述主机组托管包括所述特定机器学习模型的多个机器学习模型；确定到所确定的主机组的路径；基于所确定的路径来确定所述主机组中的特定主机以执行对所述请求的分析，所述特定主机在存储器中具有所述特定机器学习模型；将所述请求路由到所述主机组的所述特定主机；使用所述特定主机对所述请求执行推理；以及向请求者提供所述推理的结果。