Scaling of Microservices

Phonexia Microservices expose configuration parameters that allow you to tune their performance characteristics to match your deployment requirements. The two primary scaling parameters are --num_threads_per_instance and --num_instances_per_device. Understanding how each one affects the service will help you make the right trade-offs between latency and throughput.

Latency vs. throughput

Latency is the time it takes to process a single request from start to finish. Throughput is the number of requests the service can process per unit of time. In most deployments, improving one comes at the cost of the other, so it is important to know which metric matters most for your use case.

--num_threads_per_instance

This parameter controls how many CPU threads are used to process each individual request. A higher value allows the service to parallelize the work within a single request, which reduces the time it takes to get a response — in other words, it lowers latency.

This parameter applies to CPU processing only. When set to 0, the number of threads is detected automatically based on available CPU cores.

--num_threads_per_instance 4

Equivalent environment variable: PHX_NUM_THREADS_PER_INSTANCE

note

Increasing --num_threads_per_instance beyond the number of physical CPU cores available to the container will not yield further latency improvements and may actually hurt performance due to thread scheduling overhead.

--num_instances_per_device

This parameter controls how many model instances run in parallel on a single device (CPU or GPU). When the value is greater than 1, the service can handle multiple concurrent requests at the same time, which increases throughput.

--num_instances_per_device 4

Equivalent environment variable: PHX_NUM_INSTANCES_PER_DEVICE

Each instance consumes its own share of memory and compute resources. The optimal value depends on the hardware available and the expected concurrency of incoming requests.

note

Increasing --num_instances_per_device does not reduce latency for individual requests. It allows more requests to be served simultaneously, so it is most beneficial when your workload involves many concurrent clients.

GPU deployment

When running the GPU variant of a microservice, the compute-intensive parts of the model inference are offloaded to the GPU. In this case, --num_threads_per_instance has no effect because the per-request CPU thread count is not relevant when the GPU is handling the heavy computation.

To scale throughput on GPU, use --num_instances_per_device to run multiple model instances on the same GPU unit:

--num_instances_per_device 2

The number of instances that can share a single GPU depends on the GPU memory capacity and the memory footprint of the model. Start with a lower value and increase it while monitoring GPU memory utilization.

Choosing the right configuration

Goal	Parameter to tune	Applies to
Lower response time per request	--num_threads_per_instance	CPU only
Handle more concurrent requests	--num_instances_per_device	CPU and GPU

A typical starting point for a CPU deployment with latency-sensitive workloads is to set --num_threads_per_instance to the number of physical cores divided by the expected number of concurrent requests, and --num_instances_per_device to match that concurrency.

For a GPU deployment, focus exclusively on --num_instances_per_device and increase the value until GPU memory is close to full or throughput stops improving.

Example: Voiceprint Extraction on a 12-core CPU

The following table illustrates how different combinations of --num_threads_per_instance and --num_instances_per_device affect latency and throughput (faster than real-time) when running the voiceprint-extraction microservice on a 12-core CPU. All configurations use all 12 cores in total (threads × instances = 12). Note that the first row does not follow the 12-core constraint — it uses a single thread and a single instance to establish a baseline.

The dataset consisted of 8 kHz, single-channel audio recordings ranging from 1.5 to 10 minutes in length.

--num_threads_per_instance	--num_instances_per_device	Latency [s/file]	Throughput (FTRT)
1	1	2.185	60.8
1	12	3.378	392.1
2	6	1.663	402.4
3	4	1.423	343.9
4	3	1.130	296.1
6	2	1.226	215.8
12	1	1.445	77.3

note

FTRT (Faster then Real Time) Metric relates to the ratio of how much of audio in seconds can be processed compared to the processing time of a technology (e.g. if FTRT is equal to 60, 1[s] of processing time can process 60[s] of audio).

In this test, the lowest latency is achieved with --num_threads_per_instance 4, though the improvement over adjacent configurations is marginal. By contrast, using --num_instances_per_device 6 yields a larger gain in throughput.

Saturating --num_threads_per_instance does not necessarily improve latency. Beyond a certain point, it actually degrades performance due to thread scheduling overhead.

The same applies when the total thread count exceeds the number of available CPU cores — it only adds scheduling overhead without any performance benefit.

warning

These measurements are provided as a reference only — your results will differ. Actual performance depends on the model, audio content, recording length, and hardware. Always benchmark your specific setup before making deployment decisions.

Example: Voiceprint Extraction on 12-core CPU with auxiliary GPU

The following table illustrates how scaling --num_instances_per_device affects latency and throughput when running the voiceprint-extraction microservice on hardware with GPU acceleration. --num_threads_per_instance is not applicable in this case, as the workload is offloaded to the GPU. The measurements were taken on an NVIDIA RTX 4000 Ada GPU.

The dataset consisted of the same recordings as in the extraction on CPU.

--num_instances_per_device	Latency [s/file]	Throughput (FTRT)
1	0.175	671.1
2	0.223	1173.8
6	0.381	1961.5
12	0.862	1830.3

note

Regardless of the CPU used, there is a limit imposed by the hardware itself (bus, memory, storage). Saturating the device with too many instances can therefore cause throughput to plateau or even decline.