Continuous batching

Continuous batching dynamically adds and removes inference requests as sequences finish, improving accelerator utilization compared with waiting for an entire fixed batch.

Also known as Batching.