Rendered at 07:55:44 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
zoobab 13 hours ago [-]
Would it be able to swap models on demand with this?
fergus00 12 hours ago [-]
https://blog.doubleword.ai/fast-sglang-starts yeah, this is part of the idea. If you get cold starts down to seconds or subseconds, then you can run many models multiplexed on the same GPUs
zoobab 12 hours ago [-]
I have read a Linkedin post in french recently with a university lab (one big machine serving hundreds of users) mentioning that they could swap models on demand. Will try to find back the link.