I spent all day trying to get local embeddings working on fly.io.

I spent all day trying to get local embeddings working on fly.io.

The problem went like this: - too many embedding calls to openai - knee-jerk reaction: "lets embed locally" - this worked super well on MacOS, cant stress enough how pleasant that fact is - but once I deployed - everything broke

Here's the architecture: - fly.io server, 2 machines - lancedb, in-process db - but fly.io volumes aren't shared, so what to do?

Simple, I'm a creative, remember?

In github actions, we nightly rebuild the embeddings db on all machines.

This is in parallel and took under 2min in the initial architecture

But then, it all broke down when creating the embeddings on fly.io machines.

Why? Embeddings are hungry for memory. They chewed through the available memory immediately.

So the solution is to either compute embeddings on a memory-rich machine, and copy them into your servers, or compute them remotely full-stop.