LLM agents rely on their Long-Term Memory (LTM) to store and retrieve crucial memories. Conversations, thoughts, plans, actions, observations, skills, and behaviors are all stored within a vector database. The LTM enables pre-processing and post-processing of memories, ensuring optimal retrieval. By considering factors such as context, recency, imp... See more

In general, I see LLMs to be used in two broad categories: data processing, which is more of a worker use-cases, where the latency isn't the biggest issue but rather quality, and in user-interactions, where latency is a big factor. I think for the faster case a faster fallback is necessary. Or you escalate upwards, you first rely on a smaller more ... See more