6 Hard Problems Scaling Vector Search
rockset.com
6 Hard Problems Scaling Vector Search
Since data are stored on multiple replicas, we need to ensure they are synchronized across replicas.
Alex • System Design Interview - An insider's guide
Maintaining the speed of search gets harder as Enigma.io adds more data; by mid-2013, the company had 100,000 datasets (essentially, 100,000 large data tables) and more than 20 billion individual data points. On the other hand, figuring out how to incorporate new datasets gets easier over time, because there are only so many different kinds of data
... See moreJoel Gurin • Open Data Now: The Secret to Hot Startups, Smart Investing, Savvy Marketing, and Fast Innovation (Business Books)
latest research on text embeddings!
TLDR: Vector databases are NOT safe. Text embeddings can be inverted. We can do this exactly for sentence-length inputs and get very close with paragraphs...
TLDR: Vector databases are NOT safe. Text embeddings can be inverted. We can do this exactly for sentence-length inputs and get very close with paragraphs...
jack morris • Tweet
Memory Considerations
Since co-occurrence matrices are square, they grow exponential with the number of entities being embedded. For 50k entities and a 32-bit data format, a dense matrix will already be at 10GB. 100k entities puts it at 40GB.
If you are trying to embed even more entities than that or have limited RAM available, you may need to use a... See more
Since co-occurrence matrices are square, they grow exponential with the number of entities being embedded. For 50k entities and a 32-bit data format, a dense matrix will already be at 10GB. 100k entities puts it at 40GB.
If you are trying to embed even more entities than that or have limited RAM available, you may need to use a... See more