GitHub - dgarnitz/vectorflow: VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.

uniflow provides a unified LLM interface to extract and transform and raw documents.

• Document types: Uniflow enables data extraction from PDFs, HTMLs and TXTs.


• LLM agnostic: Uniflow supports most common-used LLMs for text tranformation, including
  
  
  • OpenAI models (GPT3.5 and GPT4),
  
  
  • Google Gemini models (Gemini 1.5, MultiModal),
  
  
  • AWS BedRock models,
  
  
  • Huggingf

... See more

The ideal solution for AI-native vectorDB would be something that would would be easy to set up and should integrate with existing APIs for rapid prototyping but should be able to scale without additional changes.

LanceDB is designed with this approach. Being server-less, it requires no setup — just import and start using. Persisted in HDD, allowing... See more