metricspace/EntityAnonymization-3B-V0.9 · Hugging Face

Cross-Encoder for Hallucination Detection

This model was trained using SentenceTransformers Cross-Encoder class.

The model outputs a probabilitity from 0 to 1, 0 being a hallucination and 1 being factually consistent.

The predictions can be thresholded at 0.5 to predict whether a document is consistent with its source.

Training Data

This model is base... See more

RedPajama-V2 is an open dataset for training large language models. The dataset includes over 100B text

documents coming from 84 CommonCrawl snapshots and processed using

the CCNet pipeline. Out of these, there are 30B documents in the corpus

that additionally come with quality signals. In addition, we also provide the ids of duplicated documents whic... See more

#################################

# Definition of used LLM

#################################

##########################################################################

def graphPrompt(input: str, metadata={}, model="mixtral:latest"):

if model == None:

model = "mixtral:latest"

chunk_id = metadata.get('chunk_id', None)

# model_info = client.show(model_name=m... See more