Weaviate Vector Store - Hybrid Search

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

Creating a Weaviate Client

import weaviate

resource_owner_config = weaviate.AuthClientPassword(
  username = "<username>", 
  password = "<password>", 
)

# Connect to cloud instance
# client = weaviate.Client("https://<cluster-id>.semi.network/", auth_client_secret=resource_owner_config)

# Connect to local instance
client = weaviate.Client("http://localhost:8080")

Load documents

from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores import WeaviateVectorStore
from llama_index.response.notebook_utils import display_response

# load documents
documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()

Build the GPTVectorStoreIndex with WeaviateVectorStore

from llama_index.storage.storage_context import StorageContext


vector_store = WeaviateVectorStore(weaviate_client=client)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = GPTVectorStoreIndex.from_documents(documents, storage_context=storage_context)

# NOTE: you may also choose to define a class_prefix manually.
# class_prefix = "test_prefix"
# vector_store = WeaviateVectorStore(weaviate_client=client, class_prefix=class_prefix)

Query Index with Default Vector Search

# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine(
    similarity_top_k=2
)
response = query_engine.query("What did the author do growing up?")

display_response(response)

Query Index with Hybrid Search

Use hybrid search with bm25 and vector.
alpha parameter determines weighting (alpha = 0 -> bm25, alpha=1 -> vector search).

By default, alpha=0.75 is used (very similar to vector search)

# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine(
    vector_store_query_mode="hybrid", 
    similarity_top_k=2
)
response = query_engine.query(
    "What did the author do growing up?", 
)

display_response(response)

Set alpha=0. to favor bm25

# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine(
    vector_store_query_mode="hybrid", 
    similarity_top_k=2, 
    alpha=0.
)
response = query_engine.query(
    "What did the author do growing up?", 
)

display_response(response)