Response Evaluator

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

from llama_index import (
    GPTTreeIndex, 
    GPTVectorStoreIndex, 
    SimpleDirectoryReader, 
    LLMPredictor, 
    ServiceContext,
    Response
)
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI
from llama_index.evaluation import ResponseEvaluator
import pandas as pd
pd.set_option('display.max_colwidth', 0)

# gpt-3 (davinci)
llm_predictor_gpt3 = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003"))
service_context_gpt3 = ServiceContext.from_defaults(llm_predictor=llm_predictor_gpt3)

# gpt-4
llm_predictor_gpt4 = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name="gpt-4"))
service_context_gpt4 = ServiceContext.from_defaults(llm_predictor=llm_predictor_gpt4)

evaluator = ResponseEvaluator(service_context=service_context_gpt3)
evaluator_gpt4 = ResponseEvaluator(service_context=service_context_gpt4)

documents = SimpleDirectoryReader('../test_wiki/data').load_data()

# create tree index
tree_index = GPTTreeIndex.from_documents(documents=documents)

# create vector index
vector_index = GPTVectorStoreIndex.from_documents(
    documents, 
    service_context=ServiceContext.from_defaults(chunk_size_limit=512)
)

INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 44108 tokens
> [build_index_from_nodes] Total embedding token usage: 44108 tokens

# define jupyter display function
def display_eval_df(response: Response, eval_result: str) -> None:
    eval_df = pd.DataFrame(
        {
            "Response": str(response), 
            "Source": response.source_nodes[0].source_text[:1000] + "...",
            "Evaluation Result": eval_result
        },
        index=[0]
    )
    eval_df = eval_df.style.set_properties(
        **{
            'inline-size': '600px',
            'overflow-wrap': 'break-word',
        }, 
        subset=["Response", "Source"]
    )
    display(eval_df)

query_engine = tree_index.as_query_engine()
response_tree = query_engine.query("What battles took place in New York City in the American Revolution?")
eval_result = evaluator_gpt4.evaluate(response_tree)

INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 4332 tokens
> [query] Total LLM token usage: 4332 tokens
INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens
> [query] Total embedding token usage: 0 tokens

display_eval_df(response_tree, eval_result)

	Response	Source	Evaluation Result
0	The Battle of Long Island, the Battle of White Plains, the Battle of Harlem Heights, the Battle of Fort Washington, the Battle of Fort Lee, and the Battle of Yorktown all took place in New York City during the American Revolution. These battles took place in various locations throughout the city, including Battery Weed and Fort Tompkins, Great Kills Park, and Central Park.	in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the borough of Manhattan, including Randalls and Wards Islands, and Roosevelt Island in the East River, and Governors Island and Liberty Island to the south in New York Harbor. Manhattan Island is loosely divided into the Lower, Midtown, and Uptown regions. Uptown Manhattan is divide...	NO

query_engine = vector_index.as_query_engine()
response_vector = query_engine.query("What battles took place in New York City in the American Revolution?")
eval_result = evaluator_gpt4.evaluate(response_vector)

display_eval_df(response_vector, eval_result)

	Response	Source	Evaluation Result
0	The Battle of Long Island and the Great Fire of New York.	at labor. Slavery became integrally tied to New York's economy through the labor of slaves throughout the port, and the banking and shipping industries trading with the American South. During construction in Foley Square in the 1990s, the African Burying Ground was discovered; the cemetery included 10,000 to 20,000 of graves of colonial-era Africans, some enslaved and some free.The 1735 trial and acquittal in Manhattan of John Peter Zenger, who had been accused of seditious libel after criticizing colonial governor William Cosby, helped to establish the freedom of the press in North America. In 1754, Columbia University was founded under charter by King George II as King's College in Lower Manhattan. === American Revolution === The Stamp Act Congress met in New York in October 1765, as the Sons of Liberty organization emerged in the city and skirmished over the next ten years with British troops stationed there. The Battle of Long Island, the largest battle of the American Revolutio...	YES

query_engine = tree_index.as_query_engine()
response_tree = query_engine.query("What are the airports in New York City?")
eval_result = evaluator_gpt4.evaluate(response_tree)

display_eval_df(response_tree, eval_result)

	Response	Source	Evaluation Result
0	The airports in New York City are John F. Kennedy International Airport (JFK), LaGuardia Airport (LGA), and Newark Liberty International Airport (EWR).	Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively. New York City also has an extensive web of freeways and parkways, which link the city's boroughs to each other and to North Jersey, Westchester County, Long Island, and southwestern Connecticut through various bridges and tunnels. Because these highways serve millions of outer borough and suburban residents who commute into Manhattan, it is quite common for motorists to be stranded for hours in traffic congestion that are a daily occurrence, particularly during rush hour. Congestion pricing in New York City will go into effect in 2022 at the earliest.New York City is also known for its rules regarding turning at red lights. Unlike the rest of the United States, New York State prohibits right or left turns on red in cities with a population greater than one million, to reduce traffic collisions and increase pedestrian safety. In New York City, there...	NO

query_engine = vector_index.as_query_engine()
response_vector = query_engine.query("What are the airports in New York City?")
eval_result = evaluator_gpt4.evaluate(response_vector)

display_eval_df(response_vector, eval_result)

	Response	Source	Evaluation Result
0	The airports in New York City are Long Island MacArthur Airport, Trenton–Mercer Airport, Westchester County Airport, and Teterboro Airport.	announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport. === Ferries === The Staten Island Ferry is the world's busiest ferry route, carrying more than 23 million passengers from July 2015 through June 2016 on the 5.2-mile (8.4 km) route between Staten Island and Lower Manhattan and running 24 hours a day. Other ferry systems shuttle commuters between Manhattan and other locales within the city and the metropolitan area. NYC Ferry, a NYCEDC initiative with routes planned to travel to all five boroughs, was launched in 2017, with second graders choosing the names of the ferries. Meanwhile, Seastreak ferry announced construction of a 600-passenger high-speed luxury ferry in Septe...	YES

query_engine = vector_index.as_query_engine()
response_vector = query_engine.query("Who is the mayor of New York City?")
eval_result = evaluator_gpt4.evaluate(response_vector)

display_eval_df(response_vector, eval_result)

	Response	Source	Evaluation Result
0	The mayor of New York City is Eric Adams.	by geographic population boundaries. Each term for the mayor and council members lasts four years and has a two consecutive-term limit, which is reset after a four-year break. The New York City Administrative Code, the New York City Rules, and the City Record are the code of local laws, compilation of regulations, and official journal, respectively.Each borough is coextensive with a judicial district of the state Unified Court System, of which the Criminal Court and the Civil Court are the local courts, while the New York Supreme Court conducts major trials and appeals. Manhattan hosts the First Department of the Supreme Court, Appellate Division while Brooklyn hosts the Second Department. There are also several extrajudicial administrative courts, which are executive agencies and not part of the state Unified Court System. Uniquely among major American cities, New York is divided between, and is host to the main branches of, two different U.S. district courts: the District Court for t...	YES