Query Response Evaluator

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

from llama_index import (
    GPTTreeIndex, 
    GPTVectorStoreIndex, 
    SimpleDirectoryReader, 
    LLMPredictor, 
    ServiceContext,
    Response
)
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI
from llama_index.evaluation import QueryResponseEvaluator
import pandas as pd
pd.set_option('display.max_colwidth', 0)

# gpt-3 (davinci)
llm_predictor_gpt3 = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-003"))
service_context_gpt3 = ServiceContext.from_defaults(llm_predictor=llm_predictor_gpt3)

# gpt-4
llm_predictor_gpt4 = LLMPredictor(llm=ChatOpenAI(temperature=0, model_name="gpt-4"))
service_context_gpt4 = ServiceContext.from_defaults(llm_predictor=llm_predictor_gpt4)

evaluator = QueryResponseEvaluator(service_context=service_context_gpt3)
evaluator_gpt4 = QueryResponseEvaluator(service_context=service_context_gpt4)

documents = SimpleDirectoryReader('../test_wiki/data').load_data()

# create tree index
tree_index = GPTTreeIndex.from_documents(documents)

# create vector index
vector_index = GPTVectorStoreIndex.from_documents(
    documents, 
    service_context=ServiceContext.from_defaults(chunk_size_limit=512)
)

# define jupyter display function
def display_eval_df(query: str, response: Response, eval_result: str) -> None:
    eval_df = pd.DataFrame(
        {
            "Query": query,
            "Response": str(response), 
            "Source": response.source_nodes[0].source_text[:1000] + "...",
            "Evaluation Result": eval_result
        },
        index=[0]
    )
    eval_df = eval_df.style.set_properties(
        **{
            'inline-size': '600px',
            'overflow-wrap': 'break-word',
        }, 
        subset=["Response", "Source"]
    )
    display(eval_df)

Evaluate Response

Evaluate response relative to source nodes as well as query.

query_str = "What battles took place in New York City in the American Revolution?"
query_engine = tree_index.as_query_engine()
response_tree = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate(query_str, response_tree)

display_eval_df(query_str, response_tree, eval_result)

	Query	Response	Source	Evaluation Result
0	What battles took place in New York City in the American Revolution?	The Battle of Long Island, the Battle of White Plains, the Battle of Harlem Heights, the Battle of Fort Washington, the Battle of Fort Lee, and the Battle of Yorktown all took place in New York City during the American Revolution. These battles took place in various locations throughout the city, including Battery Weed and Fort Tompkins, Great Kills Park, and Central Park.	in 2015 makes it the highest of any county in the United States and higher than the density of any individual American city.Manhattan is the cultural, administrative, and financial center of New York City and contains the headquarters of many major multinational corporations, the United Nations headquarters, Wall Street, and a number of important universities. The borough of Manhattan is often described as the financial and cultural center of the world.Most of the borough is situated on Manhattan Island, at the mouth of the Hudson River and the East River, and its southern tip, at the confluence of the two rivers, represents the birthplace of New York City itself. Several small islands also compose part of the borough of Manhattan, including Randalls and Wards Islands, and Roosevelt Island in the East River, and Governors Island and Liberty Island to the south in New York Harbor. Manhattan Island is loosely divided into the Lower, Midtown, and Uptown regions. Uptown Manhattan is divide...	NO

query_str = "What battles took place in New York City in the American Revolution?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate(query_str, response_vector)

display_eval_df(query_str, response_vector, eval_result)

	Query	Response	Source	Evaluation Result
0	What battles took place in New York City in the American Revolution?	The Battle of Long Island and the Great Fire of New York.	at labor. Slavery became integrally tied to New York's economy through the labor of slaves throughout the port, and the banking and shipping industries trading with the American South. During construction in Foley Square in the 1990s, the African Burying Ground was discovered; the cemetery included 10,000 to 20,000 of graves of colonial-era Africans, some enslaved and some free.The 1735 trial and acquittal in Manhattan of John Peter Zenger, who had been accused of seditious libel after criticizing colonial governor William Cosby, helped to establish the freedom of the press in North America. In 1754, Columbia University was founded under charter by King George II as King's College in Lower Manhattan. === American Revolution === The Stamp Act Congress met in New York in October 1765, as the Sons of Liberty organization emerged in the city and skirmished over the next ten years with British troops stationed there. The Battle of Long Island, the largest battle of the American Revolutio...	YES

query_str = "What are the airports in New York City?"
query_engine = tree_index.as_query_engine()
response_tree = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate(query_str, response_tree)

display_eval_df(query_str, response_tree, eval_result)

	Query	Response	Source	Evaluation Result
0	What are the airports in New York City?	The airports in New York City are John F. Kennedy International Airport (JFK), LaGuardia Airport (LGA), and Newark Liberty International Airport (EWR).	Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively. New York City also has an extensive web of freeways and parkways, which link the city's boroughs to each other and to North Jersey, Westchester County, Long Island, and southwestern Connecticut through various bridges and tunnels. Because these highways serve millions of outer borough and suburban residents who commute into Manhattan, it is quite common for motorists to be stranded for hours in traffic congestion that are a daily occurrence, particularly during rush hour. Congestion pricing in New York City will go into effect in 2022 at the earliest.New York City is also known for its rules regarding turning at red lights. Unlike the rest of the United States, New York State prohibits right or left turns on red in cities with a population greater than one million, to reduce traffic collisions and increase pedestrian safety. In New York City, there...	NO

query_str = "What are the airports in New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate(query_str, response_vector)

display_eval_df(query_str, response_vector, eval_result)

	Query	Response	Source	Evaluation Result
0	What are the airports in New York City?	The airports in New York City are Long Island MacArthur Airport, Trenton–Mercer Airport, Westchester County Airport, and Teterboro Airport.	announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport. === Ferries === The Staten Island Ferry is the world's busiest ferry route, carrying more than 23 million passengers from July 2015 through June 2016 on the 5.2-mile (8.4 km) route between Staten Island and Lower Manhattan and running 24 hours a day. Other ferry systems shuttle commuters between Manhattan and other locales within the city and the metropolitan area. NYC Ferry, a NYCEDC initiative with routes planned to travel to all five boroughs, was launched in 2017, with second graders choosing the names of the ferries. Meanwhile, Seastreak ferry announced construction of a 600-passenger high-speed luxury ferry in Septe...	NO

query_str = "Who is the mayor of New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate(query_str, response_vector)

display_eval_df(query_str, response_vector, eval_result)

	Query	Response	Source	Evaluation Result
0	Who is the mayor of New York City?	The mayor of New York City is Eric Adams.	by geographic population boundaries. Each term for the mayor and council members lasts four years and has a two consecutive-term limit, which is reset after a four-year break. The New York City Administrative Code, the New York City Rules, and the City Record are the code of local laws, compilation of regulations, and official journal, respectively.Each borough is coextensive with a judicial district of the state Unified Court System, of which the Criminal Court and the Civil Court are the local courts, while the New York Supreme Court conducts major trials and appeals. Manhattan hosts the First Department of the Supreme Court, Appellate Division while Brooklyn hosts the Second Department. There are also several extrajudicial administrative courts, which are executive agencies and not part of the state Unified Court System. Uniquely among major American cities, New York is divided between, and is host to the main branches of, two different U.S. district courts: the District Court for t...	YES

Evaluate Source Nodes

Evaluate the set of returned sources, and determine which sources actually contain the answer to a given query.

from typing import List

# define jupyter display function
def display_eval_sources(query: str, response: Response, eval_result: List[str]) -> None:
    
    sources = [s.node.get_text() for s in response.source_nodes]
    eval_df = pd.DataFrame(
        {
            "Source": sources,
            "Eval Result": eval_result, 
        },
    )
    eval_df.style.set_caption(query)
    eval_df = eval_df.style.set_properties(
        **{
            'inline-size': '600px',
            'overflow-wrap': 'break-word',
        }, 
        subset=["Source"]
    )
    

    display(eval_df)

# NOTE: you can set response_mode="no_text" to get just the sources
query_str = "What are the airports in New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str, similarity_top_k=3, response_mode="no_text")
eval_source_result = evaluator_gpt4.evaluate_source_nodes(query_str, response_vector)

display_eval_sources(query_str, response_vector, eval_source_result)

	Source	Eval Result
0	announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport. === Ferries === The Staten Island Ferry is the world's busiest ferry route, carrying more than 23 million passengers from July 2015 through June 2016 on the 5.2-mile (8.4 km) route between Staten Island and Lower Manhattan and running 24 hours a day. Other ferry systems shuttle commuters between Manhattan and other locales within the city and the metropolitan area. NYC Ferry, a NYCEDC initiative with routes planned to travel to all five boroughs, was launched in 2017, with second graders choosing the names of the ferries. Meanwhile, Seastreak ferry announced construction of a 600-passenger high-speed luxury ferry in September 2016, to shuttle riders between the Jersey Shore and Manhattan, anticipated to start service in 2017; this would be the largest vessel in its class. === Taxis, vehicles for hire, and trams === Other features of the city's transportation infrastructure encompass 13,587 yellow taxicabs; other vehicle for hire companies; and the Roosevelt Island Tramway, an aerial tramway that transports commuters between Roosevelt Island and Manhattan Island. === Streets and highways === Despite New York's heavy reliance on its vast public transit system, streets are a defining feature of the city. The Commissioners' Plan of 1811 greatly influenced the city's physical development. Several of the city's streets and avenues, including Broadway, Wall Street, Madison Avenue, and Seventh Avenue are also used as metonyms for national industries there: the theater, finance, advertising, and fashion organizations, respectively. New York City also has an extensive web of freeways and parkways, which link the city's boroughs to each other and to North Jersey,	YES
1	the Chicago "L", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro). Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project. ==== Buses ==== New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buses and 200,000 commuters daily, making it the busiest bus station in the world. === Air === New York's airspace is the busiest in the United States and one of the world's busiest air transportation corridors. The three busiest airports in the New York metropolitan area include John F. Kennedy International Airport, Newark Liberty International Airport, and LaGuardia Airport; 130.5 million travelers used these three airports in 2016. JFK and Newark Liberty were the busiest and fourth busiest U.S. gateways for international air passengers, respectively, in 2012; as of 2011, JFK was the busiest airport for international passengers in North America.Plans have advanced to expand passenger volume at a fourth airport, Stewart International Airport near Newburgh, New York, by the Port Authority of New York and New Jersey. Plans were announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport. === Ferries === The Staten Island Ferry is the world's busiest ferry route, carrying more than 23 million passengers from July 2015 through June 2016 on the 5.2-mile (8.4 km) route between Staten Island and Lower Manhattan and running 24 hours a day. Other ferry systems shuttle commuters between Manhattan and other locales within the city and the metropolitan area. NYC Ferry, a NYCEDC initiative with routes planned to travel	YES
2	and New Jersey Transit. The combined systems converge at Grand Central Terminal and Pennsylvania Station and contain more than 250 stations and 20 rail lines. In Queens, the elevated AirTrain people mover system connects 24 hours a day JFK International Airport to the New York City Subway and the Long Island Rail Road; a separate AirTrain system is planned alongside the Grand Central Parkway to connect LaGuardia Airport to these transit systems. For inter-city rail, New York City is served by Amtrak, whose busiest station by a significant margin is Pennsylvania Station on the West Side of Manhattan, from which Amtrak provides connections to Boston, Philadelphia, and Washington, D.C. along the Northeast Corridor, and long-distance train service to other North American cities.The Staten Island Railway rapid transit system solely serves Staten Island, operating 24 hours a day. The Port Authority Trans-Hudson (PATH train) links Midtown and Lower Manhattan to northeastern New Jersey, primarily Hoboken, Jersey City, and Newark. Like the New York City Subway, the PATH operates 24 hours a day; meaning three of the six rapid transit systems in the world which operate on 24-hour schedules are wholly or partly in New York (the others are a portion of the Chicago "L", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro). Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project. ==== Buses ==== New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buses and 200,000 commuters daily, making it the busiest bus station in the world. === Air === New York's airspace is the busiest in the United States and one of the world's busiest air transportation corridors. The three busiest airports in the New York metropolitan area include John F. Kennedy International Airport, Newark Liberty International Airport, and LaGuardia Airport;	YES

# NOTE: you can set response_mode="no_text" to get just the sources
query_str = "Who is the mayor of New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str, similarity_top_k=3, response_mode="no_text")
eval_source_result = evaluator_gpt4.evaluate_source_nodes(query_str, response_vector)

display_eval_sources(query_str, response_vector, eval_source_result)

	Source	Eval Result
0	by geographic population boundaries. Each term for the mayor and council members lasts four years and has a two consecutive-term limit, which is reset after a four-year break. The New York City Administrative Code, the New York City Rules, and the City Record are the code of local laws, compilation of regulations, and official journal, respectively.Each borough is coextensive with a judicial district of the state Unified Court System, of which the Criminal Court and the Civil Court are the local courts, while the New York Supreme Court conducts major trials and appeals. Manhattan hosts the First Department of the Supreme Court, Appellate Division while Brooklyn hosts the Second Department. There are also several extrajudicial administrative courts, which are executive agencies and not part of the state Unified Court System. Uniquely among major American cities, New York is divided between, and is host to the main branches of, two different U.S. district courts: the District Court for the Southern District of New York, whose main courthouse is on Foley Square near City Hall in Manhattan and whose jurisdiction includes Manhattan and the Bronx; and the District Court for the Eastern District of New York, whose main courthouse is in Brooklyn and whose jurisdiction includes Brooklyn, Queens, and Staten Island. The U.S. Court of Appeals for the Second Circuit and U.S. Court of International Trade are also based in New York, also on Foley Square in Manhattan. === Politics === The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022. The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all	NO
1	for the Second Circuit and U.S. Court of International Trade are also based in New York, also on Foley Square in Manhattan. === Politics === The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022. The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all five boroughs. Party platforms center on affordable housing, education, and economic development, and labor politics are of importance in the city. Thirteen out of 27 U.S. congressional districts in the state of New York include portions of New York City.New York is one of the most important sources of political fundraising in the United States. At least four of the top five ZIP Codes in the nation for political contributions were in Manhattan for the 2004, 2006, and 2008 elections. The top ZIP Code, 10021 on the Upper East Side, generated the most money for the 2004 presidential campaigns of George W. Bush and John Kerry. The city has a strong imbalance of payments with the national and state governments. It receives 83 cents in services for every $1 it sends to the federal government in taxes (or annually sends $11.4 billion more than it receives back). City residents and businesses also sent an additional $4.1 billion in the 2009–2010 fiscal year to the state of New York than the city received in return. == Transportation == New York City's comprehensive transportation system is both complex and extensive. === Rapid transit === Mass transit in New York City, most of which runs 24 hours a day, accounts for one in every three users of mass transit in	YES
2	ed.). New York: The New Press. ISBN 978-1-56584-321-9. Holli, Melvin G., and Jones, Peter d'A., eds. Biographical Dictionary of American Mayors, 1820-1980 (Greenwood Press, 1981) short scholarly biographies each of the city's mayors 1820 to 1980. online; see index at p. 410 for list.Jackson, Kenneth T., ed. (1995). The Encyclopedia of New York City. New Haven: Yale University Press. ISBN 0300055366. Jackson, Kenneth T.; Dunbar, David S., eds. (2005). Empire City: New York Through the Centuries. Columbia University Press. ISBN 978-0-231-10909-3. Lankevich, George L. (1998). American Metropolis: A History of New York City. NYU Press. ISBN 978-0-8147-5186-2. White, E.B. (1949). Here is New York (2000 reissue ed.). Little Bookroom. White, Norval & Willensky, Elliot (2000). AIA Guide to New York City (4th ed.). New York: Three Rivers Press. ISBN 978-0-8129-3107-5. Whitehead, Colson (2003). The Colossus of New York: A City in 13 Parts. New York: Doubleday. ISBN 978-0-385-50794-3. == External links == Official website NYC Go, official tourism website New York City at Curlie Geographic data related to New York City at OpenStreetMap Collections, 145,000 NYC photographs at the Museum of the City of New York "The New New York Skyline (interactive)". National Geographic. November 2015.	NO