Hey all thank you for any help you can

RRichard G

Hey all - thank you for any help you can spare. I'm familiarising with LlamaIndex, and am using VertexAI (LangChain) and getting some good results. But I can't work out however how to expose the actual prompts that are being presented to the LLM. Pretty sure this should be straightforward... I want to check that SQLTableShema objects I've built are being included in context correctly.

19 comments

LLogan M

you'll see some parts of it (specifically the schema) if you turn on info logs

Plain Text

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

RRichard G

Thanks Logan - I have logging set to debug level atm which is slight overload but I'm not able to confirm that context_str (passed as a param to SQLTableSchema) is having any effect at all.

LLogan M

You can see the exact LLM inputs/outputs using the llama_debug handler, by looking at LLM events https://gpt-index.readthedocs.io/en/latest/examples/callbacks/LlamaDebugHandler.html#llama-debug-handler

But actually, if you share some of the setup code you have, I can double check the part of the code you are using to confirm your doubts 🙂

RRichard G

Thank you Logan, here goes...

RRichard G

Plain Text

# LLM model
llm = VertexAI(
    model_name="text-bison@001",
    max_output_tokens=256,
    temperature=0.0,
    top_p=0.8,
    top_k=40,
    verbose=True,
)

embed_model = LangchainEmbedding(VertexAIEmbeddings())
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
set_global_service_context(service_context)

table_schema = {
    'users': {
        'description': 'list of users with email addresses in column eml_adr'
    },
    'programmes': {
        'description': 'list of programmes users can be enrolled on. Programme names in column progrm_n',
    },
}

tables_to_include = list(table_schema.keys())

sql_database = SQLDatabase(engine, include_tables=tables_to_include)

# Build an index of relevant tables
table_node_mapping = SQLTableNodeMapping(sql_database)

# Build a list containing SQLTableSchema obj for each table in table_schema
table_schema_objs = [
    (SQLTableSchema(table_name=k, context_str=v['description'])) for k, v in table_schema.items()
]

obj_index = ObjectIndex.from_objects(
    table_schema_objs,
    table_node_mapping,
    VectorStoreIndex,
)
table_retriever_query_engine = SQLTableRetrieverQueryEngine(
    sql_database,
    obj_index.as_retriever(similarity_top_k=1),
)

while True:
    response = table_retriever_query_engine.query(input("Enter query >>> "))

    print(str(response))

RRichard G

The issue I'm having is in some cases I the query results in some SQL statement that references attributes that don't exist, and I'm not convinced (/don't know how to check) that context_str is working its way into the prompt put to the LLM

LLogan M

Looking at the code, here's a quick thing to run to check what the schema looks like (although keep in mind its pretty common for the LLM to hallucinate columns, especially gpt-3.5)

Plain Text

from llama_index import QueryBundle

table_descrp_str = table_retriever_query_engine._get_table_context(QueryBundle(query_str))
print(table_descrp_str)

LLogan M

It's also being logged to info actually

Attachment

RRichard G

So print(table_descrp_str does list the table attributes and field types (and I've seen this in stdout also), but I dont see the context_str (e.g. 'list of users with email addresses in column eml_adr' for table users) as defined in table_schema_objs.

RRichard G

I was imagining that it would be straightforward to expose the actual prompt presented as input to the model somehow.

LLogan M

Like I mentioned earlier, the callback is the most straightforward and programatic 🙂

LLogan M

Looking at the code again closer, maybe the context str is not injected properly, it looks like it only grabs the schema, while ignoring context_str

LLogan M

likely a bug, but would be an easy PR to fix 🙂

LLogan M

Attachment