Thank you that's really helpful.
An additional challenge I'm facing that I'd be curious to know if you have any insight to is the following:
I am using BigQuery and so if I were to make an SQLAlchemy connection to a specific dataset as follows, a connection is made quickly with no problem:
python sqlalchemy_url = f"bigquery://{self.project}/{self.database}?credentials_path={self.service_account_file}"
But the langchain SQL agent only seems to look at the tables in that specific dataset, however, I have many datasets. Even specifically telling the LLM the name of the dataset (different dataset) I want it to query, it stays looking at only the dataset in the url connection.
I've also tried to leave the dataset out of the URL like this:
python sqlalchemy_url = f"bigquery://{self.project}?credentials_path={self.service_account_file}"
But just making the connection seems to take 12 minutes, I think because it is now fetching metadata of all tables in the whole server.
This has me so stumped dude xD Thanks for your help, no pressure to keep assisting though, you're a legend already!