Summary Combining Text-to-SQL with Semantic Search for Retrieval Augmented Generation | by Jerry Liu | LlamaIndex Blog | May, 2023 | Medium medium.com
2,862 words - html page - View html page
One Line
The SQLAutoVectorQueryEngine combines SQL and vector stores to handle complex natural language queries over structured and unstructured data, providing a versatile tool with improved performance and potential for new capabilities.
Slides
Slide Presentation (12 slides)
Key Points
- The article discusses a query engine called SQLAutoVectorQueryEngine that combines Text-to-SQL with Semantic Search to handle complex natural language queries over a combination of structured and unstructured data.
- The engine leverages the expressivity of SQL over structured data and joins it with unstructured context from a vector database.
- Text-to-SQL queries are well-suited for analytics use cases, while Semantic Search is suited for queries where the answer can be obtained from unstructured text data.
- By combining these two systems, it is possible to leverage both structured tables and vector databases to provide the best answer to a query.
- The SQLAutoVectorQueryEngine is able to query, join, sequence, and combine structured data from a SQL database and unstructured data from a vector database to synthesize the final answer.
Summaries
49 word summary
The SQLAutoVectorQueryEngine combines SQL and vector stores to handle complex natural language queries over structured and unstructured data. It allows the combination of structured analytics and semantic search, addressing limitations of existing tools. The engine performs well across various queries and has potential for new retrieval and query capabilities.
187 word summary
The author introduces SQLAutoVectorQueryEngine, a powerful query engine that combines a SQL database and a vector store to handle complex natural language queries over structured and unstructured data. Data lakes in enterprises contain both types of data, and Large Language Models (LLMs) can extract insights from them. Existing tools for handling structured and unstructured data are Text-to-SQL and Semantic Search with a Vector Database. However, SQL is more suitable for analytics use cases, while vector stores are less suited for queries involving aggregations or joins. The SQLAutoVectorQueryEngine addresses this by allowing the combination of structured analytics and semantic search. It has an auto retriever that infers query parameters and executes queries against the vector database, and a selector prompt to choose between the SQL database and the vector database. The engine simulates a join between the two databases to provide accurate responses. Experiments show that the engine works well across various queries. The author is excited about the potential of combining LLMs with structured and unstructured data for new retrieval and query capabilities. The article also includes information about the author's role and recommendations for related articles.
377 word summary
In this article, the author introduces SQLAutoVectorQueryEngine, a powerful query engine that can handle complex natural language queries over structured and unstructured data. The engine combines a SQL database and a vector store to fulfill the queries.
Data lakes in enterprises typically contain both structured and unstructured data. Structured data is stored in SQL databases, while unstructured data includes text documents, audio recordings, videos, etc. Large Language Models (LLMs) can extract insights from both types of data.
There are existing tools for handling structured data (Text-to-SQL) and unstructured data (Semantic Search with a Vector Database). Text-to-SQL converts natural language into SQL statements, while Semantic Search retrieves relevant documents based on embedding similarity and uses LLMs to synthesize a response.
SQL is highly expressive for operating over tabular data, making it suitable for analytics use cases. On the other hand, existing vector stores are less suited for queries involving aggregations or joins.
The author introduces the SQLAutoVectorQueryEngine to address the need for combining knowledge from both structured tables and vector databases/document stores. This query engine allows for the combination of structured analytics and semantic search.
The auto retriever in the SQLAutoVectorQueryEngine infers query parameters and executes a query against the vector database. A selector prompt determines whether to query the SQL database or the vector database. The engine simulates a join between the two databases to synthesize the final answer.
Experiments show that the SQLAutoVectorQueryEngine works well across a broad range of queries. Example queries about arts and culture in Tokyo and the history of Berlin yield accurate responses.
The author concludes by expressing excitement about combining LLMs with structured and unstructured data to unlock new retrieval and query capabilities. Readers are invited to try out the SQLAutoVectorQueryEngine and provide feedback.
The excerpt also includes information about the author's role as an editor for LlamaIndex Blog and their involvement in building the data framework for LLMs. It mentions LlamaIndex raising $8.5M in a seed round led by Greylock Partners and provides links to other articles written by the author on LlamaIndex Blog.
Overall, the article discusses the SQLAutoVectorQueryEngine, the importance of combining structured and unstructured data, and the potential of LLMs in query capabilities. It also provides information about the author's role and includes recommendations for related articles.
831 word summary
In this article, the author introduces a powerful query engine called SQLAutoVectorQueryEngine, which can handle complex natural language queries over a combination of structured and unstructured data. This engine leverages both a SQL database and a vector store to fulfill the queries.
The author explains that data lakes in enterprises typically contain both structured and unstructured data. Structured data is stored in SQL databases, while unstructured data lacks a predefined structure and includes text documents, audio recordings, videos, etc. Large Language Models (LLMs) have the ability to extract insights from both types of data.
The author mentions two existing tooling and stacks for handling these types of data: Text-to-SQL for structured data and Semantic Search with a Vector Database for unstructured data. Text-to-SQL converts natural language into SQL statements that can be executed against the database, while Semantic Search retrieves relevant documents based on embedding similarity and synthesizes a response using LLMs.
In the structured setting, SQL is highly expressive for operating over tabular data, allowing for aggregations, joins, sorting, etc. Text-to-SQL queries are well-suited for analytics use cases where the answer can be found by executing a SQL statement.
In the unstructured setting, retrieval-augmented generation systems first perform retrieval by looking up the most relevant documents to the query based on embedding similarity. Existing vector stores do not offer a SQL-like interface and are less suited for queries involving aggregations or joins.
For some queries, it may be necessary to combine knowledge from both structured tables and vector databases/document stores to provide the best answer. The author gives an example use case where information from a structured table containing statistics for each city is combined with a collection of articles about different cities stored in a vector database.
To address this need, the author introduces the SQLAutoVectorQueryEngine, which can query, join, sequence, and combine structured and unstructured data to synthesize the final answer. This query engine allows for the combination of structured analytics and semantic search.
Overall, the author highlights the importance of leveraging both structured and unstructured data for more comprehensive and accurate query results, and introduces a query engine that facilitates this combination.
The auto retriever in the SQLAutoVectorQueryEngine infers query parameters and executes a query against the vector database. A selector prompt determines whether to query the SQL database or the vector database. If querying the SQL database, a query transformation is run to create a more detailed question based on the SQL query results. The new query is then run through the vector store query engine for retrieval and response synthesis. The original question, SQL query, SQL response, vector store query, and vector store response are combined to synthesize the final answer. This approach simulates a join between the SQL database and vector database. Experiments show that this method works well across a broad range of queries. The experiment setup includes a SQL table called city-stats and a Pinecone index for storing Wikipedia articles about cities. The SQLAutoVectorQueryEngine is used to process example queries and provide responses. The first example query asks about the arts and culture of the city with the highest population, resulting in a response about Tokyo. The second example query asks about the history of Berlin, resulting in a response detailing Berlin's history.
The excerpt is a query made to a query engine, asking for the country corresponding to each city. The final response states that Toronto is in Canada, Tokyo is in Japan, and Berlin is in Germany. The query can be answered by querying the SQL database without needing additional information from the vector database. The query transform step correctly identifies that there is no follow-up question, indicating that the original question has been answered.
The author concludes by stating that the stacks around language model models (LLMs) combined with unstructured data and structured data have been largely separate. However, they are excited about the potential of combining LLMs with both types of data to unlock new retrieval and query capabilities. The author invites readers to try out the SQLAutoVectorQueryEngine and provide feedback.
The full notebook walkthrough for this topic can be found in a guide associated with the article. The article also mentions the author's role as an editor for LlamaIndex Blog and their involvement in building the data framework for LLMs. It mentions LlamaIndex raising $8.5M in a seed round led by Greylock Partners and provides links to other articles written by the author on LlamaIndex Blog.
The excerpt also includes recommendations for other articles on Medium, such as training your own language model using privateGPT, getting started with LangChain for building LLM-powered applications, and project ideas using large language models for portfolios. There are also recommendations for articles on autonomous agents with LLMs, PandasAI, and more.
Overall, the excerpt discusses a query made to a query engine, the response received, and the potential of combining LLMs with structured and unstructured data. It also provides information about the author's role and includes recommendations for related articles on Medium.