Skip to main content

Pebblo Safe DataLoader for Langchain

This document describes how to augment your existing Langchain DocumentLoader with Pebblo Safe DataLoader to get deep data visibility on the types of Topics and Entities ingested into the Gen-AI Langchain application. For details on Pebblo Server see this pebblo server document.

Pebblo Safeloader enables safe data ingestion for Langchain document loader1. This is done by wrapping the document loader call with Pebblo Safe DataLoader.

How to Pebblo enable Document Loading?​

Assume a Langchain RAG application snippet using CSVLoader to read a CSV document for inference.

Here is the snippet of Lanchain RAG application using CSVLoader.

    from langchain_community.document_loaders import CSVLoader

    loader = CSVLoader(file_path)
    documents = loader.load()
    vectordb = Chroma.from_documents(documents, OpenAIEmbeddings())

The Pebblo SafeLoader can be enabled with few lines of code change to the above snippet.

    from langchain_community.document_loaders import CSVLoader
    from langchain_community.document_loaders.pebblo import PebbloSafeLoader

    loader = PebbloSafeLoader(
                CSVLoader(file_path),
                name="RAG app 1", # App name (Mandatory)
                owner="Joe Smith", # Owner (Optional)
                description="Support productivity RAG application", # Description (Optional)
    )
    documents = loader.load()
    vectordb = Chroma.from_documents(documents, OpenAIEmbeddings())

A data report with all the findings, both Topics and Entities, will be generated and available for inspection in the Pebblo Server. See this pebblo server for further details.

Note: By default Pebblo Server runs at localhost:8000. If your Pebblo Server is running at some other location for eg. a docker container etc, put the correct URL in PEBBLO_CLASSIFIER_URL env variable. ref: server-configurations

export PEBBLO_CLASSIFIER_URL="<pebblo-server-host:pebblo-server-port>"

Supported Document Loaders​

The following Langchain DocumentLoaders are currently supported.

  1. DirectoryLoader
  2. JSONLoader
  3. CSVLoader
  4. DataFrameLoader
  5. S3FileLoader
  6. S3DirLoader
  7. UnstructuredMarkdownLoader
  8. UnstructuredPDFLoader
  9. UnstructuredFileLoader
  10. UnstructuredAPIFileLoader
  11. UnstructuredExcelLoader
  12. AmazonTextractPDFLoader
  13. GCSFileLoader
  14. GoogleDriveLoader
  15. PyPDFDirectoryLoader
  16. PyPDFLoader

Note 1: Most other Langchain document loaders that implement load() and lazy_load() methods should work. The above list indicates the ones that are explicitly tested. If you have successfully tested a particular DocumentLoader other than this list above, please consider raising an PR.