README

Overview

The AI Feed module, developed by Yale ITS, is designed to create a feed of website content that can be ingested by and generate embeddings for an AI search service. The module leverages Drupal to query and prepare content for consumption by a language model integration framework, such as LangChain.

Features

• Retrieves published nodes accessible to anonymous users.
• Processes content to create a consistent and expected format.
• Generates a JSON feed of website content suitable for ingestion by an AI search service.
• Utilizes Drupal core and entity-related functionalities for content retrieval and processing.

Feed Shape

This module creates an endpoint of website content at api/ai/v1/content. Currently, this feed is limited to published nodes that are accessible to anonymous users. In the future, this may be expanded to include filters and new entity types. The shape of the JSON is as follows:

{
  "data": [
    {
      "url": "yalesites-yale.edu-node-18",
      "source": "drupal",
      "documentType": "node/page",
      "documentId": 18,
      "documentTitle": "Resources and Workshops",
      "documentUrl": "https://yalesites.yale.edu/resource",
      "documentContent": "...",
      "metaTags": "",
      "metaDescription": "",
      "dateCreated": "2023-10-12T16:09:21+00:00",
      "dateModified": "2023-11-30T16:11:18+00:00",
      "dateProcessed": "2024-01-23T16:05:38+00:00",
    },
    { ... },
  ],
  "links": {
    "first": "https://yalesites.yale.edu/api/ai/v1/content?page=1",
    "prev": "https://yalesites.yale.edu/api/ai/v1/content?page=1",
    "self": "https://yalesites.yale.edu/api/ai/v1/content?page=2",
    "next": "",
    "last": "https://yalesites.yale.edu/api/ai/v1/content?page=2"
  },
  "totals": {
    "total_records": 52,
    "total_pages": 2
  }
}

Data Fields

Links Fields

Totals Fields

Pagination

This API supports pagination returning 50 results per page. By default without any query parameters, the API defaults to page 1.

Query different pages with the ?page=x query parameter where x is a positive integer.

If there is not a previous or next page of results, the prev and next URL's will be empty strings.

Requirements

• Drupal 8 or later

Contribution / Collaboration

You are welcome to contribute functionality, bug fixes, or documentation to this module. If you would like to suggest a fix or new functionality you may add a new issue to the GitHub issue queue or you may fork this repository and submit a pull request. For more help please see GitHub's article on fork, branch, and pull requests