Incorporate the Knowledge of the Engelverse into Your AI-Assisted Research
Today the Engelberg Center is excited to announce a new way to incorporate articles, reports, and other research from the Engelberg Center into your AI-assisted research: the Engelberg Center Corpus. Once you connect the Corpus with the AI model of your choice, you can explore the full text of hundreds of works of research and scholarship from the Engelverse. Your model will be able to return quotes, pin cites, and context from work across the Engelberg Center’s areas of study.
Read on to learn more about what we are doing and why we are doing it, and instructions on how to set things up.
Project Goals
Scholars, students, and researchers are integrating AI tools into their process in a number of ways. If Engelberg Center scholarship has information relevant to that research, we want to make sure those people find it.
That involves two separate steps. First, we want to make sure that Engelberg Center scholarship is easy to explore with AI models. That allows deeper research into that scholarship and more precise answers. Second, we want to make sure that Engelberg Center scholarship is included in the data used to train AI models. That increases the chances that models turn to that scholarship when generating answers to relevant queries.
Implementation
Each step required different implementation steps.
Making Scholarship Available During Queries
We want to make it easy for people to get specific answers to specific questions: “how can patents help and hinder people’s access to medicines?” or “how is this type of work protected by copyright post-Warhol?” Facilitating this required us to set up a new Model Context Protocol (MCP) server. Users can connect the server directly to the LLM of their choice, supercharging the LLM’s ability to draw on the entire scholarship of the Engelberg Center when responding to prompts.
The server is more than a collection of documents. It is indexed in order to make the content easier to find, with metadata mapping that allows queries to return relevant quotes and pin cites. It also includes tools that AI models can use to explore the scholarship in order to find relevant discussions. When combined with a user’s model, it facilitates deep exploration of the entire collection of scholarship.
The Corpus MCP server can be integrated into the AI model of your choice. Critically, connecting to the server does not limit the model to the information contained on it. Instead, information from the server complements other information the model has access to (both directly through its own training data and through capabilities like web search). The benefit of connecting to the server is that it makes it much easier to get directly relevant quotes and pin cites from all of the scholarship included in the Corpus.
Including Scholarship in Training Data
Not everyone will enable a direct connection between their model and the Corpus. We still want to make sure that scholarship from the Engelberg Center is visible to them.
A great deal of scholarship from the Engelverse is already available online. However, some of that scholarship is currently hard for robots to access. Sometimes that is because it is on a platform (like SSRN) that is taking steps to prevent bots from downloading data. Other times, it is simply because it is in formats like PDF that can be hard for robots to parse with accuracy.
We solve this problem by collecting all of the scholarship from the Engelverse and creating HTML versions with high quality metadata on a server we control directly. This makes it easy for robots to access the scholarship, parse it, and incorporate it into training data sets.
We are conscious of the fact that hosting a second version of some research could create confusion for both people and search engines. As a result, the collection of scholarship intended for bot consumption includes canonical links that point to the canonical version of the work. We hope this will avoid making it harder for humans to find the “real” version.
Try it Yourself
It is one thing to read about this kind of project, and another to give it a spin. Load up the model of your choice, point it at https://corpus.nyuengelberg.org/mcp, and start some research. The Engelverse is a vast place. It contains rich scholarship about patent, copyright, trademark, privacy, competition, openness, innovation, trade secrets and more.
The easiest way to get the MCP server to your model may be to just ask it for up to date instructions. Here is some information to get you started, which is accurate as of the publication of this post.
Claude Integration
To connect your paid Claude account with the Corpus, open Settings -> Connectors. Once you are in the Connectors window, click “Add” in the upper right and choose “Add custom connector”. Pick a name (like “Engelberg Center Corpus”) and add https://corpus.nyuengelberg.org/mcp as the URL. The server does not require any authentication or additional request headers. The first query to the server will trigger a “Needs your input” dialog where you can “always allow” the model to use the connector.
ChatGPT Integration
To connect your free ChatGPT account with the Corpus, open Settings -> Security and login. Toggle Developer mode (this won’t be necessary on a paid account). Then go to Settings -> Plugins and Browse plugins. In the upper right of that screen, click the “+” icon. Fill in the form, using https://corpus.nyuengelberg.org/mcp as the Connection and choosing “No Auth” for Authentication.
Other Integrations
The Corpus uses the MCP protocol, which is an open standard. Many models support it and adding the corpus should be straightforward. The key information is that the server URL is https://corpus.nyuengelberg.org/mcp and it does not require any sort of authorization.
Regardless of what model you are using, give it a try and let us know how it works at engelberg.center@nyu.edu. Of course, as with anything related to LLMs, be sure to double check any responses against the original sources. This is an experiment and we will learn how worthwhile it is as people use it. We also plan on continuing to add more scholarship over time, so check in regularly to learn how it evolves.
Two Final Notes
There are two more things to mention as we kick this off.
First, because this is an experiment, we are going to be tracking usage and analytics. That will include some query logging including timestamp, tool name, query text, result count, and top document IDs returned. That will not include any IP address information, or any persistent identifier across queries. We will also be tracking bot and IP addresses for access to the training corpus. Finally, we will implement our standard plausible.io analytics for the project. We hope that all of these will help us improve the service. They may also be incorporated into future reports on how the service is used.
Second, as we were finalizing the launch of this project, Matthew Dahl and Eric Martínez released Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Following. Their research helped us increase the formatting accuracy of the Bluebook citations returned by the server. Thank you to both of them!
