Shared Libraries

A shared library is a Python module that your team writes once and imports from many code evaluators. Put your own text cleaning, your product's rules, or a score that several evaluators need in a library. When you fix it, every evaluator that imports it gets the fix.

# In any code evaluator
from shared.acme_text import clean, is_polite


def evaluate(ctx):
    reply = clean(str(ctx.observation.output or ""))
    return EvaluationResult(scores=[
        Score(name="polite", value=is_polite(reply), data_type="BOOLEAN"),
    ])

import shared.acme_text as t and from shared import acme_text also work.

Create a library

1. Go to Agent Observability → Evaluators (ap1, us1) and open Shared libraries
2. Click New Library
3. Give it a name. This is the name that you import: acme_text is shared.acme_text
4. Write the code and click Save

A library is plain Python: functions, classes and constants. It does not define evaluate. The runtime names (Score, EvaluationResult) are not in a library's globals, so import them when a library makes scores:

# shared library "acme_text"
import re

from oodle_eval.runtime import Score
from oodle_eval.v1 import text

_SPACE = re.compile(r"\s+")
POLITE = ("please", "thank you", "thanks")


def clean(s):
    """Lower-case the text and join runs of white space."""
    return _SPACE.sub(" ", s).strip().lower()


def is_polite(s):
    """True when the text holds one of the polite phrases."""
    s = clean(s)
    return any(p in s for p in POLITE)


def politeness(ctx, name="polite"):
    """A BOOLEAN score for the reply."""
    return [Score(
        name=name,
        value=is_polite(text.reply(ctx)),
        data_type="BOOLEAN",
        higher_is_better=True,
    )]

Rules for a library

Rule	Why
The name matches ^[a-z][a-z0-9_]{0,63}$: lower case, digits and _, starting with a letter	It is a Python module name
The name is not a Python keyword, and not oodle_eval or shared	Python cannot import a keyword, and the other two names are the library and the package that hold shared libraries
The name is unique in your Oodle instance	Code imports a library by its name
The name has no dots and no folders: use utils_text, not utils.text	Each library is one module directly under shared
The name cannot change after you create the library	The code that imports the library names it
The source is at most 256 KB
Imports are absolute: from shared.other import x, not from .other import x	The sandbox checks each import by its top-level name
A library can import the sandbox's standard-library modules, oodle_eval and other shared libraries	These are the modules in the execution environment

A library runs in the same sandbox as the evaluator that imports it, so its work counts toward that evaluator's 5 second and 128 MB limits.

To give a library a different name, create a library with the new name, change the imports in each template and library that uses the old one, then delete the old library.

One evaluator can use at most 20 shared libraries, with at most 1 MB of library source in total. Libraries that other libraries import count too. Past either limit, the evaluator fails with an error that says which limit.

Versions and pins

Each save that changes the source makes a new version: 1, 2, 3 and so on. Open a library to see its versions, and who saved each one and when. You can read the source of each version. To go back to an old version, open it, click Copy into the editor, then click Save. This adds a new version with the old source, so no version is lost.

A code template runs the latest version of each library that it imports. To keep a template on a known version, pin the library on the template:

{"libraryPins": {"acme_text": 3}}

The pins replace all the pins of the template. To unpin one library, send the pins without it. To remove all the pins, send {}. A pinned template runs that version until you change the pin. A pin also applies when the template loads the library through another library. Pin a library when a change to it must not move the scores of an evaluator until you have checked the change.

Oodle finds the libraries that a template uses from its import shared.<name>, from shared.<name> import and from shared import <name> statements, and from the imports in those libraries. When an evaluator imports a library that does not exist, or pins a version that does not exist, the run fails with LIBRARY_NOT_FOUND and the evaluator writes no score for that span.

A saved change to a library reaches live evaluators within a minute.

Try a change before you save it

A test run can use draft library source in place of the saved version. Send it in libraries, a map of library name to source. With MCP (genai_evaluator_workbench):

{"action": "try_code", "sourceCode": "from shared.acme_text import is_polite ...",
 "libraries": {"acme_text": "def is_polite(s): ..."}}

Delete a library

You can delete a library only when no code template and no other library imports it. The library page shows Used by: the templates and libraries whose code imports it. Remove the import from each of them, then delete the library. A delete that Oodle refuses names each of them.

Deleting a library also deletes its versions.

With the CLI and MCP

oodle genai code-libraries list
oodle genai code-libraries get <id>
oodle genai code-libraries create ...
oodle genai code-libraries update <id> ...
oodle genai code-libraries versions <id>
oodle genai code-libraries delete <id>

Run oodle genai code-libraries <command> --help for the arguments of each command.

With MCP, genai_evaluator_workbench reads libraries and manage_genai_evaluators changes them. An agent can also read and search a library as a file, next to the oodle_eval files:

{"action": "library_files", "prefix": "shared/"}
{"action": "read_library_file", "path": "shared/acme_text.py", "version": 2}
{"action": "grep_library", "pattern": "def is_polite", "path_prefix": "shared/"}

{"action": "list_code_libraries"}
{"action": "get_code_library", "name": "acme_text"}
{"action": "list_code_library_versions", "name": "acme_text"}
{"action": "get_code_library_version", "name": "acme_text", "version": 2}

{"action": "create_code_library", "name": "acme_text", "description": "Text helpers",
 "sourceCode": "def clean(s): ..."}
{"action": "update_code_library", "name": "acme_text", "sourceCode": "..."}
{"action": "update_code_library", "name": "acme_text", "restoreVersion": 2}
{"action": "update_template", "id": "<template id>", "libraryPins": {"acme_text": 3}}
{"action": "delete_code_library", "name": "acme_text"}

restoreVersion saves the source of that version as a new version. manage_genai_evaluators does not save source that is not valid Python, and its reply gives warnings for imports that cannot load. For more, call {"action": "help", "topic": "shared_libraries"}.

Through the API, the routes are code-libraries and code-libraries/{id}/versions under the GenAI API. See the API reference for the request and response shapes.