Skip to content

Custom Vision Prompts

The prompt that is sent to the LLM vision model for each screenshot is generated by a prompt module. This page explains how prompts work, how to use an alternate prompt, and how to write your own.

By default, every step uses the built-in prompt defined in src/folge_cli/batch_process.py. For cases that need a highly customized instruction — for example, describing a specific application's editor layout — you can select an alternate prompt module with --prompt.

How prompts are modeled

A prompt module is a Python file in src/folge_cli/prompts/. Each module exposes a single function:

def generate_prompt(step, guide_title, previous_step=None, next_step=None):
    """Return the prompt string for one step."""
    ...

Arguments:

Argument Description
step Dict with step_id, title, body, and image keys
guide_title Title of the parent guide
previous_step Preceding step dict, or None for the first step
next_step Following step dict, or None for the last step

The function returns the prompt string sent to the vision model alongside the screenshot.

Using a custom prompt

Select a prompt module with --prompt on either batch-process or pipeline:

# batch-process (enrichment stage)
folge-cli batch-process --project my-guide --prompt brailleblaster

# full pipeline
folge-cli pipeline --project my-guide --prompt brailleblaster

Without --prompt, the built-in default from batch_process.py is used unchanged.

Auto-registration

Prompt modules are auto-registered: the --prompt flag's choices are discovered automatically from the contents of src/folge_cli/prompts/. Just dropping a new .py file into that folder immediately makes its name a valid --prompt choice — no registry to update, no code to edit in batch_process.py.

Scaffolding a new prompt

To avoid hand-typing (and possibly misspelling) the function signature, use the new-prompt subcommand to generate a correctly formed module:

folge-cli new-prompt my-special-case

This creates src/folge_cli/prompts/my_special_case.py containing the standard generate_prompt signature and the built-in JSON-schema prompt template. The name is normalized to a safe identifier (My Special Case → my_special_case) and validated, so typos and invalid module names are rejected up front.

Behavior:

  • Existing modules are not overwritten unless you pass --force.
  • The created module is immediately available as a --prompt choice.
  • After scaffolding, edit the generate_prompt body to add your custom instructions (for example, a mandatory opening sentence for a specific tool), then run folge-cli batch-process ... --prompt my_special_case.

Bundled example: brailleblaster

The repository ships one prompt module, brailleblaster, which demonstrates a highly customized prompt. It instructs the model that when the screenshot shows the BrailleBlaster editor, the description must begin with an exact opening sentence describing the window layout:

"BrailleBlaster window showing a style panel on the left, print text editing panel in the center, and braille preview on the right. The top menu bar and toolbars provide editing controls."

It adds two further rules for this specific tool:

  • Quoted material (button labels, menu names, tooltips) must be wrapped in double quotation marks, never single quotes.
  • Every button or control must be named exactly as it appears in the BrailleBlaster GUI, with no description of the button's appearance or glyph (no "T icon", "H button", "H (Heading)", "A button", or formatting-bar icon descriptions).

This kind of structured instruction is appropriate for BrailleBlaster documentation but would be inappropriate for other projects — which is precisely why prompts are kept as swappable modules.

The default prompt structure

The built-in prompt asks the model to return a single JSON object with a vision object containing alt_text, long_description, ocr_text, ui_controls, important_element, and confidence. It also passes the guide title, previous/current/next step titles, and the step instruction for context. Custom modules are free to keep or replace this structure so long as they return a prompt the vision model interprets consistently.

Removing a prompt

Simply delete the .py file from src/folge_cli/prompts/. It will no longer appear as a --prompt choice on the next run.