Everything OpenGSO does, and how.

Install

OpenGSO is free on npm and needs Node.js 22 or later on macOS, Linux or Windows.

npm install --save-dev opengso

That is the whole toolkit plus the local dashboard. OpenGSO is five packages built from one codebase, so a project can install only the part it uses. Every one of them exposes the same opengso command, and a package asked for something it does not ship names the package that does.

opengso-prod is the package that ends up in a production dependency, so it has zero dependencies. The library your server imports has no HTML parser and no AI provider code: it reads files that generate already wrote and does a map lookup per request. The build checks that on every release.

Quickstart

npx opengso                                  # the terminal app: where you stand, and what to do
npx opengso start                            # set the project up, then open the dashboard
npx opengso test https://your-site.com       # score it as it is today
npx opengso generate https://your-site.com   # write the twins and the three root files
# wire opengso-prod into your server (see Prod below)
npx opengso prod test https://your-site.com  # prove the deployment serves them correctly
npx opengso test https://your-site.com       # score it again

opengso on its own opens the home screen: your project, its last score, how many twins it has, and a menu of everything there is to do. Piped or run from a script it prints the help page instead, so nothing that shells out to it changes behaviour.

opengso start asks four questions the first time it runs in a directory: your site URL and web root, which AI agents get the markdown twin, blocking, and an optional API key. After that it goes straight to the dashboard. opengso settings asks the same questions in the terminal, with your current answers as the defaults, and prints what actually changed.

Nothing in the quickstart needs an API key. The structural test, generate's free tier and all of opengso-prod run without one.

Test a URL

opengso test scores a live URL from 0 to 100 and says exactly what failed. Give it the full address, scheme included. Anything your machine can reach works: production, a staging host, or http://localhost:3000.

opengso test https://your-site.com               # the checks, then an AI review of the markdown
opengso test https://your-site.com --structure   # the checks alone. Also --offline, -nq. No key, no cost
opengso test https://your-site.com --content     # the AI review alone
opengso test https://your-site.com -q            # the full run, with the review required

The structural pass is the score: 22 deterministic checks, no model involved, the same responses always producing the same number. The content review is a second, optional pass that reads every markdown document the site serves and grades it out of 100 on its own. It needs a key, refuses to run in CI, and prints as its own section. The two numbers are never blended.

Reading the report

Rows are grouped worst first: failed, partly there, suggestions, locked, passed, adopted, not applicable. Each row names the check, what it verifies and the specific reason it reached its verdict, with the header value or rule actually observed. A partial row shows a small meter and the split, 2 of 4. A suggestion is marked TIP rather than with a cross, because it is not a failure.

The Next.js demo in the repository, before anything had been generated for it:

OpenGSO test  http://localhost:3000/resume
Detected: Next.js

   65/100  ███████████████▋░░░░░░░░  4 of 6 scored checks passed, 2 failed, 1 n/a
    ideas  ██▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░  1 of 15 suggestions adopted, 12 waiting on B1


Failed (2)
----------
  FAIL A1 robots.txt exists and does not disallow known AI bots  -18 pts
      No robots.txt at http://localhost:3000/robots.txt (HTTP 404). AI crawlers
      get no explicit signal about what they may read.
      → Serve a robots.txt that allows the AI agents you want reading the site
        (+18). `opengso generate` writes one.

The report ends with Do this next: every scored fix ranked by the points it recovers, so the top line is always the most valuable edit you can make. The suggestions follow under their own heading, heaviest first. Every price the report prints is in score points, how far that one fix would move the headline, and not the check's weight.

Test the same URL again and the headline says what moved since the last run, such as +15 since 2h ago, with a sparkline of the runs before it. The history is kept in .opengso/history.json, newest 20 runs per URL, and is never served.

Options

Exit codes

opengso test exits 1 only when a scored check fails, so it drops into CI without failing builds over conventions a site has not adopted. --strict widens that to any failing check, for a project that has adopted the markdown layer and wants CI to notice the day it stops being served. Exit 2 means the command line could not be run as written: an unknown option, a malformed value, no URL. The exit code always comes from the structural checks, so a model's opinion can never fail a build.

The fix prompt

Every test result can become one Markdown prompt to paste into Claude Code or any coding agent with access to the repository.

opengso test https://your-site.com --copy              # the report, then the prompt on your clipboard
opengso test https://your-site.com --prompt            # the prompt instead of the report
opengso test https://your-site.com --prompt --only A3  # one check
opengso test https://your-site.com --copy --only B1,B7 # a few

The prompt carries the score, the URL, the expected twin URL and the detected framework, then each failed check with the points it cost, what actually happened, what to change, and a "Done when" line naming the command that proves it fixed. Suggestions follow, each marked as outside the score, then the twelve checks waiting on B1. It links only to opengso-prod and OpenGSO's own documentation.

Framework detection covers Next.js, Nuxt, Astro, SvelteKit, Remix, Gatsby, Docusaurus, Hugo, Jekyll, WordPress, Vite and plain HTML, so the prompt tells the agent where the middleware actually goes. It matches only markers a framework emits itself, never a word in the page's text.

--copy uses the clipboard your system already has: pbcopy on macOS, PowerShell (then clip) on Windows and WSL, wl-copy, xclip or xsel on Linux. Over SSH with none of those, it asks the terminal to set its clipboard. When nothing works it says so, and --prompt --out fix-prompt.md saves the prompt to a file instead. A failed copy never changes the exit code.

The same prompt is one keypress away elsewhere. The dashboard has a Fix it with Claude Code card, a copy button on every group and every failing check, and copies everything when you press c. The home screen offers c when a test finishes.

Anything the site sent, such as a header value or a JSON-LD field, is quoted as a single-line code span, fenced so it cannot close its own span, and cut at 400 characters. The prompt tells the agent that quoted text is evidence and never instructions, so a server cannot use its own responses to add steps to the prompt.

Generate

opengso generate crawls a site once and writes the files AI agents want: a clean markdown twin of every page, plus robots.txt, llms.txt and sitemap.xml.

npx opengso generate http://localhost:3000

public/
  md/
    index.md           served at /index.md
    about.md           served at /about.md
    pricing.md         served at /pricing.md
  robots.txt
  llms.txt
  sitemap.xml
.opengso/
  manifest.json        bookkeeping, outside the web root, never served

The crawl is a same-origin breadth-first search over real links from the URL you give it, up to crawl.maxDepth deep and crawl.maxPages requests. A page nothing links to goes under extraRoutes. Set publicOrigin to the deployed address and every absolute URL generate writes names it, rather than the dev server that was crawled.

Two tiers

• Free, the default. Algorithmic and deterministic, needs no key, and produces the markdown everything else is measured against.
• AI, with --tier ai. Rewrites each page with your provider, then verifies the result: every number, date, price and percentage in the algorithmic extraction must survive with its sign and scale, and the model may not add one. If anything goes missing or appears, the AI output is discarded and the algorithmic version is written instead, with a warning naming the fact. The AI tier runs only on your machine and refuses to run in CI.

It never overwrites your files

A robots.txt, llms.txt or sitemap.xml without OpenGSO's generated-by marker is left exactly as it is, and OpenGSO's version is written beside it as robots.txt.opengso-proposed with a warning naming both paths. Twins follow the same rule: a file whose frontmatter does not say OpenGSO wrote it is left alone. --force is the only way past that. Set generated_by: manual in a twin's frontmatter to claim it as yours and have it served as is.

Frontmatter

---
title: "Pricing"
url: "https://example.com/pricing"
last_updated: "2026-10-20T09:12:44.102Z"
tokens: 412
source_hash: "0cdc3ec67ce75fa0"
generated_by: "opengso-free"
description: "Plans, prices and what each one includes."
---

The prod middleware strips the frontmatter before serving and moves the same facts into response headers. source_hash is taken over the extracted markdown, not the raw HTML, so it changes exactly when the page's content changes and not on every rebuild.

Options

Generate is a local, deliberate act, so it is a devDependency and never runs in production or on a schedule. A twin is a cache, and check B14 in opengso test is the safety net that says when it has gone stale.

Static sites

A site that is a folder of HTML files needs no server and no crawl. Point generate at the folder and it writes each twin beside its page, under the name a static host serves at the page's URL plus .md.

npx opengso generate dist --origin https://example.com   # every page in dist/
npx opengso generate dist/pricing.html                   # one page, after you edit it
npx opengso generate --check                             # which pages changed since their twins

Generate from what your tool builds and publish that folder: dist for Vite and Astro, docs/.vitepress/dist for VitePress, build for Docusaurus, out for a Next.js static export, public for Gatsby and Hugo, and _site for Jekyll and Eleventy. A rebuild usually empties the folder, so run generate after building and before you deploy.

GitHub Pages and other folder sites

A GitHub Pages project site lives at user.github.io/repo/. Pass that as the origin and every URL generate writes carries the folder. robots.txt is not written, because crawlers read it only at the root of the host, and the root belongs to another site. When the folder is one GitHub Pages publishes, generate writes an empty .nojekyll so Jekyll does not build the twins into pages of their own.

npx opengso generate site --origin https://user.github.io/repo

This site is built exactly that way: plain files on GitHub Pages, with its twins, llms.txt and sitemap written by opengso generate. A static host cannot negotiate or set custom headers, so an agent asking for a page itself still gets HTML, and the header checks stay open. The twins are served at their .md URLs all the same.

Prod

opengso-prod serves the twin at the same URL to the agents that ask for markdown, with the right headers, and the HTML to everyone else. It reads the manifest once at startup, so every request after that is a map lookup.

npm install opengso-prod

Next.js 16

Next 16 calls the file proxy.ts, at the project root, exporting proxy.

import { NextResponse } from 'next/server';
import { applyPassthroughHeaders, createNextMiddleware } from 'opengso-prod/next';

// Built once at module scope, so a request is a lookup and nothing else.
const opengso = createNextMiddleware({ root: process.cwd() });

export function proxy(request: Request) {
  const { response, headers } = opengso(request);
  if (response) return response;          // markdown, a 304, or a 406

  const next = NextResponse.next();
  applyPassthroughHeaders(next, headers); // Vary, X-AEO-Version, Link on the HTML
  return next;
}

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

On Next 15 the same code goes in middleware.ts, exporting middleware, with runtime: 'nodejs' in config. A Next 16 proxy file refuses a runtime field, so leave it out there. Under next dev the adapter reloads the config and twins by itself when they change; a production server reads them once.

Express and other Node servers

import { createNodeMiddleware } from 'opengso-prod/node';

app.use(createNodeMiddleware()); // first, before your routes and static files

SvelteKit

// src/hooks.server.js
import { createFetchMiddleware } from 'opengso-prod/fetch';

const opengso = createFetchMiddleware();

export const handle = ({ event, resolve }) => opengso(event.request, () => resolve(event));

Hono

import { createFetchMiddleware } from 'opengso-prod/fetch';

const opengso = createFetchMiddleware();
app.use((c, next) => opengso(c.req.raw, async () => { await next(); return c.res; }));

Vite

// vite.config.js
import opengso from 'opengso-prod/vite';

export default { plugins: [opengso()] };

The package README on GitHub has the exact wiring for every framework in the table, and docs/prod-api.md documents every export.

No adapter can take a page down. A broken opengso.config.json is logged once, as one [opengso-prod] line naming the problem, and every request goes to your app as if OpenGSO were not installed. A broken config costs you the markdown, never the site.

On boot, once per process, the middleware reports what it found: whether robots.txt blocks any AI bot, whether llms.txt and sitemap.xml are there, and how many twins it loaded. Pass logStartup: false to silence it.

Edge runtimes

An edge function has no disk, so the twins travel as data. opengso prod data writes them, with your routing and custom crawlers, into one module for the function to import.

npx opengso-prod prod data      # writes opengso-edge-data.js (.ts in a TypeScript project)

// A Cloudflare Worker in front of its static assets
import { createEdgeMiddleware } from 'opengso-prod/edge';
import data from './opengso-edge-data.js';

const opengso = createEdgeMiddleware({ data });

export default {
  fetch: (request, env) => opengso(request, () => env.ASSETS.fetch(request)),
};

Run it after opengso generate and before your bundler. Generate refreshes the module when it finds one. It writes nothing when there are no twins, rather than an empty module that would quietly turn the markdown off, and it warns past 1 MB, since edge hosts cap a function's size. opengso-prod/edge imports no Node module, so any runtime with Request and Response takes it.

Verify a deployment

npx opengso prod test https://your-site.com/some-page

Five probes, each isolating exactly one variable, reported under the lever it belongs to. The routed-bot probe sends a browser-typical Accept header, so a markdown response can only have been caused by the User-Agent. The Accept probe sends a browser User-Agent, so a markdown response can only have been caused by the Accept header. When the two levers disagree, such as a bot you route markdown to that your own robots.txt disallows, that gets its own section.

The exit code follows the failures, not the warnings. A warning is a real finding, but it is the kind a CDN can cause and that you may knowingly accept.

The two bot levers

Which bots get markdown and which bots are blocked are two separate settings, and OpenGSO keeps them apart everywhere: in the config, in the terminal, in the dashboard and in prod test.

• bots.markdownRouting chooses a format. A bot on this list gets the twin at the page's own URL without asking for it. A bot not on it gets the normal HTML page, exactly like a person. It never blocks anybody, and an agent nobody has heard of gets HTML, not a 403.
• bots.blocked requests no access. It is opt-in, empty by default, and expressed only as Disallow rules in the robots.txt generate writes. It is advisory, as robots.txt always is, and it is never applied to a crawler OpenGSO does not know.

The production middleware has no blocking code path at all. Blocking lives in a static file a crawler chooses to honour, where a site owner can read it.

The default routing

OpenGSO recognises 28 AI crawlers by name: 13 that collect training data, 8 that build search indexes and 7 assistants that fetch a page because someone asked about it. The default routing is the 18 that are not search engines and can be recognised by User-Agent:

GPTBot, ChatGPT-User, ClaudeBot, Claude-User, anthropic-ai, Perplexity-User, CCBot, Bytespider, meta-externalagent, meta-externalfetcher, cohere-ai, MistralAI-User, DuckAssistBot, Diffbot, Ai2Bot, FirecrawlAgent, ImagesiftBot and Timpibot.

No preset routes a search crawler. Every markdown response carries X-Robots-Tag: noindex, and at the page's own URL that would tell Bingbot, Applebot or OAI-SearchBot to drop the page from their results. You can tick one by hand, and settings, the dashboard and prod's startup log all warn when you have. Any client can still ask for markdown with Accept: text/markdown.

Crawlers OpenGSO does not ship with

bots.custom adds an in-house agent, or an AI product newer than your installed version, to the catalogue. It is neither lever: an entry is routed to only if its name is also in markdownRouting, and disallowed only if it is also in blocked. A name already in the shipped list is ignored, so a typo cannot shadow GPTBot.

{ "name": "AcmeBot", "vendor": "Acme Inc", "uaPatterns": ["acme-crawler"], "kind": "assistant" }

Configuration

opengso.config.json is written by opengso settings, opengso start and the dashboard, and can be edited by hand. It is meant to be committed: the API key is never in it. OpenGSO looks in the current directory, then upward, but never past the project, and --project <dir> points at one explicitly. Any subset is fine; missing fields take their defaults.

Every field present is checked on load, and a wrong one is reported by name. A field OpenGSO does not know is never an error, so a config written by a newer version still loads: it is named on stderr with the field it was probably meant to be, so publicdir gets "Did you mean publicDir?".

{
  "site": "http://localhost:3000",
  "publicOrigin": "https://example.com",
  "extraRoutes": ["/pricing"],
  "crawl": { "maxPages": 200, "maxDepth": 4, "exclude": ["/admin"] },
  "generate": { "tier": "free", "minWords": 40 },
  "bots": {
    "markdownRouting": ["ClaudeBot", "GPTBot", "AcmeBot"],
    "blocked": [],
    "custom": [{ "name": "AcmeBot", "uaPatterns": ["acme-crawler"] }]
  },
  "ai": { "provider": "anthropic", "model": "claude-sonnet-5-5" }
}

The 22 checks

Seven checks are scored and make the 0-100: AI crawlers allowed in robots.txt, a valid sitemap.xml, real content in the raw HTML, a fast first byte, no noai tag, HTTPS with HSTS, and the HTML page still answering once the markdown layer is in place. The other fifteen are suggestions: llms.txt, JSON-LD, the markdown twin and the headers around it. The best-read sites on the web have mostly not adopted them, so not having one never moves the number. They are counted on their own, as "N of 15 adopted".

score = round(100 × earned / applicable)

applicable is the sum of the weights of every scored check that returned pass, partial or fail. The scored weights total 61 and the suggestions another 61, so a weight is not a score point: every "+N" the report prints has been converted into score points for that run. On a real HTTPS host, a 9-weight check is worth 15 points.

When the twin does not answer, the twelve B checks that read it have nothing to inspect. They become locked behind B1, the heaviest check in the table, and the report shows them as one block, "12 more suggestions apply once B1 passes", rather than as twelve findings about one missing file. Every check that can be locked is a suggestion, so the cascade never moves the score.

Three calibration points are pinned by tests: a well-built site with no markdown layer scores 100, with A2, A4 and B1 open as suggestions; the same site serving a correct twin scores 100 with the suggestions adopted; and only a scored failure moves the number below 100.

A status code is never trusted on its own. A catch-all route that answers /robots.txt, /llms.txt or a twin URL with its HTML shell and a 200 is detected by reading the body, and scored as the missing file it is. Checks B2, B4 to B12 read response headers only a server can set, so a static host such as GitHub Pages can serve the twins and still leave those open; an edge function in front of it can set them.

The content review

The optional AI pass gives the markdown a separate score out of 100, derived from five areas so it can be explained line by line. An area the model did not score leaves the denominator rather than counting as zero.

The wire spec

opengso/1.0 is the contract between a site and an agent: which URL holds the markdown, which headers say so, and what happens when an agent asks for a format the server cannot produce. It is plain HTTP that a CDN, a proxy and a curl one-liner already understand, and nothing in it requires OpenGSO.

1. Serve the same content as markdown when asked, at the same URL.
2. Say so with Vary: Accept and a Link: rel="alternate".
3. Do not index the twin.
4. Return 406 instead of guessing.

There are two ways to ask for markdown: an Accept: text/markdown header, which works for agents nobody has heard of yet, and a known AI User-Agent on the routing list, a courtesy for agents that send no Accept header. Both rewrite: same URL, same status, different body. Neither redirects, because a redirect changes the URL the agent records and cites.

On a markdown response

On an HTML response

An ETag sent back as If-None-Match gets 304 Not Modified with no body when the twin is unchanged. A request whose Accept header excludes both HTML and markdown, on a route with a twin, gets a 406 Not Acceptable naming the two formats it can have. Every other request passes through to the application untouched, so an API asking for JSON or an event stream is never the middleware's to answer.

The twin is the same content as the HTML page in a cleaner format: not a summary, not an abridgement, and not a different document. Serving different information at the two URLs is cloaking, and the spec does not permit it.

AI providers

OpenGSO is bring-your-own-key, and most of it needs no key at all. A key buys exactly two things: generate's AI tier and the content review. Twenty-five providers are built in, dispatched on the API shape a provider speaks rather than on its name, so every one is a row in one table.

Anthropic, OpenAI, Google Gemini, xAI, Mistral, DeepSeek, Alibaba Qwen, Moonshot Kimi, Z.ai GLM, MiniMax, Cohere, Perplexity, OpenRouter, Groq, Together AI, Fireworks AI, Cerebras, SambaNova, DeepInfra, Hugging Face, NVIDIA NIM, Novita AI, Ollama and LM Studio on your own machine, and any endpoint you name with custom.

opengso settings                                  # interactive, writes .env and .gitignore for you
opengso test https://your-site.com --provider groq   # one run, overriding the config
opengso help providers                            # the whole catalogue, from the code

The default model for each is a fast, inexpensive one, because rewriting a page is summarising work and a site can have hundreds of pages. Any model the provider serves works through ai.model or --model. A name people actually type, like --provider claude, stops and asks "Did you mean anthropic?" rather than guessing.

The key is written to .env in your project, after .env is added to .gitignore, and nowhere else. It is never written to opengso.config.json, never returned by the dashboard, and never sent anywhere except the direct call to the provider you picked. With Ollama or LM Studio there is no key and nothing leaves the machine.

Dashboard and terminal

There are three ways in, and they run the same code, so they never disagree about a score, a header or a file.

• Commands, for scripts and CI: opengso test, generate, prod test, prod data, settings, each with --json.
• The home screen, a full-screen terminal app when you type opengso on its own. It is a launcher: picking Test runs the real command in normal scrollback, so the report can be scrolled and copied, then comes back. It needs a real terminal on both ends and turns itself off when CI, NO_COLOR or OPENGSO_NO_TUI is set or TERM is dumb, falling back to plain prompts.
• The dashboard, with opengso start, in the opengso package. Five screens: Test, Generate, Prod, Bot settings and Settings. Generate can show you the exact bytes an agent would receive, read back off disk.

Keyboard

Dashboard options

The dashboard can write an API key to disk, so it listens on 127.0.0.1 only, refuses a request with a cross-origin Origin or a Host it was not started on, and serves a strict Content Security Policy. With --host set to anything else it requires a token, new on every launch, carried in the URL it prints. The privacy policy has the details.

Troubleshooting

The score is 100 but the report still lists suggestions

Expected. Seven checks make the score; the fifteen suggestions never move it. opengso generate plus opengso-prod adopts most of them at once.

B1 is a suggestion and B2 to B14 show as locked

The site serves no markdown twin. Run opengso generate, add opengso-prod to the server, restart it, and test again.

CI passed although a suggestion failed

Only a scored failure sets exit code 1. Pass --strict once the markdown layer is live and should stay that way.

100/100, but B14 says the twin is stale

The page changed and the twin was not regenerated, so agents are sent the old facts. The report marks it WARN and leads with it. Re-run opengso generate.

B7 scores 2 of 4 on a Next.js site

Expected. Next's App Router will not let middleware add Vary to a response it did not create, so the markdown side has it and the HTML side does not. It is a partial suggestion and never changes the exit code.

A8 says n/a

HTTPS and HSTS mean nothing on localhost, so the check leaves the score. Test the deployed URL to measure it.

"... already exists and wasn't written by OpenGSO"

Your own file was kept, and OpenGSO's version is beside it as <name>.opengso-proposed. Merge what you want, or pass --force.

"Only N words of content"

The page probably renders its content with JavaScript. OpenGSO reads the server's HTML and has no headless browser. Server-render the page, generate from the production build, or lower generate.minWords if the page really is that short.

Agents still get HTML

Check that generate ran and .opengso/manifest.json exists, that the crawler is in bots.markdownRouting, that a production server was restarted, and on Next.js that the matcher covers the page. opengso prod test probes all five cases.

Bare opengso prints prompts instead of the home screen

The home screen needs a real terminal on stdin and stdout, and turns itself off under CI, NO_COLOR, OPENGSO_NO_TUI or TERM=dumb. The prompts do the same things.

Versioning

The five packages follow semver and always share a version: today all five are 0.1.2. The first release on npm was 0.1.0, on October 20, 2026. Until 1.0, any minor release may change a command, a flag, a config field or an export.

The spec version, opengso/1.0, moves separately. It names what goes over the wire, so it changes only when the wire format does, and many package versions can implement the same spec.

Develop

The source is plain JavaScript with JSDoc types, checked by TypeScript, so there is no compile step to run it: after npm install, npm run opengso -- --help runs the CLI from the source tree.

git clone https://github.com/sherwinvishesh/OpenGSO.git
cd OpenGSO
npm install
npm run typecheck        # JSDoc types, checked by tsc
npm test                 # the Vitest suites
npm run build            # bundles the five packages
npm run check:bundles    # what each bundle may and may not contain
npm run check:packages   # what each tarball would ship
npm run install:global   # build, pack and install opengso, to drive it like a user

packages/core        @opengso/core, private and never published. Every pillar is
                     implemented here once and bundled into each package at build time.
packages/testing     opengso-testing
packages/generate    opengso-generate
packages/prod        opengso-prod
packages/cli         opengso-cli
packages/opengso     opengso, including web/ (the dashboard: plain HTML, CSS and JS)
examples/nextjs-demo the Next.js app the whole pipeline is exercised against
docs/                the user and contributor docs
website/             this site

Pull requests run the same checks as the CI badge on the repository, on Node 22 and 24 on Linux and Node 22 on macOS and Windows. Docs that state facts are tested against the code, so a weight changed in one place and not the other fails the suite. CONTRIBUTING.md has the rest, including how to add a bot.

License

OpenGSO is open source under the Apache License, Version 2.0, with its LICENSE file in the repository. Geist is used on this site under the SIL Open Font License. The source is on GitHub, and the packages are on npm.