{ "components": { "schemas": { "Declaration": { "additionalProperties": false, "description": "Body of ``POST /v1/admin/datasets``: the dataset a publisher declares.\n\n``filter`` is required and may be null. Null means the dataset covers every category,\nand a list means it covers exactly those.\n\nIt is required rather than defaulted because a sidecar that omits the same field means\n*unknown*, which refuses every activation on check 3. Nothing here would have to guess\nwhich of the two an absent key meant, and rather than guess, the model asks: the field\nhas no default, so pydantic refuses a body that leaves it out.", "properties": { "categories": { "items": { "type": "string" }, "title": "Categories", "type": "array" }, "filter": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "title": "Filter" }, "name": { "title": "Name", "type": "string" }, "prefix_bits": { "title": "Prefix Bits", "type": "integer" } }, "required": [ "name", "prefix_bits", "categories", "filter" ], "title": "Declaration", "type": "object" }, "HTTPValidationError": { "properties": { "detail": { "items": { "$ref": "#/components/schemas/ValidationError" }, "title": "Detail", "type": "array" } }, "title": "HTTPValidationError", "type": "object" }, "ValidationError": { "properties": { "ctx": { "title": "Context", "type": "object" }, "input": { "title": "Input" }, "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "Location", "type": "array" }, "msg": { "title": "Message", "type": "string" }, "type": { "title": "Error Type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" } } }, "info": { "title": "knownpass-api", "version": "0.1.0" }, "openapi": "3.1.0", "paths": { "/v1/admin/datasets": { "post": { "description": "Declare a dataset: create its directory and write the API-owned declaration.\n\nNeeds ``publish:base``, because declaring a dataset is how one comes into existence.\n\nRules:\n- Re-posting the SAME declaration is a no-op and answers 200 \"exists\".\n- EXTENDING the declaration answers 200 \"updated\" and rewrites it. An extension only\n APPENDS categories -- position i is bit i forever -- and may change the filter. A base\n may extend the registry under publish:base, and the manifest then carries the longer\n list while this declaration still names the shorter one. This path catches the\n declaration up, so every routine publish:delta carrying that registry stops tripping\n the publish:base gate in ``_enforce_declaration``. An extension must also agree with\n the registry the manifest has already adopted, position by position.\n- Any OTHER difference answers 409 and changes nothing. Reordering or dropping a category\n would renumber the permanent bits, so refuse it.\n- The whole declare runs under the dataset's writer lock, the first write included. A\n publish or compaction in flight answers 409 ``writer_busy``; retry.\n\nThis endpoint already holds publish:base, so extending is gated correctly. The declared\nfilter is what every later publish is checked against, and a publish:base holder is the\none meant to change it deliberately.", "operationId": "declare_dataset_v1_admin_datasets_post", "requestBody": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Declaration" } } }, "required": true }, "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Declare Dataset" } }, "/v1/admin/datasets/{name}/manifest": { "get": { "description": "Return what this dataset publishes, as the manifest itself.\n\nThis is the exporter's authoritative watermark: it reads ``covered_until`` here to derive\nthe next run's ``--since``. The exporter's own sidecars record what it PRODUCED, and the\ntwo diverge the moment a build succeeds and the ship fails.\n\nAnswers:\n- 404 ``dataset_not_found`` for a dataset nobody declared -- a mistake.\n- 404 ``no_manifest`` for one that has published nothing yet -- start from the beginning.\n\nThe exporter needs to tell those two apart.", "operationId": "get_manifest_v1_admin_datasets__name__manifest_get", "parameters": [ { "in": "path", "name": "name", "required": true, "schema": { "title": "Name", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Get Manifest" } }, "/v1/admin/datasets/{name}/uploads": { "post": { "description": "Open a resumable upload; the request body is the sidecar JSON itself.\n\nCheck everything that can refuse this publish without moving a byte here. The alternative\nis refusing after the pipeline has shipped 210 GB.\n\nChecks:\n- The sidecar must parse, and must describe this dataset.\n- Its ``kind`` decides which scope the caller needs, so a ``publish:delta`` key is turned\n away before it starts a base.\n- Its ``prefix_bits``, ``categories`` and ``filter`` must match the declaration.\n- ``records * RECORD_SIZE`` must agree with the length the client declares.\n\nThe client declares that length in the ``Upload-Length`` header. It covers the whole\nupload: the artifact, and a base's offset table after it. This service works both sizes\nout from the sidecar, so a disagreement means the two sides describe different files.\n\nThis is the one async route. Only an async route can read a raw body, and the body must\nstay raw for ``parse_sidecar``. Every filesystem touch goes to the threadpool, because\nthe sweep it runs may unlink an abandoned 210 GB upload.", "operationId": "create_upload_v1_admin_datasets__name__uploads_post", "parameters": [ { "in": "path", "name": "name", "required": true, "schema": { "title": "Name", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Create Upload" } }, "/v1/admin/datasets/{name}/uploads/{upload_id}": { "get": { "description": "Answer how much of this upload is committed, so a client that crashed can resume.\n\nThis is the only state a resuming client needs: it sends its next chunk from ``offset``\nand stops at ``size``.", "operationId": "get_upload_v1_admin_datasets__name__uploads__upload_id__get", "parameters": [ { "in": "path", "name": "name", "required": true, "schema": { "title": "Name", "type": "string" } }, { "in": "path", "name": "upload_id", "required": true, "schema": { "title": "Upload Id", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Get Upload" }, "patch": { "description": "Append one chunk; return the new committed offset.\n\nThe chunk is positioned by ``Content-Range: bytes -/``. The committed\noffset is how many bytes of this upload are on the disk, and the client sends the next\nchunk from there.\n\nRules:\n- A chunk BEHIND the committed offset is a no-op that answers the same offset. The client\n has already been credited with those bytes, and a retry after a lost response must be\n safe rather than merely harmless.\n- A chunk AHEAD of it answers 409 with the offset to resume from. Writing it would leave\n a hole full of whatever the filesystem had there, and nothing downstream could tell\n that from the artifact the publisher meant to send.\n- A chunk over 64 MiB answers 413.\n\nThe 64 MiB cap is really enforced by the proxy, not this route. The framework buffers the\nbody before this function runs, so the check below spends the memory and then refuses it.\n``client_max_body_size`` is what actually stops an oversized request, and\ndocs/ARCHITECTURE.md section 4.1 lists it beside the other edge controls. Keep the check\nanyway: it gives a client that reached the application directly the same 413 it would have\ngot from the proxy, and it fails loudly if the two ever drift apart.", "operationId": "append_chunk_v1_admin_datasets__name__uploads__upload_id__patch", "parameters": [ { "in": "path", "name": "name", "required": true, "schema": { "title": "Name", "type": "string" } }, { "in": "path", "name": "upload_id", "required": true, "schema": { "title": "Upload Id", "type": "string" } }, { "in": "header", "name": "Content-Range", "required": true, "schema": { "title": "Content-Range", "type": "string" } } ], "requestBody": { "content": { "application/octet-stream": { "schema": { "contentMediaType": "application/octet-stream", "title": "Body", "type": "string" } } }, "required": true }, "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Append Chunk" } }, "/v1/admin/datasets/{name}/uploads/{upload_id}/complete": { "post": { "description": "Verify the uploaded artifact and publish it.\n\nRefuse an upload short of the length its sidecar declares. The bytes that are missing are\nthe ones no check could stand in for.\n\nRead the retained sidecar back from the upload directory, and take the scope decision from\nit again. This service wrote that file at create time from bytes it had already validated,\nso re-reading it costs one small read and keeps the publish honest even if a different key\nfinishes an upload another key opened.\n\nCleanup:\n- Publishing an artifact this dataset already publishes answers \"already_applied\" and\n advances nothing. Both outcomes are a finished upload, so both clear its directory.\n- A PERMANENT refusal clears it too. A checksum that does not match these bytes, or a\n name this dataset already publishes, cannot pass on a retry of the same upload, so\n holding it only spends an upload slot every publisher shares.\n- An APPLIED upload had its files renamed out, so removing the rest in-request is\n cheap. An already-applied one still holds the whole artifact: claim it aside and let\n the after-response reap pay the rmtree, exactly as a swept upload is deleted.\n- A REFUSAL keeps the directory. A busy writer is transient, and deleting the upload would\n make the publisher re-send 209.9 GB to get past a lock it only had to wait for. The\n stale sweep reclaims one that is never retried.", "operationId": "complete_upload_v1_admin_datasets__name__uploads__upload_id__complete_post", "parameters": [ { "in": "path", "name": "name", "required": true, "schema": { "title": "Name", "type": "string" } }, { "in": "path", "name": "upload_id", "required": true, "schema": { "title": "Upload Id", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Complete Upload" } }, "/v1/bulk/{dataset}/files/{filename}": { "get": { "description": "Hand one manifest-listed file to nginx with X-Accel-Redirect.\n\nThe response body is empty: nginx maps the internal location onto the datasets root and\nserves the bytes itself, honouring Range and resume for free. The app never opens the\nfile except to size it for the quota record.", "operationId": "bulk_file_v1_bulk__dataset__files__filename__get", "parameters": [ { "in": "path", "name": "dataset", "required": true, "schema": { "title": "Dataset", "type": "string" } }, { "in": "path", "name": "filename", "required": true, "schema": { "title": "Filename", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Bulk File" } }, "/v1/bulk/{dataset}/manifest": { "get": { "description": "Return the dataset's published manifest for an entitled bulk key.\n\nSync def on purpose: read_manifest reads the disk, so FastAPI runs this in its\nthreadpool rather than on the event loop.", "operationId": "bulk_manifest_v1_bulk__dataset__manifest_get", "parameters": [ { "in": "path", "name": "dataset", "required": true, "schema": { "title": "Dataset", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Bulk Manifest" } }, "/v1/health": { "get": { "operationId": "health_v1_health_get", "responses": { "200": { "content": { "application/json": { "schema": { "additionalProperties": { "type": "string" }, "title": "Response Health V1 Health Get", "type": "object" } } }, "description": "Successful Response" } }, "summary": "Health" } }, "/v1/range/{prefix}": { "get": { "description": "Return ``{prefix, results}`` for one prefix, unioned across the key's datasets.\n\nSync def on purpose: the mmap reads below block, so FastAPI runs this in its threadpool\nrather than on the event loop.", "operationId": "range_lookup_v1_range__prefix__get", "parameters": [ { "in": "path", "name": "prefix", "required": true, "schema": { "pattern": "^[0-9a-f]{6}$", "title": "Prefix", "type": "string" } } ], "responses": { "200": { "content": { "application/json": { "schema": {} } }, "description": "Successful Response" }, "422": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/HTTPValidationError" } } }, "description": "Validation Error" } }, "summary": "Range Lookup" } } } }