ATO Bot

Technical White Paper: ATOBot Technical White Paper

Installation Tutorial: Watch on YouTube

Status: Active Development

ATO Bot is an open source platform for performing NIST SP 800 53 assessment and supporting the broader federal Authority to Operate process.

I built it to see how far AI could be applied to authorization work when the goal is not simply generating documents faster, but improving the actual assessment process.

ATO Bot takes source evidence, prepares it for assessment, evaluates controls at the objective level, identifies gaps and conflicting evidence, applies defined assessment policy, gives assessors a complete review environment, supports remediation and closure, and carries the same assessment record into reporting, SSP development, OSCAL export, and other authorization outputs.

It is a full application, not a prompt wrapped around a document library.

Evidence Management

ATO work starts with evidence, and that evidence rarely exists in one place.

ATO Bot supports separate libraries for system specific project evidence, enterprise policies, enterprise procedures, and common control providers. This allows an assessment to use evidence from different responsibility boundaries without treating everything as if it came from the system owner.

Evidence can include PDFs, Word documents, Excel workbooks, PowerPoint presentations, Visio files, text, Markdown, screenshots, scans, and other image based artifacts.

The application does more than extract text from those files. It preserves source structure including pages, sections, tables, rows, columns, slide context, and OCR derived content so later findings can be traced back to the material that produced them.

Evidence Preparation

Uploaded documents move through a staged evidence preparation process.

ATO Bot parses the original source, screens the extracted content for potential NIST control relevance, expands useful fragments with their surrounding context, removes redundant expansions, classifies the resulting evidence, and creates embeddings for semantic retrieval.

The classification process distinguishes between different kinds of evidence such as policy, procedure, technical implementation, screenshots, management approval, and testing evidence. It also records likely control support, evidence strength, confidence, and other information used later in the assessment.

This gives the assessment engine structured evidence units rather than dumping entire documents into a model and hoping it finds the right information.

Full NIST SP 800 53 Assessment

The primary capability in ATO Bot is full control assessment.

For each control, the application builds a criteria package that identifies the objectives being assessed, the applicable assessment policy, and any family or control specific handling that applies.

It then builds a bounded evidence package for that control using project evidence, enterprise evidence, common controls, semantic retrieval, and other available evidence sources.

AI evaluates the evidence against the individual assessment objectives. It can identify evidence that supports an objective, evidence that contradicts it, missing information, weak evidence, and gaps that prevent a clean determination.

The result is stored as structured assessment data rather than being left as an AI conversation.

By the time assessment is complete, ATO Bot has objective determinations, control findings, evidence citations, confidence information, identified gaps, remediation context, rollups, reviewer attention indicators, and the supporting assessment history.

Evidence Quality and Contradiction Handling

Evidence is not automatically good evidence just because a document exists.

ATO Bot explicitly looks for weak support, incomplete evidence, contradictory information, stale information, missing corroboration, and situations where the evidence only partially addresses an assessment objective.

Those conditions remain attached to the finding so the assessor can see where the system had difficulty reaching a conclusion.

This is important because a control assessment often comes down to the quality of the evidence, not whether somebody uploaded a document with the right control number in it.

Code Governed Adjudication

AI performs evidence analysis in ATO Bot, but it does not have final authority over the control determination.

The application has a separate assessment policy layer containing thresholds, assessment buckets, objective handling rules, family rules, control overrides, and override precedence.

The structured results from objective assessment are passed into that policy layer. Application code determines how those results map to the final control status.

That means the model can explain what the evidence appears to show, but it cannot simply decide that a control is compliant because its generated narrative sounds convincing.

ATO Bot can also reproduce why a particular determination was reached because the assessment criteria, evidence, objective results, policy, and final decision are all persisted.

Assessment Recovery and Reassessment

Model based workflows do not always succeed on the first pass.

ATO Bot includes retry and reassessment logic that can rebuild evidence context, rerun a bounded assessment, and recover from incomplete or weak assessment passes without throwing away the history of the original run.

Retries, refreshed findings, attempt information, and activity history remain part of the assessment record.

This also supports targeted reassessment when evidence changes or a reviewer wants a specific control evaluated again.

Findings and Reviewer Workbench

The findings workspace gives the assessor a working environment for reviewing the results.

Assessors can review controls by family or determination, identify items that require attention, inspect the evidence cited for an individual control, review objective reasoning, examine gaps and contradictions, and move between the individual finding and the larger assessment.

ATO Bot keeps the reviewer connected to the underlying evidence instead of reducing the assessment to a compliance score.

The assessment remains a collection of inspectable records that can be questioned and changed.

AI Dissent and Challenge

ATO Bot can preserve disagreement with an assessment result.

A secondary review path can challenge the original assessment and produce a dissenting analysis. That dissent is stored separately from the control determination instead of automatically replacing it.

The assessor can review the original result, the evidence, and the dissent, then decide whether the finding should stand, be changed, or require additional work.

The same approach is used for reviewer challenges, notes, and manual overrides.

ATO Bot records those interventions and the reason for them as part of the assessment history. The assessor remains responsible for the result.

Remediation and Closure

ATO Bot does not end when it identifies a failed or partially satisfied control.

The application includes a closure process that can take the actual finding and generate targeted questions for the assessor or control owner. Those questions are based on the discovered gap rather than a generic checklist.

The answers can then be analyzed to determine what a credible passing state would need to show and what evidence would be required during reassessment.

From there, ATO Bot can build control specific remediation guidance that includes close actions, likely owners, expected effort, success criteria, evidence collection guidance, and suggested content based on the deficiency that was actually found.

Remediation Artifact Generation

ATO Bot can go beyond telling someone what to fix.

The application can plan and generate draft operational artifacts intended to address identified assessment gaps. Generated artifacts can be created for individual controls or bundled where one document can address several related findings.

The artifact generation process uses the control, finding, project context, closure analysis, and evidence requirements to determine what should be produced.

ATO Bot also includes a separate artifact generation path designed to create material that looks like something an operational team would actually own rather than an assessor writing instructions back to itself.

These are drafts. The system does not treat a generated document as proof that a control has been implemented. The owner still has to validate it, modify it where needed, implement what it describes, and approve it before it can become credible evidence.

Artifact Validation

Generated evidence goes through validation as well.

ATO Bot can check whether an assembled evidence package is intact, whether it can be ingested successfully, whether the content maps back to controls, and whether the evidence can actually be retrieved during assessment.

This provides a way to distinguish between a file that was successfully generated and a package that is actually usable for reassessment.

SSP and Authorization Outputs

The assessment data does not live in a separate world from the authorization package.

ATO Bot includes an SSP Workbench that uses project information, system knowledge, implementation statements, findings, and supporting evidence to build structured SSP content.

The platform can support system overview information, architecture, tooling, roles, implementation oriented narrative, and other SSP content using the same information already collected during the assessment.

ATO Bot also includes OSCAL export, assessment reporting, control rollups, remediation reports, generated artifact packages, and POA&M related data and outputs.

The benefit of using the same underlying assessment state is consistency. The evidence used during assessment, the remediation work, and the information that eventually appears in the authorization package all come from the same data model instead of being recreated independently.

System Knowledge

ATO Bot also builds reusable knowledge about the system being assessed.

The System Knowledge capability analyzes project evidence to identify information about architecture, technologies, tools, and inheritance relationships.

Those assertions and tool records are persisted and can support architecture views, assessment context, SSP composition, and other parts of the application.

This is an area where I am intentionally keeping the distinction between discovered information and authoritative information. The platform can identify likely architecture and tooling information, but an operator reviews those assertions before they are accepted as fact.

AI Assistant

ATO Bot includes assistant and collaboration capabilities that operate against the same project and assessment context.

The assistant is not a separate generic chatbot disconnected from the rest of the application. Conversations, context attachments, and messages are persisted alongside the other project objects, allowing AI assistance to work from the evidence and assessment environment already established in ATO Bot.

Synthetic Evidence and Testing

One of the areas I wanted to address was how to test an AI based assessment system without depending entirely on real authorization packages.

ATO Bot can generate synthetic evidence packages based on project context and target assessment scenarios. It creates a system persona, plans the required evidence bundle, and produces structured artifacts that can be run back through the assessment engine.

Those packages can be reused for demonstrations, development, regression testing, and assessment calibration.

Assessment Calibration

The calibration harness compares expected control outcomes against what the assessment engine actually produced.

It records mismatches, identifies different types of assessment drift, and maintains performance information across repeated runs.

This provides an engineering mechanism for testing changes to models, prompts, assessment logic, or evidence processing instead of assuming that an AI based assessment will continue behaving the same way as the platform changes.

Administration and Runtime Management

ATO Bot also includes the parts needed to operate the platform rather than just demonstrate it.

There are administrative surfaces for users, security, audit logs, assessment policy, prompts, AI runtime configuration, ingestion configuration, common controls, enterprise libraries, system knowledge, calibration, test datasets, integrations, and the control catalog.

The platform supports role aware APIs, JWT authentication, MFA support, rate limiting, audit logging, and persistent user and security administration.

Model behavior is configurable by runtime purpose. Prompts and model routes can be managed independently instead of treating every AI function as the same model call.

Bounded AI Architecture

There are a lot of AI capabilities in ATO Bot, but the architecture is deliberately bounded.

AI can screen evidence, classify it, reason over assessment objectives, identify contradictions, draft findings, challenge assessments, generate closure questions, develop remediation guidance, create draft artifacts, generate synthetic evidence, and translate structured security information into operator guidance.

Application code controls the scope given to those models, validates the returned data, manages workflow state, applies assessment policy, handles retries, stores results, packages artifacts, and controls what becomes part of the official assessment record.

Human review remains available at the points where judgment or authority is required.

That separation is one of the core ideas behind the project.

Platform Architecture

ATO Bot is a full stack application.

The frontend is built with React 18, Vite, Tailwind, React Router, and React Query behind Nginx.

The backend uses FastAPI, SQLAlchemy, PostgreSQL, pgvector, Redis, background workers, persistent file storage, and multiple model provider routes.

PostgreSQL stores the workflow and assessment state. pgvector supports semantic retrieval across the prepared evidence base. Background workers handle long running ingestion, assessment, remediation, package generation, and testing jobs so those processes are not tied to a browser session.

The application has dedicated API and service layers for assessment, evidence, remediation, closure, artifacts, SSP composition, system knowledge, integrations, reporting, prompt management, AI runtime management, common controls, enterprise evidence, and other operational functions.