Red Hat AI for Intelligent Tax Administration

Tax administration has a shape that makes it unusually well suited to language models, and unusually unforgiving of the way they are normally deployed.

That last constraint is the one that decides the architecture. Choosing a model is a six-month decision. Choosing where models run, how they are governed and how they scale is a ten-year one.

Two rules of thumb come out of that table.

The platform is four horizontal layers with two concerns running vertically through all of them. Each layer is noted with the rung of the ladder — data, information, knowledge, judgement — that it serves.

The point of drawing the two concerns vertically is that neither is a layer you add at the end. A project that treats compliance as a phase after deployment discovers, at the worst possible moment, that it cannot produce evidence for anything that already happened.

Take the question a facilitator asks in Lab 01: "Find the missing document for this contact." Four things happen, and only four.

The demystifying sentence, worth saying out loud in the session: when Lab 01 returns missing_doc: Form 8822, that answer was not fetched from any system. It is the most probable continuation of the contact text under those weights — nothing else happened, and nothing else needed to.

The question underneath most audience confusion is: "do I need to do anything to get that response?" The answer is no — sending the prompt is the request. There are exactly three places an answer can draw from:

The prompt and any retrieved passages share the context window — the model’s short-term memory, cleared after every request.

Three reassurances follow directly from this mechanism, and they are worth stating explicitly to any audience handling taxpayer data:

Bring up a large language model on a single GPU host and use it to classify a real taxpayer inquiry — with no inference call leaving the machine.

This lab exists to answer one question directly, because it is the question that decides whether anything else in this guide is relevant to a tax agency: where does our data go when someone prompts the model?

The serving engine is the Red Hat AI Inference Server: the vLLM engine, packaged and supported as a signed container that runs anywhere Red Hat Enterprise Linux runs. (Earlier versions of this lab used the retired InstructLab ilab CLI; the endpoint it exposes — an OpenAI-compatible API on localhost — is unchanged.)

On the ladder this lab is the data → information step: an uninterpreted taxpayer contact goes in, and inference on hardware you control turns it into a classification, a routing queue and a draft for one case — the data never moves, which is exactly why where it lives decides the architecture.

For LAB_MODE=simulate, nothing. For LAB_MODE=live:

The model returns a category, a routing queue, the missing document and a draft reply. It does not state a determination as final — that behaviour comes from the system prompt in assets/inquiry.json, which is worth reading before you change it.

Latency on a single H100 for an 8B model at this prompt length lands under two seconds. If yours is much slower, check that the model actually loaded onto the accelerator rather than falling back to CPU.

That last question is Lab 02 — Teach it your notice taxonomy.

The base model from Lab 01 — Serve a model inside your boundary does not know your agency’s notice codes, your routing queues or how your correspondence is meant to read. Add that knowledge as seed YAML a subject matter expert can write and a reviewer can approve, generate synthetic training data from it with the SDG Hub library, fine-tune with Training Hub, and prove the result is better than what you started with.

On the ladder this lab produces knowledge: the seed YAML is guidance the institution owns — notice families, routing rules and house style, written by the people who apply them and reviewed like code — and tuning carries that institutional memory into the model so it informs every outcome.

Domain adaptation is usually where AI programmes stall. The conventional path needs a labelled dataset, which needs annotators, which needs a budget and six months. The seed-example approach inverts it: a handful of examples written by the people who already own the knowledge become thousands of training samples, and the contribution is reviewed like code. (This workflow began life in InstructLab; the retired ilab CLI is gone, and its ideas carry forward as the composable sdg_hub and training_hub Python libraries.)

The file at labs/lab02-customization/taxonomy/knowledge/gov/tax/notices/qna.yaml is worth reading in full. It is deliberately written the way a real contribution would be — plain language, one notice family per context block, and a house-style section that captures the rules a correspondence reviewer applies by instinct.

For live mode: a GPU host, Python with the sdg-hub and training-hub libraries, the Lab 01 — Serve a model inside your boundary endpoint running as the generating model (the "teacher" in SDG Hub’s terms — the model that writes the new examples), and several hours of GPU time for the training step.

The evaluation table shows meaningful gains on agency-specific tasks — notice family accuracy, correct routing queue, citing the right form — and a smaller gain on general readability. That pattern is the healthy one. If general benchmarks move a lot, you have probably overfitted to the seed examples.

One GPU host is a pilot. Filing season is a capacity event. Deploy the tuned model from Lab 02 — Teach it your notice taxonomy with KServe and vLLM, then run a load rehearsal and watch the platform scale from two replicas to twenty and back down again — with no application change.

On the ladder this lab serves information at scale: every request is one classification for one case, and the platform’s job is to keep producing that information at filing-season volume without changing what the data is, where it lives or who signs the result.

The single most important file in this lab is labs/lab03-openshift-ai/manifests/inferenceservice-tax-classifier.yaml. Read it before you run anything. Four decisions in it are worth arguing about with your own team:

For live mode: an OpenShift cluster with Red Hat OpenShift AI installed, GPU nodes with the NVIDIA GPU Operator, oc authenticated, and hey for the load test.

Under 400 concurrent requests the service holds p95 latency under a second, returns no errors, and reaches its replica ceiling within about 40 seconds. Afterwards it returns to minReplicas once the scale-down delay expires.

The behaviour to watch for is the shape, not the absolute numbers: fast scale-up, stable latency through the ramp, and a return to floor that does not happen so fast it re-triggers.

A model that sounds confident and cites nothing is a liability in a tax agency. Put your published guidance behind the model with retrieval, so every answer returns the passage it came from — and so a question the corpus cannot answer produces a referral rather than a guess.

On the ladder this lab is knowledge: published guidance is the institution’s memory, and retrieval makes it retrievable and cited, so every answer informs an outcome with the passage it came from — or refuses and refers to a person when the memory does not cover the question.

Most retrieval demos show the good answer. This lab spends half its time on the bad question, because that is the behaviour that decides whether a governance board will let this near a taxpayer.

labs/lab04-rag/query.py carries a RELEVANCE_THRESHOLD of 0.75. Below it the script refuses, recommends referral to a specialist, and exits with code 3 so a calling workflow can route the contact to a human. A retrieval system that always answers is worse than no retrieval system, because it launders a guess into something that looks sourced.

Step 5 prints the retrieval trace, showing every rejected candidate and its score. Show that screen to your assessors. It turns "the model refused" into "here is exactly why, and here is the number you can change."

For live mode: an OpenShift cluster, psycopg installed, an embeddings endpoint (the one from Lab 01 — Serve a model inside your boundary or Lab 03 — Survive filing season), and a corpus of your own guidance in labs/lab04-rag/corpus/ as .txt, .md, .adoc or .html.

Most of the effort in production AI is not the model. It is provisioning, patching, scaling and evidence collection. Wire a health check that tests the endpoint the way a taxpayer would, then an Event-Driven Ansible rulebook that remediates saturation before anyone opens a ticket.

On the ladder this lab executes judgement: the rulebook carries out decisions the platform team already made — which conditions to remediate and which to escalate — and it never makes a new one, which is exactly why the provenance rule pages a person instead of redeploying.

labs/lab05-ansible/rulebooks/inference_saturation.yml maps four conditions to four job templates. There is no separate runbook document to go stale, because this file is the runbook and it executes.

One rule in it is deliberately different from the others. When the served model digest stops matching the approved change record, the rulebook escalates rather than self-remediating:

Automatically redeploying over a provenance mismatch would destroy the evidence of what was actually running. That is a decision worth making explicitly, and worth defending in a design review.

The health check playbook does four things, and the fourth is the one that usually gets left out: it asserts that the model actually being served matches the digest in the approved change record. A model that has drifted from the approved artifact is an accreditation finding, not an incident, and it will be found by an assessor if it is not found by automation first.

For live mode: Red Hat Ansible Automation Platform 2.4 or later, ansible-navigator and ansible-rulebook on the path, the ansible.eda collection, an Automation Controller with the named job templates, and Alertmanager configured to post to the rulebook source.

The alert-to-resolved interval is under two minutes and no human touched it. More importantly, the awx job list output at the end is a complete record of what changed, when, and what launched it — assembled continuously rather than reconstructed at the end of a quarter.

That job list is the evidence Lab 06 — Prove it to your ISSO depends on. Automation and compliance are the same investment described from two directions.

Walk the exact checks an authorizing official will run against an AI workload: is this the model we approved, is the cryptography validated, is the platform still hardened this quarter, and is the model behaving the way it did at accreditation.

Every one of these is a platform feature you switch on. None of them is a project you fund.

On the ladder this lab runs evidence → judgement: every check produces information — a signature result, a FIPS state, a scan finding, a drift figure for this workload — and the authorizing official’s decision to accept that evidence is judgement, a determination a person signs and no platform feature makes for them.

For live mode: oc authenticated to a cluster with the Compliance Operator and TrustyAI installed, cosign on the path, the agency signing public key, and SSH access to a GPU node.

Step 3 returns NON-COMPLIANT for the ocp4-moderate profile, and step 4 shows that exactly one of 217 checks failed: audit log forwarding.

A demo where every scan passes teaches the wrong lesson. Real clusters have findings, and the useful skill is reading a result and deciding what to do about it. There are two ways to fix this one — apply the remediation directly, or let the Lab 05 — Automate the toil around the model rulebook apply it, which produces the same fix plus a job record with a launching identity. The second is the one your assessor wants.

The TrustyAI output shows channel_mix drift at 0.212 against a 0.150 threshold. The instinct is to conclude the model is failing. It is not.

Input distribution shifted because more taxpayers moved to the portal. Nothing is wrong with the model at all — drift monitoring is telling you that the world changed, which is exactly what you want it to tell you. The response is to investigate the input, decide whether the accreditation baseline is still representative, and re-baseline deliberately if it is not.

Confusing "the inputs changed" with "the model broke" is the most common misreading of these metrics, and it leads teams to retrain when they should have been asking a business question.

labs/lab06-trust/manifests/trustyai-service.yaml defines three Prometheus rules alongside the TrustyAI service:

Aim the Lab 04 — Ground answers in your own guidance retrieval pattern at operations itself. Cluster logs, runbooks, incident history and change records become a private notebook you can question during a live incident — a NotebookLM for the platform, running on the platform. Then read the numbers: the payoff is measured in mean time to restore, because diagnosis is where the hours live.

On the ladder this lab climbs from information to knowledge: the log line is data, the cited diagnosis is information about one incident, and the runbooks, postmortems and change records it draws on are the institution’s memory — a generated summary joins that memory only after a person reviews it.

When something breaks at 2 a.m., the answer usually already exists — in a runbook, a past incident’s postmortem, a change record from that afternoon, or a log line from the failing pod. What costs hours is that those four things live in four systems and three people’s heads. A notebook grounded in all of them answers the on-call question — what changed, has this happened before, which runbook applies — in seconds, with citations the engineer can check before acting.

Two properties make this defensible in a tax agency:

For live mode: an OpenShift cluster with OpenShift Logging (Loki), the Lab 04 — Ground answers in your own guidance vector store and model serving in place, logcli, and Python for the ingest and reporting helpers.

The diagnosis names a specific change record, a specific precedent incident and a specific runbook section, each with a retrieval score — and the on-call engineer can open all three before acting. If answers come back without citations, or citations that do not support the claim, treat that exactly as you would in Lab 04 — Ground answers in your own guidance: the retrieval threshold or the corpus needs work, and the model’s fluency is not a substitute.

The representative MTTR decomposition:

Mainframe and server logs come in, a predictive model scores which systems will fall over next, Ansible Lightspeed drafts the rulebook and playbook, a person reviews and merges them, Event-Driven Ansible runs them, and the assistant answers "how is server8-audits?" by probing the estate. One platform carries every step, and the loop is also the knowledge base.

On the ladder this lab climbs every rung in order. The log lines are data; the outage-risk score is information about one system; the drafts, the review thread and the job outcome become knowledge the moment they are merged and indexed; and the merge itself is judgement, a determination a named reviewer signs. Automation executes that judgement. It never makes it.

This is the lab that ties the other seven together: the tabular model from Lab 02 — Teach it your notice taxonomy, model serving from Lab 03 — Survive filing season, retrieval from Lab 04 — Ground answers in your own guidance, the rulebook and job history from Lab 05 — Automate the toil around the model, and the operations corpus from Lab 07 — Ask your own logs, on the one platform Lab 01 — Serve a model inside your boundary and Lab 06 — Prove it to your ISSO keep inside the boundary.

For live mode: Red Hat OpenShift AI with a KServe sklearn runtime for the outage-risk model and vLLM for Granite; OpenShift Logging with Vector receiving z/OS SYSLOG beside journald and container logs into Loki; Red Hat Ansible Automation Platform 2.5 with Ansible Lightspeed on-prem, its model served through OpenShift AI model serving; an Event-Driven Ansible controller with the named job templates; an ITSM webhook for the ticket; and oc, logcli, python3, ansible-rulebook, ansible-lint, git, gh and awx on the path.

As with Lab 07 — Ask your own logs, the helpers the script calls (ingest_estate.py, score.py, lightspeed_draft.py and ask.py) are representative wrappers and are not shipped in the lab directory. What is shipped beside run.sh is the Lightspeed prompt (prompts/memory-pressure.txt) and the reviewed, post-merge rulebook and playbook (rulebooks/memory_pressure.yml, playbooks/relieve_memory_pressure.yml).

The score is a number with drivers, not a verdict:

The review is visible, and the merge does three jobs at once:

The remediation runs only the approved playbook, and records itself:

The assistant probes through job templates and cites what the loop wrote:

If a score arrives without drivers, a draft arrives without citations, a job runs that nobody merged, or an answer arrives without a job id, that is the finding. Each of those is the same failure the earlier labs guard against, and the loop is only as trustworthy as its weakest gate.

Two different mathematics share one platform. The outage-risk model is tabular prediction: hourly features per system (memory headroom, GC pauses, abend rate, batch overlap) fitted to 180 days of history, so its output is a probability, not an explanation. The drafts come from a code model predicting tokens, steered by your prompt and the runbooks retrieved as context, so its output is plausible YAML that must be reviewed like any other change. Event-Driven Ansible contains no machine learning at all: the same event always runs the same approved job. The knowledge base grows because every promotion (the pull request, its review thread, the job outcome) is embedded into the same vector store the assistant searches, which is why the assistant can cite RB-214 an hour after it was written.

This session is for the IRS audience. Do not forward the invite to other accounts. Ted Craig (Red Hat account executive for the IRS) asked that the focus stay on IRS for this one; other AEs replicate it for their own accounts with Brad and their partners, and it should be duplicated. If someone from outside the IRS turns up anyway, the answer is the same one Brad gives at the close: reach out, and a similar session can be given for their mission.

Owners and deadlines from the September 15 review.

Lab 08 is the finale for an operational room. Eight minutes, the whole loop on slide 17: logs in, risk scored, Lightspeed drafts, a person merges, EDA runs, and the assistant answers with job evidence. Run it in place of the 50 to 56 skim; Lab 01 and Lab 05 stay the live ones. Two takeaways to say out loud: the platform is also the KM system and the documentation, because every promoted change is a cited record; and the assistant launches approved jobs and reports, it changes nothing itself. Then the question to ask the room: what is the last runbook you wrote that the on-call actually found?

Security posture (slide 18) stays the appendix. Brad offered at the September 15 review to move it ahead of the labs and next steps; Ted Craig (Red Hat account executive for the IRS) decided it stays where it is, because cyber will be one or two of the 52 or so at most, not the majority of the room by any stretch. Pull it up only when a cyber person engages, and offer the cyber team a separate, deeper session rather than reshaping this one.

Brad drives; Jon is the color commentator — interrupt, add commentary, call out misspeaks live, and take over if the driver flounders. The number one goal is to entertain: a tuned-out audience learns nothing. Keep it light, and if humour lands on anyone, self-deprecate first.

Five planted questions are rehearsed; the console marks each one PLANTED. On the data-to-judgement slide (5) Jon asks "so is AI taking our jobs?" — the answer is that it is not, because it has no judgement. On the architecture slide (9) Jon plays the foil with "what’s a seam?" (the pants analogy) and Brad asks Jon "what is fine-tuning to you?" — the studying-for-the-test answer. On the why-one-platform slide (11) Jon challenges lock-in ("I thought we already bought the software" — the phone-plan analogy), and on the adoption-path slide (14) Jon asks what a predicament is versus a problem. Plan deliberate pauses to engage the room.

The theme that holds the hour together is the ladder: data → information → knowledge → judgement. Every slide shows the rungs it serves in the top-right spine, and every technology, outcome and lab carries a rung chip in the same four colours (blue, amber, red, green). Use it as the connective tissue: when a product name lands on screen, say which rung it climbs and who climbs the next one. Judgement is always a person; automation executes it.

The full talk track, including the seams, lock-in and MTTR riffs, lives in slides/presenter-notes.html — open it beside the deck and advance both together. Each console step lists the slide’s rungs and an "On the ladder" beat mapping every item on the slide to its rung.

Increase your terminal font size before the session, not during it. Eighty columns is enough for every lab — none of the output wraps at that width.

If you would rather not type at all, open the interactive deck (slides/deck.html) and switch to its Demo view — it plays every lab hands-free with play/pause, per-lab jumps and a speed control, and each lab terminal also has its own Demo button. Opening the page with #demo on the URL starts the Demo view playing on a continuous loop, which doubles as a holding screen before the session begins. For screen sharing, the Present button (or F) fullscreens the 16:9 slide stage.

The interactive deck carries an EN/ES language tab in the sidebar footer — Spanish audiences get the slides, lab briefings, theory panels and glossary translated, while terminal commands and output stay English, as real tools do. Next to it sit a light/dark theme toggle and the Notes button. Every lab carries a theory panel ("what just happened"), and a glossary closes the Labs view — when someone asks where an answer came from, the foundations slides plus the Lab 01 theory panel are the prepared response.

As each command runs, a step card pops up under the terminal (beside it in the Demo view) with three facets: what is happening, why it matters to this talk, and the engineering seam a single Red Hat platform removes — the hand-built box nobody documented, the glue script between alert and fix, the scanner report nobody can diff. A wrap card closes each lab with what was just proved. The cards are the talk track: read the middle facet aloud when the room is mission-minded, the right-hand one when it is procurement-minded. They follow the EN/ES tab like everything else.

Every slide has two faces. Tell is the word-heavy version; Show keeps the central graphic and drops the paragraphs, so you talk to the picture. The toggle sits in the bar under the stage and on S; it is remembered per browser. Present in Show, switch to Tell when someone asks for the detail, and leave the deck in Tell for the take-home copy.

Two keys are deliberately not printed on the deck. N shows the speaker notes on the shared screen: do not press it while sharing. P opens the presenter console (presenter-notes.html) in its own window and keeps it on the current slide or lab as you move; put that window on the presenter’s display only. The console carries the read-aloud script, the discussion notes, the Jon cues, the graphic talking points and the lab step cards side by side.

Two more switches arrived after the September 15 review. The ladder now sits in the corner of every slide as a small stacked pyramid (judgement at the apex, data at the base, the rainbow’s yellow and blue as thin bands between); the rungs a slide serves are lit, the rest dimmed, so the room can track where each slide lives on the climb without a legend. The title slide carries the same pyramid at full size, each layer with the question it answers (what happened; what it means for this case; what we know that bears on it; what we will do and who signs), a rail that says data informs every layer, the end state at the apex (knowing enough to decide) and the rule along its base: solve a problem someone has; don’t invent a solution to one no one has. Walk it bottom to top before anything else. And the slides that have an illustrated graphic (title, data to judgement, where AI lands, five outcomes, architecture, why one platform, labs preview, next steps) carry a small graphic pill; press G, the pill, or the Graphic button under the stage to pop the picture out over the deck with three or four discussion points beside it, then Esc or G again to close. The Labs view opens the "Choose your path" map the same way. The pop-out shows the illustrated render when the file is present in slides/assets/graphics/ and a vector rendition of the same content otherwise.

Say these out loud during the session. They buy more credibility than any demo.

The first response to any proof-of-concept interest is a scheduled show-and-tell: Brad and Jon Keam (or whoever Brad picks) walk the eight labs with the interested team to find where in the stack they want to prove value right now, and to start the community of practice around it. Only if the POC gets serious or ratchets up do we bring in more resources (Dan and company).

Route the request through the account executive. For the IRS that is Ted Craig (Carahsoft), who is on the call. Anyone from outside the IRS, or anyone who does not know who their AE is, emails Brad and gets connected to their equivalent of Ted, and the session is replicated with their own account team.

56 registered: 52 via Ted Craig and Caleb’s Carahsoft outreach, 4 via Four Inc. Most are contacts we have not been talking to before, which is the point. Carahsoft sends the full contact list to Ted Craig (Red Hat account executive for the IRS) at the end; Caleb (Red Hat inside sales) runs the follow-up campaign with the recording, the deck and the labs link, plus any document links worth attaching. Brad’s action: hand Caleb the deck and the links before the campaign goes out.