Learn to build AI applications people can rely on

A model name rarely tells you the whole story. Is it built for chat or for embeddings? Can it process images? And if a client asks to run it entirely on their own servers, does the license allow that, and who looks after updates, security, and downtime? You'll learn to read model documentation with questions like these in mind. We'll compare managed APIs, hosted open-weight models, and local deployment, so you can spot a poor fit, or a responsibility nobody has claimed, before you've built an application around it.

Core topics

• Model categories: base, instruction/chat, task-specific, tuned, distilled, and reasoning variants; text, embeddings, vision, and audio.
• Model cards: intended uses, input/output contracts, limits, versions, and lifecycle.
• Why public benchmark results may not transfer to your particular workload.
• Closed/API-only, open-weight, and open-source models, including what each term does and doesn't imply.
• Commercial permissions, license restrictions, and acceptable-use conditions.
• Provider retention policies, privacy, data residency, and vendor dependency.
• Self-hosting fundamentals: CPU/GPU resources, VRAM, quantization, inference runtimes, and who owns operations.

Practice: Write a small script that builds a model inventory for three models from their cards or API listings, and send one test request to each. Then compare a managed model API, a hosted open-weight model, and a locally run model for one client scenario, and document the licensing questions and risks that shape your recommendation.

Before comparing models, we have to describe the job properly. A multilingual customer-support assistant needs different abilities from a vision-based invoice extractor. Some requirements, such as keeping data within an approved boundary or returning a required schema, aren't negotiable. This unit helps you turn a client's request into a model-selection checklist that separates must-haves from useful extras.

Core topics

• Gathering stakeholder requirements and turning them into a task specification.
• Input/output modalities, tools, structured output, reasoning, and language capabilities.
• Quality targets, context needs, throughput, latency, availability, and cost ceilings.
• Data handling, privacy, accessibility, and regional or compliance constraints.
• Weighted comparisons, hard exclusions, and stable versus preview model lifecycles.

Practice: Write a model-capability matrix for a realistic client brief before naming any candidates, in a form a script can read. Run it against your model inventory and show which requirements would automatically disqualify a model, even if its benchmark scores were impressive.

Providers often offer APIs that look similar until you depend on a feature that works differently. One may handle streaming or tool calls in a different way; another may return a different error or lack the output contract your application needs. We'll put models behind a common interface, while keeping those differences visible rather than pretending they don't exist.

Core topics

• The role of OpenAI Responses, Anthropic Messages, Gemini, Hugging Face inference, and compatible APIs as implementation examples.
• Model configuration, request/response handling, and provider-specific failures.
• Differences in tool calling, streaming, reasoning controls, and structured outputs.
• Syntax compatibility versus actual capability and behavior compatibility.
• Adapters, provider abstractions, capability maps, fallbacks, and routing boundaries.

Practice: Connect two providers, or one provider and a local/open model, through a shared application interface. Test adapter behavior and demonstrate a fallback that does not silently remove a required capability.

A cheaper model looks attractive until it starts failing on the questions that matter most. A more powerful model may improve accuracy, but not enough to justify its response time or running cost. In this unit you'll compare candidates using the same representative tasks, then look at the results from several angles. The aim is to make model selection less about preference and more about evidence.

Core topics

• Representative evaluation sets and task-specific quality measures.
• Structured-output reliability, tool-call correctness, and qualitative failure analysis.
• Time to first token, total latency, concurrency, throughput, and error rates.
• Cost per request, workflow, or user, with explicit pricing and volume assumptions.
• Quality/latency/cost tradeoffs, Pareto comparisons, availability, and uncertainty in estimates.

Practice: Test three models on one workload, then produce a reproducible comparison report. Explain which model meets the requirements and what you give up by choosing it.

Imagine your primary model becomes unavailable in the middle of a busy day. The application needs a sensible response, but sending every request to an incompatible fallback could make matters worse. We'll build routing and failure handling around the requirements you've already defined, and we'll record which model handled each request so decisions remain traceable.

Core topics

• Fixed-model, task-based, capability, cost, and latency routing.
• Model cascades, degraded service, and controlled fallback behavior.
• Availability checks, version pinning, deprecations, and migration planning.
• Serving an open model versus buying managed inference.
• Tracing selected models and understanding the quality, privacy, and cost of each route.
• A cost estimate a client can read: its assumptions, a range, and what would change it.

Practice: Add routing to the Part 1 application and deliberately take one provider out of service. Show that the application either switches to a compatible model or fails safely, with a clear trace of what happened. Then write the cost estimate you would hand to the client.

An embedding turns a piece of information into a sequence of numbers. Those numbers make certain kinds of comparison possible, but a similarity score isn't the same as a probability that a result is correct. We'll explore what close and distant vectors mean, and why different distance calculations or input choices can change the results. You'll also see examples where semantic similarity leads search astray.

Core topics

• Embedding spaces, vector dimensions, and dense versus sparse representations.
• Cosine similarity, dot product/inner product, Euclidean (L2) distance, and normalization.
• Metric direction, nearest-neighbor ranking, and the limits of similarity scores.
• Domain-specific and multilingual behavior, semantic counterexamples, and input truncation.
• How vector representations support retrieval, while leaving room for lexical and structured search.

Practice: Inspect nearest neighbors for a small dataset. Check score ordering by hand, run metric tests, and collect examples of high-scoring but unhelpful matches.

Not every embedding model interprets text in the same way. Some distinguish between a short search query and the longer document it should find. Others use different dimensions, input limits, or model versions. Mixing those choices without care can quietly damage retrieval quality. Here you'll define the rules your indexing and search pipelines must follow together.

Core topics

• Proprietary and open encoding options, including OpenAI, Gemini, Cohere, Sentence Transformers, Hugging Face, and Jina as examples.
• Query/document encoding roles and asymmetric retrieval where supported.
• Input limits, task settings, vector dimensions, batching, latency, and pricing.
• Multilingual and domain fit, licenses, local inference, and version tracking.
• Incompatible embedding spaces, re-embedding, and supported dimension reduction.

Practice: Test two embedding models on labeled business queries. Produce a reproducible encoding manifest and explain how you would replace a model without corrupting the existing index.

If a search system examines every vector, it can find an exact nearest neighbor, but doing that gets expensive as the dataset grows. Approximate indexes speed things up by accepting a carefully measured tradeoff. You'll build both approaches and compare them, so that "fast search" has a meaning beyond a reassuring demo. Then there's the question of where the vectors should live. A production service needs persistence, filtering, backups, and a plan for updates, so we'll work in PostgreSQL with pgvector and look at when another store would suit a client better.

Core topics

• Exact nearest-neighbor search and approximate nearest-neighbor search (ANN).
• HNSW and IVF as key indexing concepts, without turning the course into index research.
• Top-k, score thresholds, index tuning, and filter effects.
• Recall versus latency, with exact search as a reference baseline.
• Collections, record IDs, metadata, upserts, deletes, and consistency tradeoffs.
• Persistence, tenant boundaries, backup and recovery, and managed versus self-hosted cost.
• PostgreSQL with pgvector as the course's store, with other vector databases as comparison reading.

Practice: Implement exact and indexed search over the same records in PostgreSQL with pgvector. Measure the recall and latency differences, document the cases where the approximate index missed a result, and write a short decision record on whether this store fits the client's workload.

A search index is only useful while it reflects the information people are allowed to see. Documents get revised. Records are removed. Imports fail halfway through. We'll build an ingestion pipeline that copes with all three, keeping stable IDs and enough version information to repair or rebuild the data when something changes.

Core topics

• Source rights, normalization, deterministic IDs, deduplication, and metadata schemas.
• Batch embeddings, queues, transient failures, retry policies, and partial success.
• Idempotent re-ingestion, ingestion manifests, and reconciliation jobs.
• Document and embedding versioning, reindexing, freshness, and stale-data detection.
• Updates, deletes, tombstones, rollback, and recovery procedures.
• Pulling documents and their permissions from the system where they live, and keeping permission changes in step.

Practice: Import a real corpus, including one source pulled through a connector with its permissions. Then change and delete records while deliberately interrupting processing. Demonstrate that retrying does not create duplicates, and that deleted records and withdrawn permissions no longer appear in search.

Two users can ask the same question and be entitled to different results. That makes search permissions part of correctness, not a detail you add later. You'll expose your search pipeline through an API and interface, then check both relevance and access. We'll also look at why empty results, exact identifiers, and awkward queries deserve their own tests.

Core topics

• Authorization before retrieval and per-tenant or per-record access filters.
• Query handling, top-k, no-result behavior, document previews, and exact identifier pitfalls.
• Labeled queries, relevance judgments, and segment-based testing.
• Recall@K, reciprocal rank/MRR, nDCG awareness, latency, and query cost.
• Privacy-aware search logs, evaluation reproducibility, and safe client-facing responses.
• The data-rights questions to settle with a client before indexing: ownership, permitted use, and retention.

Practice: Deliver a searchable API and small interface. Run the relevance tests and try to retrieve another tenant's hidden records. Explain both the measured search quality and the access-control results.

Before adding RAG to a product, it's worth asking a simple question: does this problem actually need it? A customer might be better served by a search page, a database query, or carefully supplied context. We'll compare those options and build a small RAG baseline when it makes sense. Starting simple gives us something we can measure, and it makes later improvements easier to justify.

Core topics

• Choosing between RAG, direct context, SQL, search-only, and fine-tuning.
• Offline document ingestion versus retrieval at request time.
• Two-step retrieval-and-answering pipelines versus agentic RAG.
• Suitability criteria and when to recommend no RAG at all.
• Baseline quality measurements and common failure categories.
• Recognizing a question that belongs to a database query, not to document search.

Practice: Write a short architecture decision, build a basic retrieval-and-answer pipeline, and compare it with at least one approach that doesn't use RAG. Include one kind of question where a SQL query is the right design.

Real documents aren't always tidy. A PDF might have a table that extracts badly, a scanned page may lose important words, and two versions of a policy might disagree. If we ignore that mess, search quality will suffer before the model even sees a question. You'll learn to prepare documents, split them into useful passages, and keep the source details needed to check an answer later.

Core topics

• Source authority, provenance, access permissions, and document identity.
• Extracting and verifying content from PDFs, HTML, tables, and scanned material.
• Fixed, recursive, structural, and semantic chunking strategies.
• Overlap, parent/neighbor relationships, and chunk-size tradeoffs.
• Page anchors, revision history, and consistent metadata schemas.
• Measuring extraction quality against known answers, field by field and table by table.

Practice: Build a reproducible document-processing pipeline, compare chunking methods, score extracted fields and tables against known answers, and trace an answer back to its source document.

Try searching for a policy by its exact reference number. A vector search system may find related material but miss the identifier itself. Keyword search often helps with that, while vector search can be better at finding meaning. In this unit we'll compare the approaches, combine them when useful, and test whether reranking is worth the extra time and expense. The results, rather than the technology's reputation, should guide the choice.

Core topics

• BM25 and exact-term matching.
• Dense vector search versus sparse or lexical approaches.
• Hybrid search, rank fusion, multiple indexes, and deduplication.
• Candidate budgets, metadata filters, and freshness rules.
• Reranking, query rewriting, and contextual retrieval, including when their added cost is justified.

Practice: Compare vector-only, BM25-only, hybrid, and reranked retrieval on held-out questions. Use the results to justify your choice.

Finding the right passage is only half the job. The assistant still has to use it correctly. It might overlook an exception, combine details from two different policies, or cite a page that doesn't support the claim. We'll work on the connection between evidence and answers, including how to verify citations and how to respond when information is missing. A useful assistant should be able to say when the documents don't support an answer.

Core topics

• Selecting evidence within a token budget.
• Keeping retrieved documents separate from trusted instructions.
• Grounded prompting and structured answer formats.
• Citations linked to actual passages, pages, and document revisions.
• Checking claim support rather than merely displaying references.
• Uncertainty, no-answer behavior, contradictory policies, and outdated evidence.

Practice: Build an answer-and-citation verifier. Test unanswered questions and conflicting sources, and make sure missing evidence never turns into a made-up citation.

Frameworks can make RAG easier to assemble, but they can also hide what's happening between a question and an answer. You'll build a small pipeline directly first, then try the same task with a framework. We'll look at the tradeoffs together: how easy each version is to inspect, test, change, and maintain. The aim is to choose a tool because it helps your project, not because every example online uses it.

Core topics

• Building a small RAG pipeline directly with a provider SDK.
• LangChain as the course's framework for the comparison, with LlamaIndex, Haystack, and RAGFlow as alternatives.
• Clear interfaces for retrieval, reranking, and generation.
• Debugging across framework boundaries and managing framework changes.

Practice: Rebuild one piece of the pipeline with LangChain and compare it with the direct implementation for behavior, clarity, and maintenance.

A knowledge assistant needs to keep up with the company it serves. Policies change, access permissions are withdrawn, and sometimes a source document contains text designed to manipulate the model. We'll explore what can go wrong when the information around a RAG system changes. Then you'll test updates, revocations, failures, and recovery so that answers and citations respect the current sources and permissions.

Core topics

• Prompt injection through retrieved content and intentionally poisoned documents.
• Source-level permissions and cached citation checks.
• Updates, deletions, index migrations, and safe degraded behavior.
• Measuring retrieval/generation latency and cost.
• User feedback, operating procedures, and incident documentation.
• Demonstrating the assistant to a client, including what it cannot do.

Practice: Update a source, revoke access, and simulate an outage. Confirm that outdated or unauthorized information does not leak through answers, context, citations, or caches.

A customer asking for a refund doesn't necessarily need an autonomous agent. If the steps are known and the rules are clear, a conventional workflow may be easier to test and safer to operate. We'll examine what makes an agent useful, what it actually observes and controls, and when simpler software is the better choice. You'll be expected to explain that decision before writing an agent loop.

Core topics

• Ordinary application code, RAG, deterministic workflows, and agent-based approaches.
• Goals, observations, available tools, stop conditions, and bounded autonomy.
• ReAct as a constrained cycle of reasoning, tool choice, execution, and feedback.
• Task decomposition, delegation, step budgets, and escalation.
• Single-agent versus multi-agent approaches, including why more agents may add risk.

Practice: Solve one business task with an explicit workflow and a bounded tool-using loop. Compare predictability, correctness, cost, and safety, then recommend which design should ship.

Suppose an assistant prepares a request to update a customer account. Before anything is written, the application needs to know who's asking, which account they can access, and whether approval is required. Tool descriptions and friendly prompts are not enough to enforce those rules. Here you'll define the contract for a tool and protect the actual execution path. You'll also build a tool that answers plain-English questions from a database, where a wrong number that looks right is worse than no answer.

Core topics

• Typed function/JSON schemas, argument validation, and strict server-side enforcement.
• Read, write, and irreversible actions, with risk-based approval requirements.
• Delegated identities, least privilege, tenant scope, and action-specific authorization.
• Approval records tied to exact parameters, with action receipts and audit trails.
• Duplicate prevention, idempotency, retries, and partial failure handling.
• A read-only question tool over a database: a restricted role, query validation, row limits, and tenant scope.
• Turning a plain-English question into a query, checking it, running it, and testing it against questions with known answers.

Practice: Build an approved ticket-update tool. Send malformed arguments, another tenant's identifier, and repeated requests. Prove that unauthorized writes fail and retries cannot create duplicate business actions. Then build a read-only tool that answers plain-English questions from a reporting database, shows its query, and is tested against questions with known answers.

Workflows don't always finish in one request. An approval might arrive hours later, a network call might fail, or a server might restart after an action has been prepared. You'll work with state machines and checkpoints so the application knows what happened before the interruption. This is where a useful prototype starts becoming a dependable business process.

Core topics

• Chaining, routing, parallel work, and joining results safely.
• Planner/executor and evaluator loops as design options.
• State machines, step ownership, persisted state, and checkpoints.
• Pause/resume, cancellation, compensation, and controlled escalation.
• Loop and cost budgets, concurrency, crash recovery, and avoiding repeated side effects.

Practice: Force a crash between planning and executing a business action. Restart the workflow and demonstrate one correct outcome, with a trace that explains how it recovered.

Agent frameworks can save time, especially as a workflow grows. They can also hide details you need to understand when something breaks. We'll build a bounded tool loop ourselves first, then compare it with one maintained framework. You'll see where a framework helps and where plain application code gives you better visibility and control. You'll also connect your workflow to a system outside your own application, which is what most client work asks for.

Core topics

• Manual tool-use loops and the underlying messages, actions, and state transitions.
• LangGraph as the course's agent framework, with OpenAI Agents SDK, Claude Agent SDK, Google ADK, and smolagents as alternatives.
• Vercel AI SDK as an option for a JavaScript/TypeScript implementation.
• Web-search attribution, untrusted tool outputs, and execution boundaries for code or computer-use tools.
• Framework overhead, observability, permissions, and justification for multi-agent designs.
• Connecting to another company's system: delegated sign-in with OAuth, scopes, and token storage.
• Webhooks: verifying signatures, handling repeats, and delivering a result by email or chat.

Practice: Implement the same small agent workflow without a framework and with one framework. Compare behavior, complexity, failure traces, and what you would need to maintain each version. Then read a supplied application's code and tests, connect one action to an outside system's test account, and deliver a result to a controlled email or chat recipient. Show that an expired token or a repeated webhook causes no harm.

You may want your AI application to use a tool built by a different team or service. MCP provides a way for clients to discover tools and resources from servers, but finding a tool is not permission to use it. We'll look at how the pieces communicate, what is trusted, and where authorization must happen. You'll also learn to check protocol and SDK versions rather than assuming every example implements the same revision.

Core topics

• MCP host, client, and server roles; tools, resources, and prompts.
• Capability discovery, request/response concepts, JSON-RPC, and protocol lifecycle.
• Local stdio and remote HTTP transports.
• Authentication, authorization, trust boundaries, and tool-result safety.
• Protocol revisions, SDK compatibility, and testing interoperability across versions.

Practice: Run a local MCP server, inspect its advertised capabilities, and test discovery and a permissioned resource request. Explain which component makes the final access decision.

Knowing the protocol is helpful; running a secure integration is where the engineering work begins. You'll publish a small set of clearly defined tools and resources from an MCP server, then connect to them from an independent client. We'll test how the server responds to bad arguments, unauthorized access, unexpected tool content, and incompatible clients.

Core topics

• Typed tool contracts and registered resources with clear descriptions.
• Independent client integration, discovery, execution, and error responses.
• Version compatibility and handling unsupported capabilities.
• Local versus remote deployment, network exposure, allowlists, and scopes.
• Malicious tool names or content, unsafe token forwarding, tenant boundaries, and execution-time authorization.

Practice: Build an MCP server with read-only account information and approved ticket actions. Connect an independent client and demonstrate that disallowed resources and actions are rejected by the server.

An agent can make a perfectly valid tool call and still fail the customer's real task. That's why we'll test end results as well as the individual steps. You'll connect your workflow to the RAG application, examine its traces, and rehearse failures involving approvals, unavailable tools, and interrupted requests. The final question is whether someone else could operate it safely.

Core topics

• Business outcome success versus tool-call or intermediate-step correctness.
• Action traces, authorization violations, approval delays, and retry behavior.
• Latency, step budgets, cost, and failure rates.
• Human approval and escalation testing, including delayed decisions.
• Provider/tool outages, incident runbooks, reproducible tasks, and operational handoff.
• Responding to a client's change request: its effect on scope, cost, and tests.

Practice: Run successful, failing, and unauthorized workflows end to end. Demonstrate the approval history, recovery behavior, operating metrics, and evidence that sensitive actions did not happen without permission.

To protect a system, it helps to understand how information and permissions move through it. Who can upload a document? Where does a model see that document? Which tools can change customer records? You'll draw the application's trust boundaries and work through the ways an attacker or an ordinary mistake could cross them. That gives security work a sensible starting order.

Core topics

• Assets, users, identities, components, entry points, and data flows.
• Trust boundaries between the browser, backend, models, retrieval, tools, and storage.
• Common web security risks and OWASP GenAI/LLM risk families.
• Likelihood, impact, affected people, and prioritizing threats.
• Risk registers, ownership, mitigation tests, residual risks, and acceptance decisions.

Practice: Diagram the complete Client Knowledge & Operations Assistant, identify plausible attacks, and write a prioritized risk register with controls you can actually test.

An AI assistant may read a web page or document that contains instructions intended for the model rather than for the human reader. If it treats that content as an authority, it can be tricked into ignoring the real task. We'll create controlled attacks using retrieved documents and tool responses, then test defenses that don't depend on the model always recognizing a trap.

Core topics

• Direct prompt manipulation versus indirect prompt injection.
• Untrusted instructions in documents, websites, files, tool outputs, and memory.
• Instruction-versus-data boundaries, source trust, and prompt laundering.
• Poisoned corpora, chained tool abuse, and attacks across retrieval and agents.
• Layered defenses, runtime permissions, controlled red-team cases, and regression tests.

Practice: Place malicious instructions in sample source text and tool results. Show that the assistant cannot use them to bypass protected actions or data boundaries, even when the model's response is imperfect.

Consider two companies using the same knowledge assistant. They may ask similar questions, but their documents and customer records must stay separate. We'll test that separation across search, generated answers, tool calls, caches, and logs. You'll also work through what happens when a user asks for their information to be deleted or when an API key needs to be replaced.

Core topics

• Authentication versus authorization, tenant/resource access lists, and least privilege.
• Delegated action permissions and approval tied to the exact requested operation.
• Secure credential storage, rotation, and controlled access.
• Data minimization, redacted logs, retention, deletion, and privacy-by-design practices.
• Provider data flows, regional storage, licensing, and vendor due diligence.

Practice: Attempt cross-tenant access through different application paths, then exercise a deletion request. Produce negative tests, an access matrix, and a record of where user data is sent.

Not every harmful result looks like a security breach. An assistant might produce unsafe content, render hostile Markdown, or act on an ambiguous request that should have been reviewed first. You'll design clear rules for blocking, asking for clarification, escalating, or allowing an action. We'll also look at where moderation models help and where ordinary application checks remain essential.

Core topics

• Input validation, output sanitization, safe Markdown, and controlled file/code execution.
• Moderation services and the limits of automated content classification.
• Business content policies, refusals, appeals, and escalation paths.
• Approval for consequential actions and risk-tiered review thresholds.
• Abuse controls, request budgets, and helpful error messages when an action is blocked.

Practice: Add policy checks and approval flows to the application. Run cases that should pass, fail, or require review, and explain the decision shown to the user.

Two users may ask the same question in different language or circumstances. Is the system treating them consistently? And if it gives an answer that may affect someone's decision, have you made its uncertainty and evidence clear? This unit introduces practical ways to think about those questions, alongside consent, accessibility, and rights to use the content in a system.

Core topics

• Stakeholders, possible harms, and fairness questions relevant to the application.
• Comparing behavior across user scenarios or groups when lawful and appropriate.
• Accessibility, uncertainty notices, user disclosures, and the limits of generated answers.
• Document provenance, consent, licensing, copyright, and data-use rights.
• Impact assessments, specialist review, and communicating limitations accurately.
• Telling people they are talking to an AI, and marking AI-generated content, with current transparency rules as a worked example.

Practice: Review a set of support-policy scenarios for inconsistent treatment. Prepare an impact assessment, and build the disclosure into the interface so users are told they're dealing with an AI and what it can and cannot verify.

A security test is valuable only if you use what it finds. You'll bring the earlier protections together, run an attack suite, fix the failures, and test again. Then you'll simulate a real incident: identify what happened, contain it, restore service, and document what still needs attention. The point is to build a system whose safety claims have evidence behind them.

Core topics

• Red-team baselines, attack replay, regression testing, and retesting fixes.
• Severity classification, incident detection, alerts, containment, and recovery.
• Safe rollback, abuse monitoring, and maintaining security controls over time.
• Residual risk, risk ownership, and release-blocking findings.
• Incident runbooks, customer communication, and technical security handoff.
• Explaining a risk, and what remains of it after a fix, to a client in plain words.

Practice: Harden the complete assistant, rerun earlier attacks, and rehearse an incident. Produce test evidence, a containment runbook, and a clear explanation of risks that remain.

A single request can pass through authentication, document search, a language model, and one or more tools. When it fails, you need to know which part was responsible. We'll add consistent tracing and identifiers so the path can be followed without filling logs with customers' private messages and documents.

Core topics

• Logs, metrics, distributed traces, spans, and OpenTelemetry/GenAI conventions.
• Correlation, session, model, prompt, retriever, and application version identifiers.
• LLM, retrieval, generation, tool, and agent events along one request path.
• Safe telemetry schemas, sampling, sensitive-field suppression, and redaction.
• Missing trace detection and practical debugging from telemetry.

Practice: Trace a complete user request through search, generation, and an approved tool action. Introduce a failure and show where the evidence identifies its cause.

A healthy service isn't just one that responds eventually. Users notice slow starts, repeated errors, and requests that never finish. The business also needs to know whether usage is becoming too expensive. You'll choose measures that reflect those problems and build alerts that point toward useful action instead of creating noise.

Core topics

• Time to first token, latency percentiles, throughput, retries, and error rates.
• Model, embedding, retrieval, and tool cost attribution by request, workflow, or customer.
• Service-level indicators and objectives, error budgets, and operational thresholds.
• Dashboards, alert routing, actionable diagnostics, and alert fatigue.
• Capacity and cost forecasting from measured behavior.
• Prompt caching, batching, and offline processing as ways to cut cost and waiting time, measured before and after.

Practice: Build a small operations dashboard, trigger a realistic latency or error alert, and explain what the measurements suggest you should investigate first. Then make one change, such as caching or batching, and measure what it saved.

Evaluation results are only as useful as the examples and labels behind them. If the dataset contains only easy questions, the scores may look excellent while real users still struggle. We'll build a collection of cases that covers common tasks, difficult situations, and important failures. You'll document who labeled the examples and how disagreements were resolved.

Core topics

• Turning business outcomes into evaluation questions and expected behaviors.
• Representative samples, edge cases, adverse cases, and important user segments.
• Source quality, label provenance, annotation guidelines, and disagreement review.
• Versioned datasets, train/evaluation leakage, hidden cases, and reproducibility.
• Using production feedback responsibly without exposing private user information.

Practice: Create a reviewed, versioned evaluation dataset for the assistant. Include multiple kinds of tasks, negative examples, and a record of how labels were checked.

A simple rule can tell you whether a JSON field is missing, but it may not tell you whether an answer is misleading. A model can judge natural-language answers, yet that judge can be wrong too. We'll combine evaluation methods so the results are more trustworthy, and we'll check where automated judgments disagree with reviewed examples.

Core topics

• Deterministic checks, reference/oracle tests, schema validators, and programmatic graders.
• Model-as-judge rubrics, pairwise comparisons, human review, and judge bias.
• Calibrating automated judgments against independently reviewed labels.
• RAG grounding, citation and retrieval correctness, and agent outcome/trajectory evaluation.
• Tools such as DeepEval and Ragas as optional evaluation aids, not substitutes for sound test design.

Practice: Build complementary graders for the same application behavior. Compare model-judge decisions with human-reviewed cases, investigate disagreements, and improve the rubric.

You've changed a prompt, or replaced an embedding model, and the average score went up. Is that improvement reliable? Did latency get worse? Did one group of important questions start failing? We'll plan controlled comparisons and learn to discuss what the evidence actually supports, including uncertainty and the results that don't fit the success story.

Core topics

• Paired evaluation designs, baselines, and keeping test conditions comparable.
• Controlled prompt, model, retriever, and workflow changes.
• Quality, latency, and cost tradeoffs rather than single-score rankings.
• Segment analysis, failure severity, variation, and basic statistical uncertainty.
• Experiment records, cherry-picking risks, and evidence-backed release decisions.

Practice: Compare two versions of a model or retrieval pipeline on the same examples. Report improvements, regressions, uncertainty, and whether you'd ship the change.

You don't want to discover a serious regression because a customer complains after deployment. We'll bring evaluation into the release process so a known failure can stop a change before it reaches production. You'll also learn to turn incidents into new test cases, making the system's protections stronger over time.

Core topics

• A first continuous integration (CI) pipeline that runs your tests on every change.
• Golden test cases, regression suites, thresholds, and release quality gates.
• CI artifacts, failure reports, and reproducible runs.
• Quality and security checks alongside ordinary software tests.
• Production monitoring, version markers, incident-to-evaluation feedback, and rollback triggers.
• Privacy-safe test reporting and auditability of release decisions.
• A quality report a client can read and sign.

Practice: Add a CI check that rejects an intentionally harmful prompt, model, or retriever change. Show why it failed, demonstrate that a corrected change passes, and preserve the test as a future guardrail.

Think about someone using your assistant for the first time. Can they tell when it's still working, when it is uncertain, or when it needs permission to take an action? Good AI product design helps people make those distinctions. We'll map a few realistic user journeys, including the awkward ones, and test how clearly the interface communicates progress, errors, sources, and recovery options.

Core topics

• Stakeholders, user journeys, and how people interpret AI responses.
• Displaying uncertainty, citations, approval requests, retries, and progress.
• Loading states, cancellation, errors, and useful feedback.
• Accessibility, escalation to a person, and recovery from failed interactions.
• Running a small usability test yourself: finding two testers, giving them tasks, and watching without helping.

Practice: Test the assistant with at least two people you recruit, using staff and customer scenarios that include failed operations and missing answers. Use what you observe to improve the interface.

As the application grows, even a small change can unexpectedly affect another feature. It's easier to manage that risk when the frontend, backend, model services, and data stores have clear responsibilities. You'll work on those boundaries and on the contracts between them. We'll pay particular attention to interrupted streams, unsafe content rendering, authentication, and migrations, because these are exactly the sorts of details that surface during maintenance.

Core topics

• API contracts, authentication and session handling, and streaming integration.
• Safe Markdown rendering, state management, persistence, and uploads.
• Schema migrations, modular code boundaries, and versioned APIs.
• Mapping runtime dependencies so their failures can be understood.

Practice: Refactor the application into a maintainable deployment, document its APIs, and test error paths and interrupted streams without exposing secrets.

Imagine fixing a bug and then realizing nobody can remember which version was deployed yesterday. A good release process should make that answer easy to find. You'll set up repeatable builds and tests, keep staging separate from production, and practice releasing from a known commit. We'll also examine configuration changes, since an AI application can behave differently even when its code hasn't changed.

Core topics

• Pinned dependencies, environment configuration, and secrets.
• Git workflows, code review, automated build and test pipelines.
• Containers and runtime fundamentals.
• Amazon Web Services (AWS) as the course's cloud: accounts, access roles, regions, and private networking.
• Describing the deployment as code so it can be rebuilt.
• Continuous integration and delivery (CI/CD), staging, preview checks, and provider-configuration changes.

Practice: Build and deploy to AWS from a known commit, verify the release in staging, and run smoke tests before a controlled production rollout.

A backup is reassuring only if you've tried restoring it. AI applications may keep ordinary database records, uploaded files, and vector indexes, and they don't always recover in the same way. This unit helps you understand what your application stores and how changes affect that data. You'll rehearse a migration and a recovery, then document the steps another developer would need if you weren't available.

Core topics

• Relational, object, and vector data stores; transactions and schema compatibility.
• Database migrations and how they affect running applications.
• Backups, verified restoration, and rebuilding or preserving indexes.
• Change rollouts, rollback constraints, and disaster-recovery scenarios.

Practice: Run a tested data migration, restore from backup, and document what can be recovered and the steps another operator should follow.

Sometimes the first sign of a production problem is simply that users stop getting answers. The cause could be a provider outage, a stalled background job, or a service that didn't start correctly. We'll learn how health checks, logs, and alerts help narrow things down. You'll practice responding to an incident, restoring a usable service, and recording what happened so the same failure is easier to handle next time.

Core topics

• Liveness and readiness checks, timeouts, graceful shutdown, and process signals.
• Rate limits, reliability objectives, alerts, and useful logs.
• Incident severity, runbooks, and ownership of incident response.
• Degraded operation and maintenance windows.

Practice: Simulate a provider outage, stalled work, and a failed health check. Diagnose the problems, restore service, and record what happened.

A feature that works well with a few users can become slow or expensive when more people arrive. That doesn't always mean you need bigger servers. Sometimes you need better limits, smarter caching, or a different model choice. We'll use measurements to explore those decisions. You'll estimate the cost of a real workflow, test concurrency, and explain what you would change if usage increased.

Core topics

• Concurrency, vertical and horizontal scaling, quotas, and abuse limits.
• Caching, index and retrieval budgets, and provider capacity.
• Cost per user and per workflow, with explicit forecasting assumptions.
• Sustainable monitoring and the quality/cost effects of optimization.
• Moving work that doesn't need an instant answer to batch processing.

Practice: Run a realistic load test, estimate operating costs, and defend a capacity or cost decision using measurements.

You've built and deployed the application. What does the client need before they can accept it and take responsibility for it? We'll look at the handoff from their point of view: how to operate the service, what it costs, what to do if it fails, and which requests belong outside the original scope. You'll also practice handling feedback and change requests without losing track of the agreed requirements.

Core topics

• Writing a proposal: scope of work, exclusions, dependencies, and assumptions.
• Acceptance criteria, demonstration, and sign-off, including who owns the data, the accounts, and the running costs.
• User feedback triage and managing change requests.
• Dependency and license inventory, ownership, and access handoff.
• Maintenance responsibilities, support plans, limitations, and incident communication.

Practice: Write a short client proposal and respond to a change request that arrives partway through. Then prepare a complete client handoff with tests, an operating budget, recovery instructions, and a rollback demonstration another engineer can follow.

Fine-tuning is sometimes presented as the answer whenever a model behaves badly. But if the real problem is outdated information, a better retrieval system may help more. We'll compare several ways to improve an AI feature and ask what measurable change training is expected to make. Without that question, it's difficult to know whether the effort paid off.

Core topics

• Fine-tuning versus prompting, retrieval, tools, deterministic code, and switching models.
• Task definition, baseline behavior, measurable improvement hypotheses, and success thresholds.
• Constraints on compute, time, cost, and acceptable regressions.
• Training-data rights, licenses, and permission to adapt a model.
• No-training alternatives and documenting when customization is a poor choice.

Practice: Write an experiment proposal with two testable improvement hypotheses, a preserved baseline, a compute budget, and at least one plausible solution that doesn't involve training.

Training data carries the mistakes of whoever collected and labeled it. Duplicate examples can distort results; private records may be used without permission; and a leaked test example can make a model appear better than it is. You'll build a small dataset with documented sources, clear labeling rules, and a split that lets you evaluate results fairly.

Core topics

• Data rights, privacy, consent, retention, and provenance records.
• Task taxonomies, labeling instructions, annotation consistency, and disagreement review.
• Schemas, formatting, deduplication, quality checks, and token templates.
• Train/development/test splits, contamination, leakage, and synthetic data limitations.
• Dataset cards, validation scripts, versioning, and auditability.

Practice: Produce an approved dataset with a versioned manifest, validators, held-out examples, and an annotation-review record. Show how the test set stays separate from training decisions. If you take Part 9 early, an included bridge supplies the label-review and protected-split methods from Part 7.

Once the dataset is ready, you can run an actual adaptation. We'll start with supervised fine-tuning, where the model learns from example inputs and desired outputs. Then we'll look at parameter-efficient approaches such as LoRA that avoid updating every model weight. The goal is to understand the mechanics, resource requirements, and failure points well enough to reproduce a small experiment.

Core topics

• Supervised fine-tuning (SFT), instruction/chat formatting, tokenization, masking, and packing.
• Optimization basics, checkpoints, stability checks, and experiment logs.
• Full fine-tuning compared with LoRA and other parameter-efficient fine-tuning (PEFT) approaches.
• Quantized-base adapters, memory considerations, and compatibility restrictions.
• Versioned adapters, model artifacts, and reproducibility.
• Optional reading: preference-based tuning such as DPO, and how it can reward the wrong behavior.

Practice: Complete one genuine, resource-bounded SFT or LoRA experiment on a suitable open model. Save the configuration, training logs, data versions, and checkpoints so the run can be audited. You'll know the expected GPU time and cost before you start.

Training finishing successfully doesn't tell you whether the model is better. You need to compare it with what you already had, using tests that weren't used to shape the training process. We'll look for real improvements, but also for new weaknesses: overfitting, poorer behavior on certain users' requests, or a safety failure that the average score hides.

Core topics

• Baseline versus prompted and tuned model comparisons on matched test cases.
• Task-level quality metrics, exact and semantic measures, and qualitative review.
• Held-out cases, data leakage, overfitting, segment-level performance, and negative tests.
• Safety, reliability, regression, and maintenance implications of an adapted model.
• Go/no-go recommendations, including when the trained model should be rejected.

Practice: Evaluate the original model, an improved prompt, and the tuned model on the same held-out dataset. Write a report that explains gains, regressions, uncertainty, and whether deployment is justified.

A model you serve yourself might answer correctly but take too long or use more memory than you can afford. To improve that, we need to see where the time and resources go. You'll measure how quickly generation starts, how fast tokens arrive, and what happens when several people use the service together. Then you'll test changes rather than assuming an optimization must help.

Core topics

• Prefill and decode, key-value (KV) caching, and model memory use.
• Time to first token (TTFT), time per output token (TPOT), percentiles, throughput, and concurrency.
• Quantization, batching, queues, caches, precision, and cold-start behavior.
• Runtime and hosting alternatives with their resource constraints.
• Capacity planning, per-request costs, quality tradeoffs, and rollback criteria.

Practice: Benchmark two inference configurations under controlled load. Compare latency, throughput, memory, cost, and any quality changes before recommending one. An included measurement bridge supports learners taking this Part before Part 7.

A useful experiment needs a clear ending. What was trained, which data was used, what improved, and how could someone safely run or undo the change? You'll package your adapter and evaluation results as a versioned service experiment. If the evidence doesn't support using the tuned model in the main application, keeping it separate is a sound engineering result.

Core topics

• Packaging model/adapter artifacts and recording exact versions.
• Serving endpoint behavior, authentication, health checks, and operational logs.
• Model rollback, release criteria, and failure recovery.
• Managed versus self-hosted serving costs and maintenance effort.
• Model cards, dataset cards, runbooks, evaluation summaries, and technical defense.
• A deploy or no-deploy recommendation written for the client.

Practice: Deliver the adaptation as a separate, testable branch of the main application. Demonstrate its endpoint and rollback path, then defend a deploy or no-deploy decision using real measurements.

When a user sends an image or a recording, the system has new choices to make. Should one model process the whole request, or would a separate transcription or extraction step be more dependable? Then comes the handling itself. A scanned form may look perfectly clear to a person while an extraction system misreads a date or mixes up columns in a table. You'll choose an approach against real user needs and costs, then build an intake path that handles files carefully and gives people a chance to check important extracted details.

Core topics

• Text, image, audio, and video inputs; understanding or extraction tasks versus generation tasks.
• Single-model versus multi-step workflows, with capability, limit, and cost comparisons.
• File type and MIME validation, size limits, consent, secure uploads, and storage lifecycle.
• Image understanding, OCR, scanned-document extraction, tables, and page layout.
• Speech-to-text (STT), noise, accents, and transcription accuracy.
• Timestamps, page or segment anchors, source provenance, and structured fields.
• Batch ingestion, bounded queues, retries, cancellation, and duplicate prevention.
• A review screen for doubtful fields, accuracy for each field, and cost per document.
• Optional reading: image generation, text-to-speech, and video understanding.

Practice: Build a batch scanned-document intake workflow (voice intake is an accepted alternative) that produces reviewable fields. Interrupt and resume a batch without duplicate records. Test labeled examples, report accuracy per field and cost per document, and show how doubtful or unreadable items reach a person in the review screen before a sensitive action can proceed.

A system that works with clean sample images can struggle when the lighting changes. A transcription that looks accurate overall can still get an account number wrong. We'll test these variations deliberately and pay attention to the consequences downstream. If an intake result is uncertain, the application should make that clear and know when to stop for review.

Core topics

• Labeled ground truth for images, document fields, audio, and other media.
• Extraction and speech-recognition errors, noisy inputs, and multilingual accents.
• Quality by segment, downstream RAG or tool impacts, latency, and storage costs.
• Privacy, consent, copyright, authenticity, accessibility, and human verification.
• Safe abstention, review paths, and handling contradictory media evidence.

Practice: Run normal, noisy, and ambiguous media examples through the intake flow. Report the errors that matter most and demonstrate review or abstention when confidence in the result is insufficient.

A coding assistant can draft a component, suggest tests, or help explain unfamiliar code. It can also misunderstand a requirement, add an unsafe dependency, or change more than you asked. You set the basic habits in Unit 1.1. This unit goes further: getting real value from those tools while keeping ownership of the engineering work. You'll work with clear task instructions, inspect the resulting diff, and verify the behavior yourself.

Core topics

• Capabilities and limitations of tools such as Claude Code, Codex, Gemini, Cursor, OpenCode, and Replit.
• Writing bounded implementation tasks and providing useful repository context.
• Reviewing generated diffs, tracking provenance, and documenting AI assistance.
• Repository/file prompt injection, permissions, sandboxing, and dependency risk.
• Code tests, security and license checks, and human accountability for merged changes.

Practice: Use one coding assistant to implement a small scoped feature. Review every change, run relevant tests and scans, and explain what you accepted, changed, or rejected.

It's possible to prompt your way to a working-looking prototype surprisingly quickly. The difficult part comes when the feature needs tests, secure data handling, understandable architecture, and someone who can maintain it. You'll take a deliberately rough AI-generated component and work through those missing engineering responsibilities. Sometimes a rewrite will be the simplest responsible choice.

Core topics

• Rapid exploratory AI coding versus maintainable production engineering.
• Requirements, tests, types, architecture, and explicit API/data boundaries.
• Secrets, permissions, safe storage, deployment, and user experience.
• Dependency review, ownership, and long-term maintenance.
• Criteria for hardening, rewriting, or discarding an experimental prototype.

Practice: Inspect a weak AI-generated prototype, document its problems, and repair it into a tested service component. Show the before-and-after behavior and defend the changes.

You'll finish by adding your selected media intake capability to the Client Knowledge & Operations Assistant. A user should be able to inspect extracted information and correct it before the application relies on it. The new feature must respect the same permissions, evaluations, and operational checks you've developed throughout the course. We'll also verify that adding media hasn't broken the original text-based application.

Core topics

• Integrating consented non-text intake into the existing product.
• Review interfaces, source attribution, uncertainty, and approval boundaries.
• Media error handling, traces, costs, and evaluation datasets.
• Regression tests for existing text-based and tool-using paths.
• Reviewed AI-assisted implementation, updated security records, documentation, and product demonstration.
• Telling the client where AI was used, and demonstrating the feature to them.

Practice: Deploy and demonstrate the multimodal extension with labeled test cases and user verification. Present the code review, performance measurements, updated threat model, and evidence that earlier features still pass their checks.