Optimize & Operationalize GenAI Solutions

Optimisation and operationalisation terms

Controlling how the model generates text

Every call to the Chat Completions API can include optional parameters that change how the model generates its response. The exam tests whether you know which parameter to use for which problem.

Temperature — the most important parameter

All parameters in a single Python call

import os
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint = os.environ.get("AZURE_OPENAI_ENDPOINT"),
    api_key        = os.environ.get("AZURE_OPENAI_KEY"),
    api_version    = "2024-02-01"
)

response = client.chat.completions.create(
    model    = "my-gpt4o",
    messages = [
        {"role": "system", "content": "You are a product description writer."},
        {"role": "user",   "content": "Write a product description for wireless headphones."}
    ],

    # ── Generation control parameters ────────────────────────────────────

    temperature       = 0.8,
    # 0.8 = moderately creative (good for marketing copy)
    # Range: 0.0 (deterministic) to 2.0 (very random)
    # Use EITHER temperature OR top_p — not both

    max_tokens        = 300,
    # Maximum tokens in the response
    # Response is cut off if it reaches this limit
    # Does NOT include input tokens — just output

    frequency_penalty = 0.5,
    # Reduces repetition of words already used in the response
    # Range: -2.0 to 2.0  |  0 = no penalty  |  positive = less repetition

    presence_penalty  = 0.3,
    # Encourages new topics/vocabulary
    # Range: -2.0 to 2.0  |  0 = no penalty  |  positive = more novelty

    stop              = ["###", "END"],
    # Stop generating if the model outputs "###" or "END"
    # Useful for structured outputs with known end markers

    n                 = 1,
    # Number of response alternatives to generate
    # n=1 is standard. n=3 generates 3 different responses — you choose the best
    # Higher n costs proportionally more tokens
)

answer = response.choices[0].message.content
print(answer)

Getting better responses through better prompts

Prompt engineering is the practice of crafting prompts that reliably produce the outputs you want. These are the techniques the exam tests.

When to fine-tune a model and how it works

Fine-tuning customises a pre-trained model by training it further on your own dataset. The model's internal weights are updated to make it better at specific tasks or to adopt a specific writing style or domain knowledge.

Fine-tuning vs RAG — the most important decision

Fine-tuning training data format

Azure OpenAI fine-tuning requires training data in JSONL format (one JSON object per line). Each object is a conversation with system, user, and assistant messages — showing the model the style of response you want.

# Fine-tuning training data format — JSONL (JSON Lines)
# Each line is one training example — a complete conversation
# The model learns to respond to "user" messages the way "assistant" does in your examples

# Line 1: training example 1
{"messages": [
    {"role": "system",    "content": "You are a concise technical support agent."},
    {"role": "user",      "content": "My printer won't connect to WiFi."},
    {"role": "assistant", "content": "Try: 1) Restart router. 2) Forget and rejoin network. 3) Update printer firmware."}
]}

# Line 2: training example 2
{"messages": [
    {"role": "system",    "content": "You are a concise technical support agent."},
    {"role": "user",      "content": "I can't login — password reset email not arriving."},
    {"role": "assistant", "content": "Check spam folder. Whitelist noreply@company.com. Try alternate email if registered."}
]}

# Minimum: 10 training examples (recommended: 50-100+)
# More examples = better fine-tuning results
# Upload this file to Foundry: your project → Fine-tuning → + New fine-tune job

Managing scale and avoiding throttling

Handling HTTP 429 (throttling) in Python

New Python concept: try/except — a way to handle errors gracefully instead of crashing. Explained fully in comments.

import os, time
from openai import AzureOpenAI, RateLimitError

client = AzureOpenAI(
    azure_endpoint = os.environ.get("AZURE_OPENAI_ENDPOINT"),
    api_key        = os.environ.get("AZURE_OPENAI_KEY"),
    api_version    = "2024-02-01"
)

# New concept: try/except ────────────────────────────────────────────
# "try:" — attempt to run this code
# "except SomeError:" — if THIS specific error occurs, run this instead
# This prevents your program from crashing when an error happens
# It's like a safety net — "try this, but if it fails, do that"

MAX_RETRIES = 3   # How many times to retry before giving up

for attempt in range(MAX_RETRIES):   # range(3) = [0, 1, 2] — loops 3 times
    try:
        # Try to call the API
        response = client.chat.completions.create(
            model    = "my-gpt4o",
            messages = [{"role": "user", "content": "Explain quantum computing simply."}]
        )
        # If the call succeeded, print the result and break out of the loop
        print(response.choices[0].message.content)
        break    # "break" exits the loop immediately — success!

    except RateLimitError as e:
        # RateLimitError happens when we exceed our TPM quota (HTTP 429)
        # "as e" gives us access to the error details
        if attempt < MAX_RETRIES - 1:   # If this isn't our last attempt...
            wait_seconds = 2 ** attempt    # Exponential backoff: 1s, 2s, 4s
            print(f"Rate limited. Waiting {wait_seconds}s before retry {attempt+1}...")
            time.sleep(wait_seconds)      # time.sleep() pauses the program for N seconds
        else:
            # We've used all our retries — give up and report the error
            print(f"Failed after {MAX_RETRIES} attempts: {e}")

Keeping your GenAI solution healthy in production

From Part 1 Topic 4 you know Azure Monitor and Application Insights. GenAI solutions add additional monitoring needs specific to LLM behaviour.

Key metrics to monitor for GenAI

Enabling tracing in Foundry

Tracing records the complete execution of a Prompt Flow — every node, every input/output, every duration. This is invaluable for debugging.

Self-improvement and multi-model pipelines

Model Reflection — the model critiques itself

Model Reflection is a pattern where you ask the model to generate a response, then ask it to critique that response, then generate an improved version. This iterative loop produces higher quality outputs than a single generation pass.

import os
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
    api_key=os.environ.get("AZURE_OPENAI_KEY"),
    api_version="2024-02-01"
)
MODEL = "my-gpt4o"

# ── Step 1: Generate initial response ─────────────────────────────────
user_task = "Write a 3-sentence summary of quantum computing for a non-technical audience."

initial = client.chat.completions.create(
    model    = MODEL,
    messages = [{"role": "user", "content": user_task}]
)
initial_response = initial.choices[0].message.content
print(f"INITIAL:\n{initial_response}\n")


# ── Step 2: Ask the model to critique its own response ────────────────
# We include the initial response in the conversation history
# so the model can see what it generated and evaluate it
critique = client.chat.completions.create(
    model    = MODEL,
    messages = [
        {"role": "user",      "content": user_task},
        {"role": "assistant", "content": initial_response},  # The initial response
        {"role": "user",      "content": "Critique this response. Is it accurate? "
                                          "Is it truly non-technical? What could be improved?"}
    ]
)
critique_text = critique.choices[0].message.content
print(f"CRITIQUE:\n{critique_text}\n")


# ── Step 3: Generate improved response based on the critique ──────────
improved = client.chat.completions.create(
    model    = MODEL,
    messages = [
        {"role": "user",      "content": user_task},
        {"role": "assistant", "content": initial_response},
        {"role": "user",      "content": "Critique: " + critique_text},
        {"role": "user",      "content": "Now write an improved version addressing the critique."}
    ]
)
improved_response = improved.choices[0].message.content
print(f"IMPROVED:\n{improved_response}")

Orchestration — multiple models working together

Real enterprise AI solutions often combine multiple models and services, each doing what it does best. The exam tests understanding of when to combine models and how.

Optimisation — what the exam specifically tests

Test your understanding

5 questions — select an answer to see the explanation immediately.