Dashboard Part 5 Azure AI Search
🔍 Part 5 · Topic 1 of 3

Azure AI Search

Azure AI Search is Microsoft's enterprise search platform that turns any data source into a searchable knowledge base. It combines full-text search, AI enrichment via skillsets, semantic ranking, and vector search into one service. It is the backbone of most knowledge mining and RAG solutions on Azure.

🐍 Python — every line explained ⏱️ ~55 min 🔥 Very high exam weight

Azure AI Search terms — defined before code

Index

A searchable container of documents in Azure AI Search — like a database table but optimised for search. Each document in the index has fields (title, content, category). The index schema defines which fields are searchable, filterable, sortable, and facetable.

Document (Search)

A single record in a search index. Like a row in a database table. Each document has fields defined by the index schema. Example: one document = one product listing with fields: id, name, description, price, category.

Field Attributes

Properties set on each index field: searchable (full-text search over this field), filterable (use in WHERE clauses), sortable (sort results by this field), facetable (use for category counts/navigation), retrievable (include in search results), key (unique document identifier).

Indexer

An automated crawler that reads data from a data source and populates an index. Runs on a schedule or on-demand. Tracks which documents have changed since the last run (change detection). You don't push data manually — the indexer pulls it.

Data Source (Search)

A connection definition pointing to where the indexer should pull data from: Azure Blob Storage, Azure SQL, Cosmos DB, Azure Table Storage, SharePoint Online. Stores the connection string and container/table name.

Skillset

A pipeline of AI enrichment skills applied to documents during indexing. Each skill reads fields, applies AI processing (OCR, NER, sentiment, key phrase extraction, translation, custom skill), and writes enriched output back to the document. Transforms raw content into searchable knowledge.

Built-in Skill

A pre-made AI skill provided by Azure — no model training required. Examples: OCR (extract text from images), EntityRecognition (extract named entities), KeyPhraseExtraction, LanguageDetection, SentimentSkill, TranslationSkill, SplitSkill (chunk text), MergeSkill (combine fields).

Custom Skill

Your own AI processing step in a skillset, implemented as an Azure Function or any web API. The skillset calls your endpoint with document content, your function processes it and returns enriched data. Used for domain-specific AI processing not covered by built-in skills.

Knowledge Store

An optional output of a skillset — enriched data saved to Azure Blob Storage or Azure Table Storage for use outside of search. Useful for downstream analytics, Power BI reporting, or feeding other pipelines with the enriched content.

Semantic Ranking

A re-ranking layer that uses a language model to re-order search results by semantic relevance — finding results that match the meaning of a query, not just the keywords. Returns semantic captions (highlighted relevant excerpts) and semantic answers. Requires the Standard tier or higher.

Vector Search

Search using numerical vector embeddings instead of keywords. Documents and queries are converted to vectors (using an embedding model like text-embedding-ada-002). Search finds documents whose vectors are most similar to the query vector. Handles semantic meaning, synonyms, and cross-language matching.

Hybrid Search (RRF)

Combining keyword search and vector search results using Reciprocal Rank Fusion (RRF). RRF merges two ranked lists into one — documents appearing high in both lists score highest. More robust than either search type alone. RRF is the Azure AI Search default fusion method.

Scoring Profile

Custom rules that boost or reduce the relevance score of documents based on field values, freshness (date), or distance (geo). Example: boost documents where category="Featured" by 2x. Applied after initial ranking, before semantic re-ranking.

Facets

Aggregated counts of values in filterable fields — used for navigation filters. Example: a search for "laptop" returns facets: {brand: [Dell(24), HP(18), Lenovo(15)], price_range: [under_500(12), 500_to_1000(30)]}. Facetable fields must be marked facetable in the schema.

Projection (Knowledge Store)

Defines the shape and destination of enriched data saved to a knowledge store. Table projections go to Azure Table Storage (structured). Object projections go to Blob Storage (JSON). File projections save extracted images to Blob Storage.

The full indexing pipeline

📚
Analogy — A very smart library cataloguing system

Imagine a library receiving thousands of new books (your data). A cataloguing system (the indexer) automatically reads each book. As it reads, a team of specialists (the skillset) annotates each book — one person identifies all the people mentioned, another extracts key topics, another translates the summary. The annotated book is filed in a searchable catalogue (the index). When a reader searches the catalogue, they can search by keyword, by topic, or by meaning — and get the most relevant books ranked at the top.

The 5-component pipeline

1
Data Source

Where the data lives — Azure Blob Storage, Azure SQL, Cosmos DB, SharePoint. The data source stores the connection details. The indexer reads from here.

2
Indexer (the crawler)

Reads documents from the data source, optionally passes them through the skillset, and writes enriched output to the index. Runs on a schedule or on-demand. Tracks changes to only process new/updated documents.

3
Skillset (optional AI enrichment)

A chain of AI skills applied to each document during indexing. Skills can: extract text from images (OCR), detect language, extract entities, extract key phrases, generate embeddings for vector search, or call custom skills. Output fields are added to the document.

4
Index (searchable store)

The final searchable container. Each field has attributes (searchable, filterable, sortable, facetable). This is what your application queries at runtime. All search and filter operations happen against the index.

5
Knowledge Store (optional output)

Saves enriched documents to Azure Blob or Table Storage for use outside of search — Power BI dashboards, downstream pipelines, analytics. Defined inside the skillset as projections.

Field attributes — which to use when

AttributeWhat it enablesTypical fields
keyUniquely identifies each document — required on one fieldid, documentId
searchableFull-text search — the field is analysed and indexed for keyword searchtitle, description, content
filterableExact match filtering — use in $filter expressionscategory, price, date
sortableSort results by this field — use in $orderbydate, price, rating
facetableGenerate aggregated counts for navigation filterscategory, brand, location
retrievableInclude field value in search results returned to the clientAlmost all fields
🎯
Exam Tip — Field attribute scenarios

"Users must be able to search documents by their content." → content field must be searchable

"Users must be able to filter results to only show documents from the Legal department." → department field must be filterable

"The UI shows a list of categories with document counts next to each." → category field must be facetable

"Results must be sortable by publication date." → date field must be sortable

AI enrichment during indexing

Skills are applied in the skillset during indexing. Each skill reads from the document's enriched field tree and writes new enriched fields back. Skills are chained — the output of one skill can be the input of the next.

SkillWhat it doesExam trigger
OcrSkillExtracts text from images embedded in documents (PDFs with scanned pages, Word docs with image attachments)"Documents contain scanned images with text"
MergeSkillMerges original text with OCR-extracted text into a single content field for downstream skills to processAlways used after OcrSkill to combine text streams
EntityRecognitionSkillExtracts named entities: persons, locations, organisations, dates, quantities, URLs, emails"Extract people or company names from documents"
KeyPhraseExtractionSkillExtracts the key topics and concepts from document text"Tag documents with their main topics automatically"
LanguageDetectionSkillDetects the language of the document — output used by other skills that need language context"Documents come in multiple languages"
SentimentSkillAdds a sentiment score (positive/negative/neutral) to each document during indexing"Pre-index reviews with sentiment for fast filtering"
TranslationSkillTranslates document content to a target language during indexing"Non-English documents must be searchable in English"
SplitSkillBreaks long documents into smaller pages or chunks — required before embedding generation for vector search"Documents too large for embedding model's token limit"
AzureOpenAIEmbeddingSkillCalls Azure OpenAI to generate vector embeddings for document chunks — enables vector search"Enable semantic / vector search on the index"
CustomWebApiSkillCalls your own Azure Function or web API as a skill — for domain-specific AI processing"Standard skills don't cover our domain requirement"

Keyword, vector, hybrid, and semantic search

🔤
Full-Text (Keyword) Search

Finds documents containing the exact query words (with stemming and linguistic analysis). Fast and deterministic. Best for: known terminology, structured queries, filter-heavy search. Misses synonyms and phrasing variations.

🧮
Vector Search

Converts query to an embedding vector, finds documents whose vectors are nearest (cosine similarity). Handles semantic meaning, synonyms, cross-language matching. Best for: natural language questions, "find similar" queries. Requires embedding generation at index and query time.

🔀
Hybrid Search (RRF)

Runs keyword AND vector search in parallel, then merges results using Reciprocal Rank Fusion (RRF). Documents appearing high in both ranked lists score highest. More robust than either alone — best of both worlds. Recommended for most production RAG scenarios.

🧠
Semantic Ranking

A re-ranking layer applied AFTER keyword/vector/hybrid search. A language model re-scores results by semantic relevance. Also generates semantic captions (highlighted passages) and semantic answers (direct answer extraction). Requires Standard tier+.

💡
Search type decision — exam quick guide

Keyword only: users search for exact product codes or known terms, speed is critical, no AI budget.

Vector only: users ask natural language questions, cross-language matching needed, all documents have embeddings.

Hybrid (RRF): best overall relevance needed, mix of structured (keyword) and semantic (meaning) queries. Most RAG solutions use this.

Semantic ranking on top of hybrid: maximum relevance quality needed, budget for premium tier, need semantic captions/answers for display in UI.

Creating an index, uploading documents, and searching

Install: pip install azure-search-documents azure-core

import os
from azure.search.documents import SearchClient
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex,          # Represents the full index definition
    SearchField,          # Represents one field in the index schema
    SearchFieldDataType,  # Data types: String, Int32, Double, Boolean, etc.
    SimpleField,          # A field that is NOT full-text searchable (filterable/sortable)
    SearchableField,      # A field that IS full-text searchable (analysed and indexed)
)
from azure.core.credentials import AzureKeyCredential

# ── Connection details ─────────────────────────────────────────────────
SEARCH_ENDPOINT = os.environ.get("SEARCH_ENDPOINT")  # https://your-name.search.windows.net
SEARCH_KEY      = os.environ.get("SEARCH_KEY")      # Admin key from the Azure portal
INDEX_NAME      = "products-index"                  # Name for our index

credential = AzureKeyCredential(SEARCH_KEY)

# ── Step 1: Create the index schema ───────────────────────────────────
# SearchIndexClient manages index creation and deletion (admin operations)
index_client = SearchIndexClient(endpoint=SEARCH_ENDPOINT, credential=credential)

# Define the fields for our product index
# SimpleField: NOT full-text searchable — used for filtering, sorting, faceting
# SearchableField: IS full-text searchable — content is analysed and inverted
fields = [
    SimpleField(
        name       = "id",
        type       = SearchFieldDataType.String,
        key        = True,          # key=True makes this the unique document identifier
        filterable = True           # Allow filtering by id
    ),
    SearchableField(
        name      = "name",
        type      = SearchFieldDataType.String,  # Free-text product name — full-text searchable
        sortable  = True                         # Allow sorting results by name
    ),
    SearchableField(
        name = "description",
        type = SearchFieldDataType.String        # Long-form text — full-text searchable only
    ),
    SimpleField(
        name       = "category",
        type       = SearchFieldDataType.String,
        filterable = True,    # Allow filtering: $filter=category eq 'Electronics'
        facetable  = True     # Enable category counts for navigation filters
    ),
    SimpleField(
        name       = "price",
        type       = SearchFieldDataType.Double,
        filterable = True,    # Allow filtering: $filter=price lt 500
        sortable   = True     # Allow sorting by price
    ),
    SimpleField(
        name        = "in_stock",
        type        = SearchFieldDataType.Boolean,
        filterable  = True    # Allow filtering: $filter=in_stock eq true
    ),
]

# Create the index definition and send it to Azure
index = SearchIndex(name=INDEX_NAME, fields=fields)
index_client.create_or_update_index(index)  # Creates if new, updates if exists
print(f"Index '{INDEX_NAME}' created.")


# ── Step 2: Upload documents to the index ─────────────────────────────
# SearchClient is used for document operations (upload, search, delete)
search_client = SearchClient(
    endpoint   = SEARCH_ENDPOINT,
    index_name = INDEX_NAME,
    credential = credential
)

# Documents are plain Python dictionaries
# Field names must exactly match the index schema
documents = [
    {"id": "1", "name": "Dell G15 Gaming Laptop",   "description": "High-performance gaming laptop with RTX 3060", "category": "Laptops",   "price": 999.99,  "in_stock": True},
    {"id": "2", "name": "Sony WH-1000XM5 Headphones", "description": "Industry-leading noise cancelling wireless headphones", "category": "Audio",     "price": 349.99,  "in_stock": True},
    {"id": "3", "name": "Logitech MX Master 3 Mouse",  "description": "Advanced wireless mouse for productivity",              "category": "Peripherals","price": 99.99,   "in_stock": False},
]

# upload_documents() adds or replaces documents in the index
# Returns a list of IndexingResult objects — one per document
result = search_client.upload_documents(documents=documents)
print(f"Uploaded {len(result)} documents.")


# ── Step 3: Basic keyword search ──────────────────────────────────────
print("\n--- Keyword Search: 'wireless' ---")

# search() returns a SearchItemPaged iterator
# search_text: the query string (* = return all documents)
# select: which fields to return (comma-separated)
# filter: OData filter expression (like a WHERE clause)
results = search_client.search(
    search_text = "wireless",
    select      = ["name", "category", "price"],
    filter      = "in_stock eq true"   # Only return in-stock items
)

for doc in results:
    print(f"  {doc['name']} | {doc['category']} | £{doc['price']}")


# ── Step 4: Search with facets ────────────────────────────────────────
print("\n--- Search with Facets ---")

# facets: request aggregated counts for specified facetable fields
# Results include a @search.facets property with the counts
results_with_facets = search_client.search(
    search_text = "*",       # * = all documents
    facets      = ["category", "price,interval:100"]  # Category counts + price buckets of 100
)

# Access facet data from the result's get_facets() method
facets = results_with_facets.get_facets()
print("Category facets:", facets.get("category"))
print("Price facets:",    facets.get("price"))

Python — Semantic search query

Semantic ranking re-ranks results by meaning. Requires the index to have semantic configuration defined and the service to be on Standard tier or above.

from azure.search.documents.models import QueryType

# Semantic search query
# query_type=QueryType.SEMANTIC activates the semantic re-ranking layer
# semantic_configuration_name: the name of your semantic config (set in index definition)
# query_caption: "extractive" returns highlighted excerpts from matching documents
# query_answer: "extractive" extracts a direct answer if one is found in the results
semantic_results = search_client.search(
    search_text                = "best laptop for gaming under 1000",
    query_type                 = QueryType.SEMANTIC,
    semantic_configuration_name= "my-semantic-config",
    query_caption              = "extractive",
    query_answer               = "extractive",
    select                     = ["name", "description", "price"],
    top                        = 3   # Return top 3 results
)

for doc in semantic_results:
    # @search.reranker_score: the semantic model's relevance score (0-4)
    score = doc.get("@search.reranker_score", 0)
    print(f"  [{score:.2f}] {doc['name']} — £{doc['price']}")

    # Semantic captions: highlighted excerpts showing WHY this document is relevant
    captions = doc.get("@search.captions", [])
    if captions:
        print(f"    Caption: {captions[0].text}")

Azure AI Search — what the exam tests

⚠️
Top 6 Tested Facts

1. Index → Indexer → Skillset → Data Source relationship — Know how these 4 components connect. Indexer is the component that orchestrates the pipeline (reads from data source, applies skillset, writes to index).

2. Field attributes: searchable vs filterable vs facetable vs sortable — These are distinct. A field can be filterable but not searchable. Facetable fields are used for navigation counts. Key field must be unique per document.

3. OcrSkill + MergeSkill always go together — OCR extracts text from images; MergeSkill combines OCR-extracted text with the original document text. Without MergeSkill, OCR output isn't part of the searchable content.

4. Semantic ranking requires Standard tier+ — Cannot be used on the Free or Basic tier. Re-ranks results after initial retrieval using a language model. Returns reranker_score (0-4).

5. Hybrid search uses RRF (Reciprocal Rank Fusion) — Merges keyword and vector ranked lists. Documents appearing high in both lists rank highest. Most robust for production RAG pipelines.

6. Knowledge Store saves enriched data outside of search — For Power BI / analytics use cases. Uses projections (table, object, file) to define shape and destination.

Test your understanding

Topic 1 — Azure AI Search 1 / 6
Scenario: A company indexes a large library of PDF documents stored in Azure Blob Storage. Many PDFs are scanned (image-only) and contain no searchable text layer. The company wants all text — including text inside scanned images — to be searchable.

Which two skills must be included in the skillset together to achieve this?

💡 Explanation

B is correct. OcrSkill extracts text from embedded images but writes it to a separate field. MergeSkill is always paired with OcrSkill — it combines the original document text with the OCR-extracted text into a single unified content field that downstream skills (and the index) can use. Without MergeSkill, OCR output exists as an isolated field not included in the main searchable content. Option D is wrong — OcrSkill writes to its own output field, not the main content field.

A search index has a "category" field. Users need to see a sidebar showing "Electronics (45)", "Books (23)", "Clothing (12)" — aggregated counts of documents per category. Which field attribute must be set on the category field?

💡 Explanation

B is correct. Facetable enables the aggregation of document counts by field value — exactly what powers "Electronics (45)" style navigation filters. Searchable (A) enables full-text search within the field. Sortable (C) allows ordering results by the field. Retrievable (D) includes the field value in result documents. A field can have multiple attributes — category would typically be filterable AND facetable.

Scenario: A company builds a product search engine. Users search for "affordable noise-cancelling headphones" but the keyword search misses results because documents use the term "active noise reduction" instead of "noise-cancelling". The company wants search to understand the semantic meaning of queries, not just match keywords.

Which search approach should be implemented?

💡 Explanation

B is correct. Vector search (or hybrid search combining vector + keyword) handles semantic similarity — it finds documents that are conceptually related to the query, even when different vocabulary is used. "Noise-cancelling" and "active noise reduction" will have similar vector representations. Synonyms (A) could work but require manual maintenance for every synonym pair. Filters (C) narrow results — they don't improve relevance. Increasing top (D) returns more results but doesn't fix the relevance problem.

What is Reciprocal Rank Fusion (RRF) in Azure AI Search?

💡 Explanation

B is correct. Reciprocal Rank Fusion (RRF) is the algorithm used in hybrid search to merge two ranked lists (keyword results and vector results) into a single combined ranking. A document's RRF score is calculated based on its position in each ranked list — appearing at rank 1 in both lists gives the highest combined score. This makes hybrid search more robust than either keyword or vector search alone.

Scenario: A company enriches their document index with skillsets (entity extraction, key phrase extraction, sentiment). The data science team wants to use this enriched data in Power BI dashboards and run custom analytics — but they cannot query this data directly from the search index.

Which Azure AI Search feature allows enriched data to be saved outside the index for analytics use?

💡 Explanation

B is correct. Knowledge Store is specifically designed for this use case — persisting enriched data outside the search index for downstream consumption. You define projections (table projections → Azure Table Storage, object projections → Blob Storage) inside the skillset definition. Power BI can then connect directly to Table Storage. Scoring Profiles (A) affect ranking, not data persistence. Semantic Ranking (C) re-orders results — doesn't export data. Custom Skill (D) could technically work but Knowledge Store is the purpose-built Azure-native solution.

Which component in the Azure AI Search pipeline is responsible for reading data from the source, applying skills, and writing enriched documents to the index?

💡 Explanation

B is correct. The Indexer is the orchestration engine of the Azure AI Search pipeline. It: connects to the data source and reads documents, passes each document through the skillset for AI enrichment, writes the enriched document fields to the index, tracks changes to only process new/updated documents on subsequent runs, and can run on a schedule or be triggered on-demand. The Skillset only defines the AI processing steps — it doesn't pull data or write to the index itself.

Mark this topic as complete