⬡ Part 4 · Topic 4
Deduplication & Remediation Playbooks — Finding and Fixing Duplicate CIs
Even with perfect Identification Rules, duplicate CIs can creep into your CMDB — from legacy migrations, misconfigured integrations, or before IRE was enabled. IRE's third phase detects these duplicates. Remediation Playbooks define what happens next: auto-merge, keep-primary, or flag for human review. This topic completes your IRE knowledge and links it to the broader CMDB data quality picture.
Section 1
How Duplicate CIs Form in the Real World
Duplicate CIs are one of the most damaging CMDB quality problems. A duplicate CI means two records in the CMDB represent the same real-world device or application. This causes:
- Incorrect reporting: Reports count the server twice, inflating capacity numbers
- Wrong impact analysis: Change impact analysis may evaluate only one of the two duplicate CIs, missing relationships on the other
- Split data: One duplicate has the correct ip_address; the other has the correct serial_number. Neither has complete data.
- Service Mapping breakage: A Business Service links to one duplicate; incidents link to the other
Common Causes of Duplicates
- Missing or wrong Identification Rules: IRE couldn't match the incoming payload to the existing CI, so it created a new one. Classic cause: Discovery pattern not collecting the serial number that the Identification Rule expects.
- Data migration without IRE: Before IRE existed (or when it was bypassed), someone imported CIs directly into CMDB tables via Table API or Import Sets without IRE. No deduplication occurred at import time.
- CI class mismatch: The same device was discovered as cmdb_ci_computer once and as cmdb_ci_linux_server another time. Different class = different Identification Rule = IRE may not link them.
- Multiple integration sources without rule alignment: Both AWS SGC and Discovery created the same EC2 instance as separate CIs because their identification approaches didn't share a common field.
Analogy — Duplicates as Filing Error
Imagine a file room where staff files client folders by name. One clerk files "Johnson, Bob"
under J. Another files the same client as "Bob Johnson" under B. Now the same client has
two folders with split information. Deduplication is the process of finding these pairs,
determining which is the primary folder, copying unique information from one to the other,
and then discarding the duplicate.
Section 2
IRE Deduplication Detection — Phase 3 in Detail
IRE's Phase 3 (Deduplication) can be triggered in two ways:
- Real-time: during ingestion. When Phase 1 (Identification) finds multiple existing CIs that match the incoming payload, IRE knows potential duplicates exist. It flags them and creates a deduplication event.
- Scheduled: Deduplication Job. A separate, schedulable process scans the entire CMDB (or specific CI classes) looking for existing records that match each other based on Identification Rules. This catches duplicates that exist from before IRE was configured or that slipped through during the transition.
What IRE Stores When It Finds a Duplicate
When IRE identifies a potential duplicate pair, it creates a record in the cmdb_dedup table. Each record contains:
- Candidate CI — the CI that appears to be a duplicate
- Master CI — the CI that should be kept as the primary record
- Match reason — which identification attributes caused IRE to consider them duplicates
- Status — Open, In Progress, Resolved, or Dismissed
- Remediation Playbook — which playbook will be used to resolve this duplicate
Key Concept — Dedup Finds Pairs, Not Single Records
Deduplication always works on pairs: a "master" (the one to keep) and a "duplicate"
(the one to remove or merge into the master). IRE uses the Identification Rules to
determine which CIs are candidates for being the same real-world entity.
IRE does not automatically delete duplicate CIs — it creates a dedup event and
a Remediation Playbook decides what to do.
Section 3
Determining the Master CI — Which Duplicate Is Kept?
When two CIs are flagged as duplicates, IRE must determine which one becomes the master CI (the one kept) and which becomes the duplicate CI (the one merged/deleted).
IRE evaluates several criteria to determine master status:
| Criterion | Logic | Rationale |
|---|---|---|
| Source Priority | CI from higher-priority source (e.g., Discovery over Import Set) becomes master | Higher-priority sources have more accurate data |
| Date Created | Older CI may become master (has more history and relationships) | Older records have accumulated relationships, tickets, and history |
| Data Completeness | CI with more non-null fields may be preferred as master | More complete record is better starting point after merge |
| Relationship Count | CI with more existing relationships may be preferred | Preserving relationship links is critical for impact analysis |
The specific criteria and their weights can be configured in the Remediation Playbook. You can also manually designate which CI is the master when reviewing duplicates.
Exam Trap — Master Determination Is Configurable
There is no single fixed rule for which duplicate becomes master. The exam may present
a scenario and ask which CI would be chosen as master. The answer depends on the
Remediation Playbook configuration and the specific criteria. However, the principle
is consistent: the master CI should be the one with the most accurate, complete data
and the most existing relationships.
Section 4
Remediation Playbooks — Automating the Dedup Response
A Remediation Playbook is a configured set of instructions that defines exactly what IRE should do when it finds a duplicate CI pair. Think of it as the "recipe" for resolving a detected duplicate.
Playbook Actions — What Can Happen to the Duplicate
| Action | What Happens | When to Use |
|---|---|---|
| Merge | Data from the duplicate CI is merged into the master CI. Fields from duplicate fill in blanks on the master. Then the duplicate CI is retired/deleted. | When you want to preserve unique data from both CIs into one record |
| Keep Primary (Mark Duplicate) | The master CI is kept unchanged. The duplicate CI is marked as "Duplicate" or retired. No field merging occurs. | When the master has all the data you need and the duplicate has nothing unique to contribute |
| Flag for Manual Review | A task is created for a CMDB administrator to review and resolve manually. No automatic changes to either CI. | When the duplicate situation is complex or uncertain enough that automation could make mistakes |
| Ignore/Dismiss | The dedup event is dismissed. Both CIs remain unchanged. Used when IRE incorrectly flagged two different things as duplicates. | When the two CIs are actually different entities that just happen to match on identification attributes |
Configuring a Remediation Playbook
Navigate to Configuration → CMDB → Remediation Playbooks. A playbook configuration specifies:
- Which CI classes this playbook applies to
- The action to take (merge, keep primary, manual review, etc.)
- Master determination criteria
- Field-merge rules: which fields from the duplicate should overwrite blanks on the master
- What to do with relationships on the duplicate (re-point to master, or discard)
Tip — Relationships Are the Most Valuable Thing to Preserve
When merging duplicates, the most important thing to preserve is not the field data
(that can be re-discovered) but the relationships: open incidents,
change requests, service mappings, and contracts linked to either CI. A Remediation
Playbook should always be configured to re-point all relationships from the duplicate
to the master before retiring the duplicate.
Section 5
The Merge Process — Step by Step
When a Remediation Playbook with action=Merge executes, here is exactly what happens:
↓
Trigger
Dedup Event: Master CI ↔ Duplicate CI
IRE has determined that CI
sys_id=A (Master) and CI sys_id=B (Duplicate) represent the same physical or logical entity. A Remediation Playbook with action=Merge is triggered.1
Field Compare
Fill Blanks on Master from Duplicate
For each field that is blank on the Master CI but has a value on the Duplicate, IRE copies the Duplicate's value to the Master. Example: Master has no
serial_number; Duplicate has one → copy to Master.2
Playbook Rules
Apply Field-Merge Rules from Playbook
The Remediation Playbook may define explicit field-merge rules beyond simple blank-fill. Some fields from the Duplicate may override Master values regardless of whether Master has a value. This is configurable per organization policy.
3
Relationship Migration
Re-point All Relationships to Master
All records pointing to Duplicate CI B are updated to point to Master CI A instead. This includes incidents, change requests,
cmdb_rel_ci rows, and service map connections. No relationship is orphaned.4
Lifecycle Update
Duplicate CI Retired
The Duplicate CI (B) is marked as Retired in its operational status field. It is not immediately deleted — retirement preserves audit history. Fully automated deletion can be configured after a grace period.
5
Resolution
Dedup Event Marked Resolved
The dedup remediation event record is updated to Resolved status. The CMDB Dedupe dashboard reflects the resolution. Both the merge action and the resolution are logged for audit purposes.
What Happens to the Retired Duplicate CI
The duplicate CI is not always deleted — it's often "retired" (its operational_status field is set to "Retired"). This preserves history: you can still see the old CI and understand what happened. A truly deleted CI would remove audit history.
Some organizations configure playbooks to fully delete duplicates after a configurable period (e.g., 90 days after retirement). This is a governance decision.
Warning — Test Merge in Non-Production First
Merging CIs is partially reversible — you can restore from backup — but
re-pointing hundreds of relationships, incident links, and change records
is very difficult to undo. Always test Remediation Playbook configurations
in a sub-production environment with representative data before enabling
automatic merges in production.
Section 6
Finding and Managing Duplicates — The Dedup Dashboard
ServiceNow provides a dedicated Deduplication Dashboard for CMDB administrators to monitor and manage duplicate CIs.
Accessing Deduplication Tools
- Deduplication Dashboard: Navigate to Configuration → CMDB → Deduplication → Deduplication Dashboard. Shows a summary of open dedup events, by CI class, with trends over time.
- Duplicate CI list: Shows all CIs currently flagged as duplicates with their master CI counterpart.
- Dedup Job: The scheduled job that proactively scans for duplicates. Can be run on-demand or scheduled. Navigate to Configuration → CMDB → Deduplication → Run Deduplication.
Running an On-Demand Dedup Scan
Useful scenarios for running an on-demand deduplication scan:
- After fixing a broken Identification Rule that had been creating duplicates for months
- After a large data migration (Import Set of hundreds of CIs)
- After enabling a new SGC that discovered CIs already in the CMDB via Discovery
- During a CMDB health remediation project
You can scope the dedup scan to specific CI classes rather than scanning the entire CMDB, which reduces processing time in large environments.
Exam Pattern — Dedup After Identification Rule Changes
When an Identification Rule change is made to fix a root cause of duplicate creation,
fixing the rule alone doesn't clean up existing duplicates — it only prevents
new ones from being created. An on-demand deduplication scan must be run separately
to detect and resolve existing duplicate pairs. The exam may test whether you know
this two-step process.
Section 7
Deduplication vs. Manual Merge — When Each Is Right
Besides IRE's automated deduplication, administrators can also manually merge CIs using the CMDB Workspace or the "Merge CI" action directly on a CI form.
| Approach | How It Works | Best For |
|---|---|---|
| IRE Auto-Dedup + Playbook | IRE detects duplicates via Identification Rules and executes a pre-configured Remediation Playbook automatically or on a schedule | Large-scale, ongoing deduplication. Hundreds or thousands of potential duplicates. Clear matching criteria. |
| Manual Merge (CMDB Workspace) | Administrator opens two CI records side-by-side, selects which fields to keep from each, and executes the merge manually | Complex individual cases where automated merge rules would make wrong decisions. When administrator judgment is required. |
| Dismiss Dedup Event | Administrator reviews the flagged pair and determines they are actually different CIs that just happen to match on identifiers. Dismisses the event. | False positives from IRE — where IRE detected "duplicates" that are legitimately separate CIs. |
Key Concept — Some "Duplicates" Are False Positives
IRE identifies potential duplicates based on matching identification attributes.
But matching attributes don't always mean the same real-world entity. Example:
a serial number of "Not Specified" might match across hundreds of low-end devices
whose BIOS doesn't report a real serial number. IRE would flag all of these as
duplicates of each other — but they're actually different devices. Dismiss these events
and improve the Identification Rule to exclude unreliable values.
Section 8
Deduplication as Part of CMDB Data Quality
Deduplication is not a one-time project — it's an ongoing quality control process. Even with perfect Identification Rules and Reconciliation Rules configured today, future changes (new integrations, rule modifications, data migrations) can introduce new duplicates.
Integrating Dedup with CMDB Health Metrics
CMDB Health (covered in depth in Part 6) tracks duplicate CI counts as one of its key quality indicators. A high duplicate rate in any CI class should trigger:
- Investigation into root cause (broken Identification Rule? New integration without rule?)
- Fix the root cause first (prevent new duplicates from being created)
- Run dedup scan to detect existing pairs
- Apply Remediation Playbook to resolve detected pairs
- Monitor CMDB Health metrics to confirm improvement
Key Metrics to Monitor
- Duplicate CI count per class: How many CIs are flagged as duplicates right now
- Dedup event resolution rate: Are flagged duplicates being resolved or piling up?
- Dedup age: How old are unresolved dedup events? Older = more embedded in reports and relationships
- Post-merge data completeness: After merging, are the resulting CIs more complete?
Tip — Fix Root Cause Before Running Remediation
Never run a large-scale Remediation Playbook before fixing the underlying root cause.
If the Identification Rule is still broken, the dedup scan will merge pairs —
but Discovery will immediately re-create duplicates the next time it runs.
You'll be back to the same state within 24 hours. Always root-cause fix first.
Exam Quick Reference
Deduplication & Remediation — Key Facts
- Duplicates form when Identification Rules miss, data migrated without IRE, or class mismatch between sources.
- IRE Phase 3 (Dedup) triggered two ways: (1) real-time when Phase 1 finds multiple matches, (2) scheduled Dedup Job scanning entire CMDB.
- cmdb_dedup table stores dedup events — each record has a master CI and a candidate duplicate CI.
- Master CI is determined by source priority, data completeness, age, and relationship count. Configurable in Remediation Playbook.
- Remediation Playbook actions: Merge (copy data + re-point relationships + retire duplicate), Keep Primary, Flag for Manual Review, or Dismiss.
- Relationships are most critical to preserve. All relationships from duplicate must be re-pointed to master before retirement.
- Duplicates are usually retired, not deleted — preserves audit history.
- Fix root cause first, then run dedup. Running dedup before fixing the Identification Rule means duplicates return immediately after the next Discovery run.
- Manual Merge available for complex cases where automated playbook decisions would be wrong.
- Dismiss dedup events when IRE incorrectly flags two different entities as duplicates (false positives).
Term Grid
Deduplication (IRE Phase 3)
IRE process that detects CI records representing the same real-world entity. Runs during ingestion (when Phase 1 finds multiple matches) and via scheduled Dedup Job.
cmdb_dedup Table
ServiceNow table storing duplicate detection events. Each record pairs a master CI and a candidate duplicate CI with match reason, status, and assigned playbook.
Remediation Playbook
Configured set of rules defining the automated response to a detected duplicate pair. Actions: merge, keep primary, flag for review, or dismiss.
Master CI
The CI designated to survive after deduplication. Receives merged field data and all relationships from the duplicate CI. Determination criteria are configurable.
Dedup Job
Scheduled (or on-demand) process that scans the CMDB for existing duplicate pairs. Does not fix duplicates — creates cmdb_dedup events for Remediation Playbooks to resolve.
Dismiss (Dedup)
Dedup event resolution action used when IRE incorrectly flagged two different real-world entities as duplicates. Both CIs remain; the event is closed as a false positive.
Exam Checkpoint
Practice Questions
Click any question to reveal the answer and explanation.
1. A CMDB administrator fixes a broken Identification Rule that was causing Discovery to create duplicate Linux server CIs. After fixing the rule, she wants to clean up the thousands of existing duplicate CIs created during the past 6 months. What is the correct next step?
💡 Fixing the rule prevents new duplicates. What process finds and resolves existing ones?
Answer: B — Fixing the Identification Rule prevents new duplicates from being created going forward, but it doesn't clean up the existing ones. The Deduplication Job (run on-demand, scoped to the affected CI class) scans the CMDB and creates cmdb_dedup events for each duplicate pair. Then a Remediation Playbook (configured to Merge) automatically resolves them. Discovery re-run (A) won't merge existing duplicates; it just updates/creates CIs.
2. IRE detects two server CIs as potential duplicates: CI-A (created 2 years ago from Discovery, has 12 open incidents linked to it, ip_address=10.0.1.5, no serial_number) and CI-B (created 1 week ago from an AWS SGC, has serial_number=SN-XYZ, no incidents, ip_address=10.0.1.5). Which should be designated as the master CI and why?
💡 Relationships are the most critical thing to preserve. Which CI has relationships?
Answer: B — While CI-B has better identifier data (serial number), CI-A should be master because it has 12 open incident relationships. Making CI-A master means those incidents are automatically preserved on the surviving CI. The Remediation Playbook would be configured to copy serial_number from CI-B (the duplicate) into CI-A (master) to fill in the missing data, then retire CI-B. Relationships are always the primary driver in master selection.
3. The CMDB Dedup Dashboard shows 800 duplicate events for cmdb_ci_network_adapter. Investigation reveals that network adapters from virtual machines all have MAC address "00:00:00:00:00:00" (a placeholder value). The Identification Rule uses MAC address to identify network adapters. What is the best solution?
💡 When multiple different devices share a fake "placeholder" identifier, are they really duplicates?
Answer: B — This is a false positive scenario. The 800 adapters are different real-world network interfaces that happen to share a meaningless placeholder MAC address. They are NOT duplicates — merging them would be catastrophically wrong. The correct response: dismiss all 800 events (they're false positives), then modify the Identification Rule to skip the identification check when MAC address = "00:00:00:00:00:00" or use an additional qualifying attribute. Option D (use IP) might work but IPs also change, so the root fix is excluding known-bad placeholder values.
4. An administrator merges two duplicate server CIs. After the merge, users report that incident INC0012345, which was linked to the duplicate (now retired) CI, no longer shows the CI in its Configuration Items field. What went wrong during the merge?
💡 What must a Remediation Playbook do before retiring the duplicate CI?
Answer: B — The Remediation Playbook must re-point all relationships from the duplicate to the master BEFORE retiring the duplicate. If the playbook wasn't configured to handle incident relationships, the incident link to the duplicate CI became broken when the CI was retired. Fix: configure the Remediation Playbook to include relationship re-pointing for cmdb_ci (task_ci table) relationships. After fixing the playbook, the relationship to the master CI can be manually re-established for affected incidents.
5. A CMDB administrator runs a Deduplication Job and finds it has generated 5,000 dedup events in a single evening. CMDB Health was previously showing 5,000 duplicate CIs for this class. The administrator wants to configure a Remediation Playbook to auto-merge all 5,000 pairs tonight. What critical step should they take first?
💡 What happens if the root cause is not fixed before running mass remediation? And what safeguard is needed before bulk automation?
Answer: B — Two critical pre-conditions: First, confirm the root cause (the broken Identification Rule or integration) is already fixed — otherwise Discovery will recreate all 5,000 duplicates within 24 hours of the merge. Second, test the Remediation Playbook on a small pilot set to verify it correctly identifies masters, merges fields, and re-points relationships before running it on 5,000 CIs. A mass merge that incorrectly designates masters or breaks relationship links is extremely difficult to undo and can cascade into reporting and incident management problems.
Exam Checkpoint
Practice Questions
4 questions · Select an answer to see the explanation immediately.
Dedup & Remediation Quiz
1 / 4
An administrator fixes a broken Identification Rule that caused Discovery to create thousands of duplicate server CIs. What is the correct next step to clean up existing duplicates?
💡 Explanation
B is correct. Fixing the Identification Rule prevents new duplicates but doesn't clean up existing ones. The Deduplication Job scans the CMDB for existing duplicate pairs and creates cmdb_dedup events. Then the Remediation Playbook (configured to Merge) automatically resolves them — copying data from duplicates to the master CI and re-pointing all relationships before retiring the duplicates. Re-running Discovery (A) just updates/creates CIs; it won't merge existing duplicates.