JSON · May 28, 2026 · 8 min

Remove Duplicate JSON Keys: Detection and Prevention

Understand how parsers handle duplicate keys, detect them in your data, and prevent subtle bugs.

The Duplicate Key Problem

JSON objects are collections of key-value pairs, but the JSON specification does not define what happens when the same key appears more than once in a single object. RFC 8259 states that "the names within an object should be unique" but does not mandate parser behavior for duplicates. In practice, most parsers silently keep the last occurrence and discard earlier ones — but this is implementation-dependent, not guaranteed.

Duplicate keys create subtle bugs. A configuration file with both `"port": 8080` and `"port": 3000` might load port 3000 without any warning. An API payload with duplicate user IDs might pass validation but resolve to unexpected values. Security researchers have exploited duplicate key handling differences between parsers to bypass input validation.

Detecting duplicate keys requires going beyond standard `JSON.parse()`, which typically does not report them. Use the JSON Validator for syntax checking, then inspect objects programmatically or with the JSON Viewer to spot suspicious patterns in your data.

How Different Parsers Handle Duplicates

JavaScript's `JSON.parse()` in V8 (Chrome, Node.js) keeps the last value for duplicate keys. Python's `json.loads()` does the same. Go's `json.Unmarshal()` into a map also keeps the last value. However, parsing into a structure that preserves order (like Python's `collections.OrderedDict` with a custom hook) can reveal all occurrences.

Some strict parsers reject duplicate keys entirely. PostgreSQL's `jsonb` type does not allow duplicate keys — inserting JSON with duplicates normalizes to unique keys. If your pipeline includes multiple parsers (a web server, a message queue, a database), they may disagree on the final value.

The safest approach: never rely on duplicate key behavior. Treat duplicate keys as errors in your validation pipeline. Parse with a custom reviver or decoder that tracks seen keys and raises an error on the second occurrence.

Detecting Duplicate Keys

Standard `JSON.parse()` will not warn you about duplicates. To detect them in JavaScript, use a reviver function that tracks keys: maintain a Set of seen keys at each object level, and throw an error when a key is encountered twice. This adds minimal overhead and catches the problem at parse time.

In Python, parse with `object_pairs_hook` to receive key-value pairs as a list before dict conversion. If any key appears twice in the pairs list, raise an error. This approach preserves the ability to see all duplicate values for debugging.

For manual inspection, paste JSON into the JSON Viewer and look for keys that appear multiple times at the same nesting level. Format with the JSON Formatter first to make the structure scannable. For programmatic detection in CI, add a duplicate-key check step after syntax validation.

Removing Duplicate Keys

If you need to clean a JSON document by keeping only the first (or last) occurrence of each key, parse with a custom handler that enforces uniqueness. In JavaScript, a reviver that skips already-seen keys produces a clean object. In Python, iterate `object_pairs_hook` results and build a dict keeping the first occurrence of each key.

When duplicate keys exist at different nesting levels, they are not duplicates — `{"user": {"name": "Alice"}, "name": "Bob"}` has two different `name` keys at different depths, which is perfectly valid. Only same-level duplicates are problematic.

After cleaning, validate the output with the JSON Validator and format with the JSON Formatter for a clean, readable result. Compare the cleaned output against the original to verify no intended data was lost.

Duplicate Keys as a Security Concern

JSON duplicate key attacks exploit parser inconsistencies between systems. An attacker sends JSON with duplicate keys to a web application firewall (WAF) that parses with one engine, while the backend application parses with another. The WAF sees one value (e.g., a safe URL) while the backend processes a different value (a malicious URL).

Mitigate this by using the same JSON parser throughout your stack, rejecting documents with duplicate keys at the API gateway, and validating payloads against a strict JSON Schema that defines exactly which keys are allowed.

Never parse JSON from untrusted sources with lenient parsers. Webhook payloads, user uploads, and third-party API responses should all pass through duplicate-key detection before processing. Combine with the validation techniques in How to Validate JSON.

Preventing Duplicates at the Source

The best fix is prevention. Use JSON Schema validation to define allowed keys — duplicate keys in the raw text will still parse, but schema validation catches unexpected structures. Linters and pre-commit hooks that check JSON files catch duplicates before they enter your repository.

When merging JSON objects programmatically, use explicit merge strategies instead of string concatenation. `Object.assign()`, spread syntax (`{...a, ...b}`), and lodash's `merge()` handle key conflicts predictably — later values overwrite earlier ones by design, with no ambiguity.

Document your team's policy: if duplicate keys are found, is the first or last value authoritative? Enforce this policy in code rather than relying on parser defaults. Consistency across services prevents the class of bugs that only appear in production under specific parser combinations.

Tools and Workflow

A practical workflow: receive JSON, validate syntax with the JSON Validator, check for duplicate keys with a custom parser or linter, format clean output with the JSON Formatter, and validate against JSON Schema. Each step catches a different class of problem.

For exploring suspicious JSON visually, the JSON Parser shows structure alongside validation status. The JSON Viewer tree view makes it easy to spot keys that appear more than once at the same level.

Understanding JSON syntax fundamentals helps contextualize duplicate key behavior. Read What is JSON? for the complete type system and syntax rules. For related parsing pitfalls, see JSON Escape Characters Guide.