You are currently viewing Deduplicating Config Lines Before a Diff

Deduplicating Config Lines Before a Diff

TL;DR / Summary: A config diff full of duplicated lines is usually a merge artefact, not a real change. Sort and deduplicate each file into a working copy, compare the cleaned copies, then map any real difference back to the original line numbers before editing anything users depend on.

Why the diff looks noisy after a merge

Version control merges keep both sides when a conflict is resolved by hand. The result often contains the same setting twice: one copy from each branch, identical in value but separated by other lines. The file may still behave correctly because the last assignment wins, yet every future review drowns in noise.

Duplicate keys are also common in config formats that allow them without complaint. YAML parsers often take the last value silently, and shell-sourced env files simply overwrite earlier exports. The file looks longer than it is meaningful.

Before deleting anything, decide what the file is supposed to guarantee. If it is a list where order matters, deduplication is the wrong tool and sorting is worse. Those files need a structural diff, not a text trick.

Generated files need one more check. If a template produced the duplicates, the next build puts them back, and a manual dedupe becomes a recurring chore. Fix the generator, or have the file reviewed as an artefact rather than as source.

Sort, dedupe, then sort back

Make a copy, never work on the live file. Sorting groups identical lines together so duplicates become visible and removable in one pass. After deduplication, sorting back by key restores a predictable order that makes the next review easier.

Most editors and command-line pipelines can do this in one line. The important part is keeping the copy separate so the original line numbers survive for the mapping step later.

Count the lines before and after. A file that drops from four hundred lines to three hundred and eighty tells you something concrete: twenty duplicated settings, not twenty deletions of intent.

Keep the comment lines you still need

Blanket deduplication will happily collapse two identical comment lines that explain two different blocks. Comments are documentation, and a repeat in a different section is often deliberate.

Separate comments from settings before the pass. Deduplicate the settings, then restore the comments in their original positions. If a comment sits directly above a key, treat the pair as a unit so the explanation does not end up attached to the wrong setting.

Inline comments are worse. Two identical lines with different trailing comments may look duplicated after the comment is stripped, but they were never the same line. Strip, compare the key and value, and decide with your eyes rather than with a flag.

Duplicate keys with different values

These are not duplicates at all, they are a bug. Two values for the same key mean one environment is silently ignoring the other. List them separately and resolve which one is intended; a dedupe pass that keeps only the first occurrence can flip production behaviour.

Whitespace and line endings defeat naive dedupe

A line ending in a carriage return is not equal to the same line ending in a newline. Files edited on Windows and then moved to Linux routinely contain both, and a dedupe tool that compares raw bytes will treat them as different settings.

Normalise line endings in the working copy first, then compare. Trailing spaces and tabs produce the same false negatives. Indentation matters in YAML, so normalise only the trailing whitespace there, not the leading whitespace.

Encoding is the last trap. A non-breaking space pasted from documentation looks identical on screen and breaks a key comparison. Convert to a single encoding, inspect the suspicious lines in a hex view if needed, and only then run the pass.

Comparing two files without destroying the original

Keep three artefacts: the original copy, the sorted copy, and the deduplicated copy. Compare original against deduplicated to see what the pass removed, then compare the two deduplicated files from different branches to see what genuinely differs.

This three-way habit answers the question reviewers actually ask: did the merge duplicate a line, or did someone change a value while duplicating it? One comparison cannot distinguish those cases.

A clean diff is evidence. A rewritten file is only a claim.

For quick passes, the text utilities at Masters Tool sort and deduplicate in the browser, which is handy when the file contains internal hostnames you would rather not paste into a hosted service.

When duplicates are meaningful

Ordered lists, repeated array entries, and multi-value headers can legitimately contain the same string twice. Deduplicating them changes semantics. Check the schema before treating repetition as an error.

Include files are another case. The same line may appear once in the base config and once in an override, where its position is what makes it an override. Merging the two files into one text blob erases that distinction.

Certificate chains and public key blocks also repeat similar-looking lines that are not interchangeable. Structural comparison wins over text comparison whenever the format has real structure.

A repeatable pass for the next config change

Write the steps down where the next person can find them: copy, normalise line endings, strip or park comments, sort, deduplicate, count, restore comments, compare against the original, and only then edit the live file.

The pass takes a few minutes and removes the most common reason a small config change looks terrifying in review. It does not decide which value is correct; it only makes the question visible.