TL;DR / Summary: HMAC signatures over query strings fail when two sides serialize parameters in different orders. Sort keys with a documented rule, encode values the same way both clients do, then hash the canonical string. Pretty printers and “helpful” URL cleaners that reorder params will invalidate a signature that was correct five minutes earlier.
Why order becomes part of the secret contract
An HMAC does not understand HTTP. It understands bytes. If one client builds a=1&b=2 and the server rebuilds b=2&a=1, the digests diverge even when the logical map is identical. Teams discover this only after a “harmless” refactor that swapped map iteration order.
Language runtimes differ. Some hashes preserve insertion order. Older ones do not. Mobile SDKs may sort; browser URLSearchParams may not match your backend’s Go map walk. Treat order as an explicit step, never as an accident of the dictionary type. Write the rule into the partner onboarding PDF so the next engineer does not “improve” the client into intermittent 401s.
When onboarding a second mobile platform, port the canonicalization tests before the first production traffic. A Kotlin Map and a Swift Dictionary will not invent the same iteration order just because both teams read the same prose spec. Shared fixtures beat shared optimism.
Canonicalization rules worth writing down
Common rules: sort keys lexicographically by Unicode code point, join with &, use key=value with a single equals, and percent-encode using a named RFC. Document whether empty values appear as key= or are omitted. Document whether keys are case-sensitive. Decide whether absent optional fields are dropped or serialized empty; a client that drops nulls and a server that emits coupon= will disagree forever.
Boolean false and numeric zero are values, not absences, unless the spec says otherwise. When a new optional field ships behind a feature flag, both sides must agree whether unsigned extras are ignored or whether any extra key invalidates the signature. Strict APIs prefer a fixed allowlist of keys in the signed set so surprise parameters cannot be injected after the digest is computed.
Encoding traps that look like order bugs
Space as %20 versus + changes the byte string. So does encoding a slash inside a value, or double-encoding a value that was already escaped. Before you blame sort order, dump the exact preimage both sides hashed. A URL encode utility is useful for comparing one value at a time, but the signature preimage is the whole joined string.
Arrays are another silent fork: tag=a&tag=b versus tag[]=a&tag[]=b versus a comma-joined single key. If the API allows repeated keys, define the repeat order after key sort—usually stable by original sequence among equals. Include an unsorted fixture in CI so deleting the sort call fails a test immediately.
A failing case with three parameters and a nonce
Client sends timestamp, user id, and action. Insertion order on the phone was action, timestamp, user. Server sorted lexicographically and matched by luck. A later release inserted nonce. Phone still appended; server sorted. Signatures broke only for requests that included nonce.
The fix was not “more retries.” It was logging the canonical string behind a redacted flag in staging, then forcing both sides through one shared sort helper. Once the helper owned ordering, feature flags that add parameters stopped being signature events. Ship that helper as a versioned package so mobile, web, and backend cannot drift one merge at a time.
Body signatures, gateways, and rewritten URLs
Some APIs sign the body only. Others sign method plus path plus sorted query plus body hash. Copying a query-signing routine onto a JSON body without clarifying separators creates a hybrid nobody else can reproduce. Keep query canonicalization in one function and body hashing in another, then compose them to match the vendor diagram exactly.
API gateways, WAFs, and link cleaners sometimes drop empty parameters or reorder query keys for cache efficiency. That is fine for cache keys and fatal for HMAC preimages. Sign before any normalizing proxy, or configure the proxy to leave the query string byte-stable. Locally, avoid pasting a signed URL into a browser bar that auto-encodes characters again; prefer curl with a single-quoted URL.
Clock skew is adjacent, not identical
Timestamp fields inside the signed payload stop replay, but they do not fix order bugs. A skew window of five minutes can make a correctly ordered signature look dead if devices drift. Check NTP on both ends after you have proven the canonical strings match. Mixing the two incident classes in one ticket doubles mean time to repair.
Log the received timestamp beside a hash of the canonical string in staging. Failures that cluster around minute boundaries point to clocks; failures that cluster around deploys point to serializers. That split keeps cryptography tickets from absorbing every auth failure.
Practical checklist before you blame the secret key
Compare canonical strings first, then hex versus base64 digest formatting, then clocks, and only then rotate keys. Key mismatch is real, but order and encoding cause more “it used to work” tickets than silent key corruption. When two environments disagree, paste both preimages into a text diff or hash helper and look at the first differing character.
The fix is usually a sort, an encoding alphabet, or a stray ? included in one preimage and not the other—not a new cryptography library. Store golden fixtures: input map, canonical string, hex digest for a fixed key. Run them on every client language you ship so the next encoder change fails in CI instead of in a customer dashboard at lunch.