It started with a file that looked like it was born to be boring: utils/normalize.ts. Fifty-one lines of tidy helpers. It trimmed strings, massaged object keys into camelCase, and carried a couple of TODOs like tiny postcards from a better future. The only thing that stood out about it was the smell, not code smell, instinct. The feeling you get when your front door is locked, but the key turns a little too easily.
I’ve written about technology for a decade and learned to see stories the way security engineers see traffic: most of it is noise until a single packet blinks wrong. 3ear, the small, wickedly smart startup that built a voiceanomaly engine, had a packet blinking wrong. Someone had stolen their mind and hadn’t bothered to leave a signature.
“Signature,” a developer once told me, “Is Tinder for trust. No signature, no relationship.” He meant a digital signature, of course, the cryptographic proof that what you received is exactly what the author sent, when they sent it, and that nobody played surgeon in the middle. I remembered that line when Leo, 3ear’s sleepless CTO, pinged me on a thunderless night: “We’ve got a patch we didn’t write. Looks like us, sounds like us, acts like us… until it doesn’t.”
We met the next morning outside. Too early for people, too late for bugs. Leo slid a laptop across the table, eyes half red, half steel. In the latest pull request, normalize.ts had a one-line change: the order of two tiny functions flipped. Cosmetic, if you didn’t know what to look for. In practice, it lets certain short audio samples skip the outlier check and take a side street through the pipeline.
“What does that buy an attacker?” I asked.
“A quiet backdoor,” Leo said. “No crash, no mess. Just… quiet. One model out of fifteen misses anomalies in carefully chosen frequency bands. If you know our ecosystem, that’s enough.”
“Rogue intern? Drunk contractor?”
He shrugged. “We thought of everyone. But there’s no signature. No cosign. The worst part is that we never enforced signing. It was on the roadmap.”
That word roadmap does a lot of heavy lifting right before the car hits the guardrail. In a perfect world, every meaningful file is signed. Every artifact is signed. Every build is accompanied by a trusted timestamp. In the real world: ship clean, harden later.
Lines of Code, Lines of Testimony
I ordered a double espresso and launched the terminal in my head. When you don’t have signatures, you’re left with forensics from the graylands: who opened the PR, from which IP, at what hour. Leo showed me the Git logs: a stream of commits wandering the desert of plausible deniability. Usernames can be forged, email identities can be minted, and system clocks are as honest as whoever patched them at two a.m.
There it was: chore/refactor-normalize. Author: netaly.dev@3ear.ai. Netaly was a veteran dev who had left a month earlier. “It wasn’t her,” Leo said. “I’d sign on that… if I had anything to sign with.”
We traced the timeline. The commit landed five minutes after the CI runner declared version 1.4.2 gloriously green. The patch slipped between builds, sailed past a bot review, and auto-merged into main. 3ear loves automation: fewer humans, fewer mistakes. Irony jumped the fence, waved, and vanished into morning traffic.
“You’re missing code signing,” I said. “Artifact signing. And timestamps.”
“Yeah,” Leo sighed. “We punted. MVP.”
That night, half the internet broke into arguments about politics and crypto, and in a small research forum, someone wrote: “Silent patch slipped into 3ear.ai pipeline.” Monopoly lost: someone knew exactly where to look.
The First Twist: Check the Mirror
Good thrillers hate straight lines. In the morning, I got an email from an anonymous sender. Subject: It’s not the patch, it’s the mirror.
Two sentences and a line of code. Check the registry. And: Whoever holds the mirror holds the truth. The code was an API call for pulling tags on 3ear/voice-search: 1.4.2 from their container registry.
Enter Tom, a DevOps lead with the permanent look of a man who sleeps with a pager instead of a pillow. “Nobody stole our code,” he said. “They stole our history.”
He opened their registry. Next to 1.4.2, three friendly-looking tags pointed to the same digest. But the manifest told a different story: the top layer had been nudged minutely, precisely, after two in the morning. A change visible only to a surgeon’s lamp and a forensic diff.
“Show me the signatures,” Tom said and stopped. He didn’t have to finish. There were no signatures.
Where cautious companies attach cryptographic signatures with keys locked in an HSM, 3ear had an email address. A good one, with a beautiful domain. Compared to a trusted timestamp from a real-timestamping authority, it’s a postcard without a stamp.
“Remember the mirror?” I asked.
Tom nodded. “Someone stood up a look-alike, probably a proxy fronting our registry during a DNS hiccup in the dead hours. Clients and investors running docker pull in that window got a slightly… edited layer. And without signature enforcement, who would suspect?”
“How long?”
“Forty-three minutes.” He pointed at a bandwidth graph. “A short link in a long chain long enough to seed doubt into every morning demo.”
So, it wasn’t a drunk contractor or a rogue intern. This was a smart crime. Don’t change the diamond in the vault; rent a perfect glass copy to the museum’s visitors.
When Git Forgets
“But if they didn’t change the source, why does the commit look forged?” I asked.
“They did change something,” Tom said. “Not code consensus.” They created a mirror branch, ran a build on it, and in the ten seconds before the real merge, the hammer swapped anvils. It looked like Netaly’s commit because an old CI bot key was still trusted, and someone coaxed it into declaring “I’m Netaly.” One legacy server held a GPG identity that should’ve been revoked and wasn’t.
In expert witness jargon, it’s called “a trust chain cut too short with no guard rails.” It also sounds like a hemline you regret the minute you step outside.
“Inside job?” I asked Leo after Tom left to fight another fire.
Leo put on his poker face. “Someone knew where we were two weeks ago, where we hadn’t closed a screen. Doesn’t have to be internal. Could be a competitor. A partner. Or someone who decided we were moving too fast.”
Two weeks earlier, 3ear had pitched a media giant on smarter podcast search. In the partner’s office, someone offhandedly mentioned “anchorstyle signing in the cloud” that “saves key headaches.” I didn’t ask whose anchor. Reporters pretend we’ll be smarter later; usually, we’re just later.
The First Time I Heard About Time
The next clue came from Lana: two decades of code, one decade of cryptographic signatures. “You’re living without trusted timestamps?” she raised an eyebrow. “In 2025?”
“It was planned,” Leo recited the company mantra.
Lana didn’t argue. She just opened a folder: a client case where file sources, binaries, and Docker images were signed with keys held in an HSM, and each signature was sealed with a timestamp from a trusted authority. “It’s not about a product they bought,” she said, “it’s the discipline those systems forced. You can’t later claim a file existed earlier than it did. And you can’t rewrite history without leaving shards.”
“What if someone steals the private key?” I asked.
“They don’t,” she said. “Because the private key never leaves the hardware. Remote signing only. Every call is audited. Certificate lifecycle managed end to end.”
She wasn’t pitching. She was describing hygiene. That’s the difference between marketing copy and oxygen. Some vendors know how to do this properly, including Comsigntrust, whose services several of her clients used from the code all the way to legal documents, with HSM-backed signatures and authoritative timestamps. “The point,” Lana said, “is that the experience of the system erases the fear by design,” I asked for a phone charger and got a master class.
“What now?” Leo asked.
“Now you freeze the world,” Lana said. “Short freeze. Rebuild the trust chain: code signing, artifact signing, and authoritative timestamps at every seam. CI refuses to build unless the request comes from the HSM. CD refuses to deploy unless signatures and time stamps verify. The registry enforces signature-only pulls. No signature, no ride.”
“And the damage already done?”
“Luckily,” she said, “your attacker was a surgeon, not a butcher. Which means they left traces. With the right time-stamped anchor, something you did sign somewhere, maybe a proposal PDF, a model snapshot, you can pin a timeline nobody can argue with.”
An Edge of Story Inside the Edge of a File
Luck is a bad habit but a good alibi. 3ear had an annoying practice that turned out to be a lifeline: for investor demos, they shipped a one-page “cover letter” listing the release and its checksum, so the recipient could verify later. Ninety-nine times out of a hundred, nobody checked. But one particular investment firm was always a heartbeat ahead, and a foot on Highway 6 kept their bureaucracy sharp. Their PDF had been digitally signed and time-stamped by an external authority. By chance, the time-stamping service was integrated through Comsigntrust in their back office. Not for code for documents.
“If we have a trusted timestamp that contains the hash of the version we demoed,” I said, “we can prove that whatever was pulled later wasn’t the same.”
“And then?”
“Then whoever rented the mirror meets the stick.”
Tom extracted the hash from the PDF. It was a SHA256, long and straight like an empty highway at night. We compared it to the registry’s digest. No match. The demo artifact existed earlier in time than the silent patch. The story fell into place the way objects fall when you finally turn gravity back on.
The anonymous sender returned: this time with an IP for a server in Eastern Europe and a temporary login from a machine named with a fairytale wink: mirror-mirror. It would’ve been funny if it weren’t true. I replied; a second later, I got: check with the partner.
The partner. The media company. The deputy head of security, polite and wintering in Iceland in August, took my call. “Our files are tightly sealed,” he said. “Two chains of audit logs. But your registry? That’s yours.”
“And DNS?”
Silence, dense and familiar. “We had a maintenance window. Secondary provider. It’s possible someone pushed a button that used to be called something else. Not malicious on our side. But… someone smart might have noticed.”
Who Benefits
Old rule of fraud: follow whoever profits. 3ear is too small for big enemies. But sowing doubt before a funding round? Holding a stage for one more month? Money loves delays.
We got there the way you always do, through invoices. A small vendor selling “cloud signing” at a price that doesn’t exist in physics. Dig a little and you find a reseller or the shadow of one peddling a “signature solution” that isn’t. No real HSM. No certificate lifecycle. A “signature” that was just a private key file sitting on a remote VM, no timestamp, no root of trust.
That vendor, it turned out, also “advised” the partner’s CI/CD. Industry evolution in its wildest form: mixing quality with illusion.
“All this because we didn’t sign?” Lior asked, like a father wondering if the kid got sick for not wearing a jacket.
“Because you didn’t build a chain of trust,” I said. “Signatures are the jacket. But so are DNS controls, registry policy, and CI identity. Without guard rails, a breeze becomes a storm.”
How You Sign a Story
Magazines want stories, not internal postmortems or criminal indictments. The trick is to turn lessons into something a human can read without a Gantt chart.
This story’s hero isn’t the person who saved the day; it’s the moment that fixed time. Think of a trusted timestamp like your grandmother’s pendulum clock, slow, steady, unimpressed by football noise. When you attach that moment to a file’s hash, you protect it from the future. Anyone claiming “this file existed then” has to bring a better clock than yours. They usually don’t.
The signing stack itself is a symphony. A private key that never leaves its brass instrument, the HSM. A certificate lifecycle that keeps today’s players honest tomorrow. A policy that refuses to move unless everything is in tune. Providers like Comsigntrust run this orchestra for organizations that don’t have time to build it from raw metal; not the only name in town, but a name that tends to lower my pulse when I see it in the architecture review.
At 3ear, after a long night, CI changed. Builds required remote signing via HSM. Every artifact was signed. Important documents were time-stamped. The registry refused unsigned pulls. Anyone pretending to be someone else met a gate that didn’t care about pleasantries. Paradoxically, they got faster. Noise drops, speed rises.
The Last Twist: Who Wrote the Emails
A day after we pushed the corrected layer and hardened DNS, our anonymous friend emailed again. “Nice,” the message began. “Now check the Voice.” Two words that rang a bell the size of a container ship. The Voice at 3ear is a module John wrote, a quiet developer, with rock-solid code. He never touched normalize.ts. He made everything else look unimportant.
The Voice hadn’t been stolen. It had been bait. The attacker knew that if anyone touched the Voice, John would feel it. So, they nudged normalize.ts instead. Every email they sent us carried a hidden watermark and an import string that spelled the name of an internal NPM package at a competitor. A catflap in the door.
“So, they were testing whether we were pulling their code?” I asked.
“Or testing whether we’d notice the fish they tossed,” Leo said. “Hang on.”
John arrived compact, calm, smiling with half a mouth. “Your emailer,” he said, “is a bot.” He showed us a small script that profiled the writing: perfectly round second marks on send times, the same typo in two languages, and the kind of apology pattern you see in a fine-tuned LLM instructed to be curt in English and deferential in French. “They spoke machine,” he said.
That revelation doesn’t make a criminal less criminal. But it drains the desire to imagine one face to hate. You’re left with a totem of modern industry: automation versus automation.
“What now?” I asked Leo.
“Now we clean,” he said. “Then we tell the story. Tuition paid by us.”
Epilogue: Where Stories Live
Some features are written under low lamps, some on park benches. This one got both. For three weeks, we trailed shadows and forced their names onto a wall. We learned that code without signatures is like music without copyright; you can enjoy it, but you can’t live off it. Documents without time stamps are sticky notes on a fridge, forever open to debate about when they were written. A voice without verification is a rumour.
No team needs to turn into a monastery of cryptographers. But every team needs the minimum: digital signatures for anything that ships. Certificate and credential management that treats keys like surgical tools: issued, rotated, revoked, logged. Authoritative timestamps for every version you share with the world. And it helps to use services that already know how to thread code, documents, and hardware keys into one chain, such as Comsigntrust, among them.
Like all decent thrillers, this one ends with a quiet before a few more shots in the distance. A month later, at a DevSecOps meetup in a basement with bad coffee and excellent sarcasm, I noticed a black T-shirt with an old print: Trust, but verify. The person wearing it raised a paper cup in my direction. Maybe it was Tom. Maybe it was the vendor who vanished. Maybe another mirror. I raised my cup back. We live in a world where truth is measured in hashes, trust in chains, and stories in time stamps. It’s less romantic than it used to be. And still, press your ear against the glass and you’ll hear the heart of modern systems beating: tick, tick, tick. Sign, sign, sign.
