All stories

(story)

Fintech

Halyard

Halyard’s four-person security team went from a 1,300-alert weekly queue to two night pages in a quarter. Both were real. Both took one tap.

840 engineers

(quote)

In their words

“In our first quarter, Signal woke me up twice. Both times it was right, and both times the fix was one tap from done.

I’ve started charging my phone in the kitchen.”

Ines Albuquerque

CISO

312,044

alerts read

2

pages at night

0

incidents past containment

Before Signal

Halyard moves money for a couple of hundred payment companies and has 840 engineers, each of whom holds the key to something. The security team was four people and a ticket queue. In the 90 days before Signal, that queue took in about 1,300 alerts a week, and the four of them read as many as they could before Friday.

The tooling was fine. The hours were the problem. Ines Albuquerque, the CISO, was on the rotation herself. A night page meant a forty-minute argument with a dashboard, followed by someone saying it was probably nothing. It was probably nothing about 19 times a month.

The first week

Halyard connected cloud, identity, EDR and git read-only in 34 minutes, most of it waiting for an admin to click through an OAuth screen. Signal read everything and touched nothing. On day one it closed 4,100 alerts and wrote down why for each.

The team did not take this on faith. They pulled 200 closures and read the reasoning. They disagreed with three, and two of those were rules nobody had documented. Then they switched on three playbooks, named an on-call approver and put Ines second in line. Nothing runs without a yes, and everyone checked that twice.

03:12 UTC

The first page came in May. At 03:12:07 UTC, an engineer who had been chatting from Lisbon at 03:05 logged in from Singapore, on a device nobody had seen before. Signal pulled 36 related events: six denied MFA pushes, a seventh approved 2.1 seconds later, and a new OAuth grant, “Drive Sync Pro”, asking for mail.read. No travel on the calendar.

At 03:12:15 the verdict was likely account takeover, confidence 0.94, and the on-call engineer was paged with the playbook attached. She read one card and approved from her phone at 03:12:39. Four sessions were revoked, her MFA factors reset and the OAuth grant removed by 03:12:46. Contained at 03:12:48, 41 seconds after the first event. The engineer woke up at seven to a plain-English note explaining why she was signed out. Nothing had been read.

The second page, in June, was a billing-exports bucket that a terraform apply had made public. The ACL was back to private in 19 seconds.

What changed

In Q2 Signal read 312,044 alerts and woke someone twice. Before, the pager went off at night about 19 times a month. Median time to contain moved from days to minutes, and no incident got past containment.

The less measurable change is the Monday standup. It used to open with a list of things nobody had looked at. Now it opens with the report Signal drafted overnight, which fits on one screen.

What’s next

Halyard is adding a second approver for anything that touches the production database, which approval chains handle without a new playbook. The cloud footprint grows from two providers to three in the autumn, and the quarterly threat review with a human is already in the calendar.

The CISO is off the night rotation. She says she will believe it in a year.

Create a free website with Framer, the website builder loved by startups, designers and agencies.