Back in 2023, I led a massive detection stack move to Google Chronicle (now SecOps) at a top-notch fintech. Before that, the team ran on Splunk. Beyond a SIEM, Splunk is a data and log processing platform that reaches well past security, and at our company it was an engineering tool used by many teams, Security included. We ran our detections as scheduled queries over a shared pool where everyone shipped every log they could think of to query later. The house rule was one sentence long: log everything — we might need it someday.
That rule has a bill. It showed up as slowness and cost, and that bill was part of why Security went looking for a dedicated, purpose-built SIEM in the first place. Chronicle won the evaluation and arrived carrying something the team had never had to respect before: a fixed annual ingestion license.
Splunk’s world had felt effectively unmetered, especially because Security was just one tenant in a company-wide engineering contract. You paid for volume in performance and invoices, but no counter ever stopped you. The new world was metered: every byte draws down a number. Nobody had internalized that yet. I certainly hadn’t. Two months later, we had burned through five times our annual license.
The Problem
CloudTrail was one of the first sources we onboarded, and for good reason: it fed more custom detection rules than anything else the team had. So it went in early, and it went in the old way — pointed at the SIEM, no one watching the meter.
About two months into the deployment, still in phase one, I pulled our ingestion figures out of curiosity more than concern. Seeing five times our annual license on the dashboard stopped me cold. Not five times the monthly rate. Five times the year in sixty days, on a deployment that wasn’t even finished.
The first instinct is to panic or start deleting things. I did neither. Before you fix a number like that, you have to know where it comes from, and the answer was almost embarrassingly clear the moment I broke ingestion down by source. One outlier towered over everything else. It was CloudTrail.
To be honest, as early adopters we decided to start small, so our first contract was just a few terabytes of logs per year.
Immediate Actions
The next call I made was to Google. Not to ask for a discount, just to be straight about where we were: early adopters, no deployment partner, still mid-rollout, and already way past the line. They were reasonable about it. Given the stage we were at, the overshoot wasn’t going to become an invoice; they treated it as a rollout lesson. That bought me the one thing I actually needed: time to fix it properly instead of hacking something together over a weekend.
Then I did the unglamorous part, the part before the math. I brought the team together and put the whole problem on the table: here’s what happened, here’s why, and here’s the plan — stop, filter, test, restart. We split the work along the obvious lines: teammates would stop ingestion, code the filter, and restart the feed later; I’d own the strategy behind it — which logs to keep, the engineering to prove the case, and the validation once it was live. I gave the SOC managers an immediate heads-up before we paused the feed and promised to bring them the exact keep-and-drop list for sign-off before anything went back into production. Nobody likes surprises, least of all the people who own the alerts you’re about to change. Everyone agreed. Then I went and did the math.
Data Analysis
CloudTrail is chatty because AWS is chatty: every API call is an event, and most of those events are nobody’s idea of a detection — and they’re JSON, not simple unstructured Syslog payloads, which translates into more bytes per log. So the question wasn’t whether to filter, but on what. You want the cheapest field that cleanly separates the logs you need from the logs you don’t, and for CloudTrail that field is eventName. It maps almost one-to-one onto what detections care about, and it’s enumerable — you can list every value you’re seeing and reason about each one.
So I worked it from both ends.
First, what did we actually need? I went through our detection rules, still written in SPL at that point, and pulled every eventName they depended on. Then I widened the net a little using the documentation and Google searches, adding the calls that weren’t in a rule yet but that you’d be sorry not to have the day something went wrong. That was the keep list.
Second, what was flowing in production? In Splunk, I pulled every eventName seen over the previous 60 days (the maximum retention period at the time) and how many events each one had generated. That’s the real distribution, not the one you assume you have. I left the query running overnight and calmed my anxiety playing Zelda: Breath of the Wild. The next morning, I got the email from Splunk telling me the query had finished, and I was able to proceed.
The rest was arithmetic. I lined the keep list up against that 60-day count, matched them, and summed the total events for the exact set I wanted over those 60 days. Dividing by 60 gave me events per day. A bit of digging gave me the average size of a CloudTrail event, and from there it’s just multiplication — events per day, times bytes per event, times 365. That’s the yearly bill. And I had the volume per eventName.
I ran the number. It was still too high.
Strategy Fine-Tuning
Here’s where it would have been easy to keep trimming event names until the arithmetic came out under the license and call it engineering. I’m glad I didn’t, because trimming by volume alone would have told me what was expensive without telling me whether it was worth it.
So I turned the analysis around and looked at the detections themselves, asking what each rule was worth rather than what it cost. And the two turned out to be almost perfectly, uncomfortably correlated: the event names driving the most volume were feeding the least precise rules. A couple of dozen of them sat at effectively zero (0x00) precision — every alert they’d ever produced was either a false positive or a benign true positive triggered by routine platform activity. Precision here is just actionable true positives over total alerts, measured across a long-enough window that the number means something.
The detection team there had just been formed, so although we had around 300 custom rules, the team used to focus on new rules rather than existing ones — the infamous set-and-forget problem. The SIEM deployment kicked off a new threat detection era with a clearer rule lifecycle process — even if it wasn’t mature at the time. After that, the detection team matured alongside the SIEM.
The worst offenders were high-frequency S3 data events (GetObject, HeadObject, ListObjects) — the read chatter of every app and automation touching a bucket, firing nonstop and feeding custom rules that had never caught a real intrusion. And I want to be clear here, because dropping S3 data events sounds reckless: we weren’t going blind on S3. Control-plane mutations (a bucket turning public, an ACL or policy change, replication tampering) ride on low-volume management events, and those stayed on the keep list. Meanwhile, anomalous data-plane reads and exfiltration were already covered by AWS GuardDuty, which evaluated S3 read behavior inside AWS and forwarded only the resulting findings to the SIEM. Shipping billions of raw GetObject calls into Chronicle just to run noisy threshold queries of our own was duplicating work at the most expensive layer in the stack. We cut the raw read noise, not the signal.
That reframed the decision completely. We were spending the bulk of our brand-new, tightly licensed budget on rules that caught nothing — paying premium SIEM rates for alerts our analysts already ignored. We walked the SOC managers through the numbers and retired most of those zero-precision rules. The byte savings were a side effect of doing the detection audit the migration had finally forced on us.
99%
With the noisy rules gone, I updated the eventName allowlist and reran the model. The final set filtered about 99% of CloudTrail’s volume and kept every rule the team relied on. The 1% we let through was the 1% that did the work.
We couldn’t just disable those noisy events at the AWS CloudTrail source though: every trail exported first to an S3 bucket in a centralized AWS logging account, where engineering teams used the raw logs for troubleshooting and compliance required full retention. Filtering had to happen on our SIEM consumer path — CloudTrail -> S3 -> EventBridge -> SQS -> Chronicle.
I handed the keep list to a developer on our team, who built the control as a Python worker sitting between SQS and Chronicle. Whenever EventBridge and SQS notified on a new CloudTrail batch in S3, the worker pulled the object, filtered the records against our eventName allowlist, and forwarded only the matches to Chronicle. Instead of paying to ingest everything and discarding noise inside the SIEM, we dropped it upstream before it ever touched the meter. He applied the list, we re-enabled ingestion, and it just… worked. No drama, no scramble. Boring, which is the highest compliment I can pay a data pipeline.
Validation
I don’t trust a model until reality has had a chance to embarrass it, so I watched this one closely — daily at first, then weekly once it settled. After three months, our actual CloudTrail consumption had landed almost exactly on the number I’d predicted from a 60-day Splunk sample and an average event size I’d looked up online.
I’ll be honest: I expected to be roughly right and I was nearly exact, and part of me still finds that slightly suspicious. But the method is sound: the distribution of event names is stable, event sizes don’t wander much, and when your inputs are steady your prediction has no reason to drift. That allowlist went into production during the deployment and coupled our telemetry intake to detection engineering from then on: a new eventName only entered the pipeline when a new rule needed it — and we had the spreadsheet I’d created to predict how many bytes the new events would consume. Because we had already padded the initial list with the high-signal control-plane calls worth investigating, the core filter ran in production with barely a change until I left the company three years later.
Bottom Line
“Log everything, we might need it” sounds responsible. In practice, it’s wishful thinking. Budgets are finite, and per terabyte, SIEM ingestion is about the most expensive place you can put a log in a SOC — it’s the same reason almost nobody forwards their full EDR telemetry to the SIEM. So at some point the “everything” runs into the invoice, and something has to give.
Filtering is often how you win that fight, but truth be told: filtering should be the last resort, not the first. Every filter you add is a decision to stop seeing something. Most of the time it’s noise you’ll never miss, but you don’t get to know that for sure, and if the filter is the whole story, the day you need that log it isn’t there. Ours wasn’t quite that stark: the raw CloudTrail still landed in S3, so nothing was really gone. But “it’s in a bucket somewhere” is cold comfort mid-incident — compressed JSON buried across a maze of subfolders is retained in the technical sense and painful in every other. The data exists; reaching it fast is a different problem. That’s the trade, and it’s real.
Which is why the math was never the hard part. The hard part is that dropping data is a risk, and a risk like that isn’t an engineer’s to accept alone — it belongs to the people who own the consequences. That’s the whole reason I walked the SOC managers through it before touching anything: I could tell them exactly what we’d keep, what we’d lose, and why the loss was worth it, but the call to accept that blindness was theirs.
Do the analysis, couple volume to value, and filter at the source on the cheapest key you can — but put the trade in front of the right people and let them own it. That’s the part that outlives Chronicle, or CloudTrail, or that particular fintech. Sooner or later your security data stops being free, and when it does, someone has to decide what you can afford not to see.