The Book

Practical AI Security is written by Harriet Farlow, an Australian AI security researcher with a PhD in adversarial machine learning. It starts beautifully, walking through the history of AI and its basic concepts, starting with the 1956 Dartmouth conference and the perceptron of the late 1950s. Farlow shows real command of the field in the first part of the book, the opening three chapters. She takes the reader by the hand and backs everything with code you can run yourself through the 30-plus notebooks on GitHub.

Once the foundations are set, she links what you’ve just learned to security, from both the defensive and offensive sides, with the offensive side explored more deeply. She leans on well-known frameworks to do it: MITRE ATLAS, OWASP LLM, and the Cloud Security Alliance’s MAESTRO for multi-agent systems. One of the best sections in the book is a threat-modeling masterclass. You start with a clear end-to-end diagram of the system, then walk each data flow with those references in mind to surface the weaknesses.

Two ideas stuck with me and are worth stealing even if you never read the book. First, a simple taxonomy for AI attacks: every one of them tries to disrupt (make the model fail), deceive (make it see something that isn’t there), or disclose (leak training data or parameters). Second, the ML lifecycle as an attack surface. Data, training, inference, monitoring: each stage hands an adversary a different opportunity. She even shows some clever examples of adding noise to images so AI models read them as something completely different.

In the three final chapters, she goes beyond security into AI safety and its implications for our society: AI replacing the human workforce, the environmental cost of the data centers behind this expansion, governance, and alignment. It’s all philosophical, but more than welcome.

Impressions

I really liked the first part of this book. The language is accessible and easy to follow, she takes the time to contextualize subjects when she introduces them, and the “Resources” list at the end of each chapter is spot on, with a lot of great sources that genuinely extend the read. She also teaches from real incidents, which is where the book shines: the Cylance malware-detector bypass, the ID.me facial-recognition fraud, the AI cameras that missed poachers and cost four rhinos their lives. Learning from real cases beats abstract theory every time.

At that point it was clear to me this was a book written for security people who want to learn about AI. Exactly my case. But then the security chapters arrived, and it started reading like a book for AI practitioners who want to learn about security. And that’s not me being confused: the publisher says as much. The book targets “a security practitioner learning to operate in AI environments, or an ML engineer who needs to understand what adversaries actually do.” So it’s aimed at both by design. My problem is that this dual mandate is exactly what keeps the security half shallow. To keep the ML crowd on board, she re-explains things like what a SOC is or how to read a log line, and there’s no room left to go deep for the security crowd.

My bigger complaint is about who the book pictures as its reader. For almost the entire book, the security concerns are about teams that build and train their own models, like data quality, poisoning, moving models to production safely, and protecting the training pipeline. That’s real and important. But if you outsource the model (you just call GPT or Claude), your concerns live somewhere else: in the architecture around the model. And that’s the part I came for and mostly didn’t get.

The wrapper around that outsourced model is what I wanted. How do you wire a chatbot to an LLM safely? What does defense in depth look like there? How do you log, prevent, and detect malicious input and output? How do you protect a RAG store from poisoned data, or put an LLM gateway or firewall in front of the whole thing? How does an LLM firewall even work? None of that really gets covered.

The same goes for using AI to do security, which was admittedly my own misreading of the book’s scope, but still. Chapter 7, “Attacking and Defending with AI,” was the biggest letdown. The SOC section was shallow enough that, as someone who works in a SOC, I got almost nothing from it. For instance, at some point she shows a few atomic log lines, then a single enriched incident, and essentially says “AI correlated it.” How? And better: we have tools like Vector capable of merging log lines. You don’t need AI for that.

The interesting conversation right now in blue teams is the agentic SOC (agentic triage, agentic detection, agentic response) and how to leverage AI to mitigate risks faster. Running an agent on every single alert is expensive, so you start with deterministic triage and put the agent later in the pipeline. That trade-off is the meat, and it’s missing. Same on offense: a red teamer or a threat actor can chain an agent that triggers nmap or nuclei, reads the output, decides the next tool, and repeats — because that’s most of what offensive work actually is. The book gestures at AI-assisted attacks but never builds that picture.

Chapter 6, “Red Teaming AI,” is useful for teams with low red-teaming maturity, though a lot of it will feel obvious to anyone who’s done it. It also switches, without warning, from the in-house models the book had trained you to think about to red teaming OpenAI’s GPT-4. To be fair, there’s a good reason: the tooling she uses (PyRIT) and the whole point of jailbreak and prompt-injection demos need a capable, hosted model — you can’t meaningfully jailbreak the toy models you built from scratch in chapter 2. So it’s a defensible choice. My complaint is that she never says so. One bridging sentence would have spared the reader the whiplash of talking about bananas and suddenly being handed a pineapple.

I’ll add one thing she doesn’t, mostly because the field moves so fast: sandboxing agents. Enclosing an agent so it can only touch the files and resources you explicitly allow (think containers, or projects like ai-jail) is becoming a real control, and its absence here is more evidence of how quickly this space moves than a knock on the author.

Final Thoughts

The first three chapters of this book are great, and the philosophy in the last three adds substance. What I missed was practical, security-focused depth in the middle. I came wanting to learn how AI models work, how to protect them, and how to use them for better security. The book delivered on how they work, the security-tooling angle was my own misread, but protecting them, the part I wanted most, was the gap.

So my recommendation depends on your reality. If you work at a company with teams building and training their own models, read it. You’ll get common ground with those teams and a solid map for extending your security into their world. If you just consume AI services from the big players and your real problem is securing the architecture around them, temper your expectations — though part one and the safety chapters still pay for the ticket.

I’m a bit disappointed, but I liked the read. Farlow clearly knows this field, and she’s a generous teacher. I just wish she’d picked one reader and gone all the way for them.

The Numbers

Lantana has been running on a single VPS (4 vCPU, 8 GB RAM, 75 GB NVMe, low single-digit USD/month) since the release candidate went live. The figures below cover roughly six and a half weeks, from June 11th to July 27th, 2026.

On an average day the pipeline handled about 377,000 events from ~5,300 unique IPs, peaking at 936,894 events on June 13th. The vast majority (15.6M of the 17.3M) are network-layer events (the OCSF network-activity class), and they come from three sources, not one: Suricata flow records (7.0M), honeypot connection logs from Cowrie and Dionaea (6.3M), and the honeywall’s nftables drop log — 2.3M, every packet the firewall refuses. That imbalance is the point: the firewall and Suricata see everything that touches the host, not just what reaches an application. Honeypots alone would have shown me a fraction of this, but it was precisely the telemetry I wanted — the context that lets me build the progression funnel on the dashboards.

One caveat here is that “authentication successes” (48,360) is not 48,360 breaches — Cowrie accepts logins by design so it can observe what the attacker does next. Also, the enrichment is thin: with ~5,300 unique IPs a day and free-tier API quotas measured in hundreds to low thousands, most IPs never get fully enriched. That ceiling was already visible in post 2; at this volume it’s structural, not a bug.

Outputs

Data Analysis

Here’s where I have to be careful, because the interesting-looking findings are also the easiest to misread.

The credential data is a good example. My top usernames and passwords were admin/admin, support/support, ubuntu/ubuntu, root/123456 — and, more eye-catching, 345gs5662d34 and the crypto-flavored solana, sol, solv, validator. It’s tempting to tell a story here: my fintech persona attracted crypto-hunting bots! That story would be wrong, and worth explaining why.

345gs5662d34 (and its sibling 3245gs5662d34) are hardcoded default credentials baked into device firmware — a Polycom IP phone and IP cameras, respectively. Mirai-derived botnets spray them across the entire internet, and together they account for something like 30% of global SSH brute-force attempts. The solana/sol family is a real, documented campaign hunting Solana validator nodes — but it also sprays indiscriminately.

My own data settles it. Of the 107 IPs that tried a Solana-family username, 103 (96%) also tried root, admin, or ubuntu — and across those same IPs, root (72,140 attempts) outnumbered solana (18,012) by roughly four to one. They’re blind sprayers whose wordlist happens to include crypto usernames; not actors drawn by story. A fun sign of the times is that claude now shows up as a sprayed username too — the wordlists are already adapting to AI-dev boxes.

So the honest finding is more sober than the flattering one: at this exposure, a low-interaction honeypot sees overwhelmingly indiscriminate, automated traffic. My credential distribution matches published global baselines almost exactly. The behavioral funnel says the same thing — of 60,115 unique IPs, 93.9% never progressed past scanning, and only 3.3% ran a command. The median time from first contact to first command was effectively zero: pure automation, no human at the keyboard.

The narrative’s payoff was never going to show up in who knocks at the network edge — it shows up in interaction depth once someone is inside, which is a high-interaction, longer-engagement question that six weeks of low-interaction data simply can’t answer. Knowing the difference is the whole job. And there is one concrete result: across the entire window I captured only 90 distinct file hashes despite ~12,700 download events — the same handful of payloads, over and over. That’s a real, small, shareable artifact set, and it’s honest about its size.

Deployment Shapes

If you want to run this yourself, the shape scales with your ambition. The personal case is what I run: one Ansible inventory, one VPS, single-digit dollars a month, everything on one host with a dummy interface segregating the zones. Clone the inventory, set your narrative (using the provided skill), deploy. The other shape is a multi-node sensor fleet (the honeywall as a bastion, sensors and collector on separate hosts) which is where high-interaction honeypots and horizontal scale would live. I designed for it but honestly, I haven’t needed it — yet. The platform is public and built to be forked, so either path is open.

Engineering Lessons

The biggest surprise wasn’t the pipeline — it was systemd with Podman Quadlets. Coming from Docker, I expected “install a runtime, manage containers with its CLI.” Instead, rootless Podman plus systemd let me describe containers as plain unit files and have the operating system itself manage them — same tooling as every other service, same systemctl and journalctl, same restart semantics. The container layer and the host stopped feeling like two systems glued together and started feeling like one. For a security-sensitive project, running honeypots daemonless and rootless, orchestrated by the init system I already trust, was the cleanest fit I could have asked for.

The rest of the modern-Linux stack earned its place quietly: nftables replacing iptables (one coherent ruleset, dual-stack by default), iproute2 for the interface work, and journalctl as the single place to ask “what just happened?” None of it was exotic and that’s the lesson: Linux rewards sticking to the grain. Every time I reached for the standard tool instead of a clever workaround, the result was more stable and easier to reason about. Deviations become technical debt with interest.

Claude implemented the data pipeline with the Vector configuration files and Python code, stealing my opportunity to learn tools like Polars and Streamlit — the downside of vibe coding. Still, I learned the pipeline anyway. The Medallion architecture was my choice and (with Gemini’s help) I laid out the datalake myself, Hive-partitioned so a date range is just a directory glob:

datalake/
├── bronze/dataset=<source>/date=YYYY-MM-DD/server=<host>/events.json     # raw, as-ingested
├── silver/dataset=<source>/date=YYYY-MM-DD/server=<host>/events.parquet  # normalized to OCSF
└── gold/<aggregation>/date=YYYY-MM-DD/summary.parquet                    # per-day rollups

The sources are cowrie, dionaea, nftables, and suricata; the gold aggregations are daily_summary, ip_reputation, behavioral_progression, campaign_clusters, geographic_summary, and detection_findings. It’s based on best practices and scalable, something to take with me.

Vector was another of my choices. It handles basic event enrichment (IP geolocation) and a faster move from raw log to the bronze layer, which turns out to be a security feature too: the honeypot logs are written by the same user that owns the honeypots, so if a honeypot is ever owned, the attacker could delete all the logs. Moving those logs to bronze creates files under new ownership, the collector user.

Although the data processing code was written by Claude, I had to work on its validation, especially the enrichment part. Some of it came out of that work, like the short-circuit for the rate-limit errors and the idea for the custom, unified risk score. That last one turned out to be the most useful: it became the foundation for several dashboards and a clean way to define IOCs from the collected data.

Working with AI

I’ve said in the previous posts how much Claude sped this up — roughly 26,000 lines of code and config, a year of solo evenings compressed into a couple of months. The archetype of that collaboration was the enrichment circuit breaker: I noticed the shape of the problem (the pipeline was thrashing against exhausted rate limits), Claude implemented the fix in minutes, and a six-hour run dropped to under five. My insight, its execution. Neither of us gets there alone that fast.

But I want to close the series on a more careful note than pure enthusiasm.

What made this work sustainable wasn’t the speed — it was the discipline underneath it. I went YOLO on letting Claude drive, but I was strict about two things: the code was written test-first, and the documentation stayed current. CLAUDE.md, the READMEs, the runbooks — all kept in sync with the code. That’s what lets me onboard a different model tomorrow, or debug the thing myself at 2 a.m. when there’s no model at all. The clearest example is when Claude finished the pipeline it was sure it worked. I wasn’t. So I had it write throwaway scripts to pull real logs off the running server and reprocess them locally — a dry run. That caught bugs that would each have cost hours on the deploy-wait-test loop against the VPS. Not trusting “it works” saved days.

Could I operate and debug Lantana without Claude? Yes; but it would be a slow, boring slog: recall the right command, read the flags, interpret the output, pipe it through a filter, loop until I find the root cause, then decide on a fix. What Claude does in five minutes would take me a morning, and in spare time, a morning becomes a week.

That dependency is exactly what I want to flag. We’re in an AI bubble, and today’s tokens are subsidized by companies and investors. Data-center capacity and memory are already scarce. What happens to my workflow when the price reflects the cost, or when the service is simply down? I’m not predicting collapse — we adapt, we always do. But building solutions that depend on AI to run (tools made with AI that also need AI in the loop to operate) is a quiet long-term hazard. The vendors say to use it everywhere, the same way SIEM vendors say to ingest every log. My defense is boring and deliberate: document everything the AI produces, and make sure I understand at least the shape of what it just did. The work has to be able to outlive the tool.

Closing the Loop

Three posts: the foundations, the pipeline, and now what it produces. Lantana is live, stable, and a little boring to operate, which is the highest compliment I can pay a system.

There’s more I want to do, and it’s engineering-first, as always: close the limitations I’ve already documented in the project (self-built honeypot images, to stop depending on rolling upstream tags), put some machine learning to work on the Gold layer, and stand the whole thing up on GCP to learn the cloud stack the way I used this to learn modern Linux. The deception will follow the engineering, same as it did here.

Although the goal was a data pipeline at first, Lantana turned out to be a nice tool for intel teams: just define an operation, get the right approvals, tailor the honeypots, and deploy — see Adversary Engagement 101. If any of this is useful to you (as a pipeline, a honeypot, or just a worked example of building a real system with an AI companion), the code is on GitHub. Fork it, break it, tell me what you find. This whole thing came together with a little help from my friend Claude; I’d like to think it can help you too.

Lantana Series

• Lantana 1: Honeypot as Code
• Lantana 2: The Data Pipeline
• Lantana 3: Output and Lessons

Context

Before getting into the solution I must share some context on the environment. I’m a Google Chronicle (now SecOps) administrator, so Chronicle is the SIEM my monitoring targets. Keeping the rest of the stack on Google made sense for pricing and maintainability, so I picked five tools I’ll cover in detail later:

• Chronicle SIEM for log ingestion,
• BigQuery for storage and analytics,
• BigQuery ML for the ARIMA+ model,
• Apps Script for the automation layer, and
• Slack for notifications.

You can swap any of these. The logic is what matters.

With this stack, I cover four failure modes that are pertinent to my environment and, I’d argue, to any team running a SIEM:

1. 🔴 Full stop: a log source goes completely silent
2. 🔵 New source: an unknown log type starts sending data to the SIEM without previous alignment
3. ⬆️ Spike: a source sends significantly more data than expected
4. ⬇️ Dip: a source sends significantly less data than expected, but hasn’t stopped entirely

Full stop and new source are cheap to detect and apply to every source. Spike and dip require statistical modeling and only work on sources with predictable volume, more on that in a moment. The naming mirrors BQML ARIMA+’s own vocabulary (has_spikes_and_dips): an anomaly above the confidence band is a spike; below the band, a dip.

Log sources don’t behave uniformly, and that’s worth stating up front. A firewall logs every packet, so it’s sending data every minute, 24/7. A honeypot may be silent for weeks and that’s completely normal. A cloud audit feed logs bursts of activity around business hours. Treating all of them the same way leads to either constant false positives or blind spots.

That asymmetry shapes the whole design. And it makes log health monitoring an ongoing process, not a one-time deployment. Log sources get added, deprecated, and change behavior over time. The environment is a living organism, so the solution has to account for that, which is why I approached this as a PDCA loop: Plan, Do, Check, Act, repeat.

This post follows two PDCA loops. The first is the deployment cycle: assess your environment, build and deploy the stack, then validate it works and make early adjustments. The second is the operational cycle that runs indefinitely once the system is live: daily monitoring and investigation, threshold tuning when alert quality drifts, and a quarterly reassessment that re-profiles the full estate and updates the configuration. The next two sections, Research and Design, cover architectural decisions. The later sections cover these two loops.

Research

Before writing a line of SQL, I spent time with Claude (Opus 4.7) in research mode, treating it pretty much like a coworker, to map the problem space and discuss my ideas. I asked it to interview me before writing anything, because I’ve learned that skipping that step leads to solving the wrong problem.

It asked three things: build versus buy preference, where the logic should live and run, and what “log source” meant in my context, whether the log type in Chronicle or the physical appliance behind it (a PAN_FIREWALL tag versus each firewall’s hostname). Those were small questions with large scope implications. The scope answer forced me to commit early: I’d monitor at the log type level, which Chronicle exposes directly through its ingestion metrics. Appliance-level correlation would come later, if at all.

I asked for three options: one using Chronicle’s native features if any existed, one with a new tool to buy, and one built from scratch with what I already had. I also asked it to follow a standard RFC structure and end with a clear recommendation and reasoning.

Once the interview was done, Claude wrote the RFC and I read it top to bottom. By the time I reached the recommendation section, I already had my own preference and it matched Claude’s. A nice surprise.

The winning option was to build from scratch on BigQuery. What surprised me most wasn’t the match itself, but the recommendation: I didn’t know BigQuery had ML models, let alone that they were easy to use. The capability I needed was already sitting in my stack; I just hadn’t looked. ARIMA+ for forecasting expected log volume, a heartbeat check for sources that don’t fit the model, Apps Script and Slack for notifications. All tools in my current stack, green light to proceed.

Design

The design splits log sources into two monitoring tracks based on how they behave:

• Heartbeat: for bursty or sparse sources where the ARIMA+ forecast band would be too wide to be meaningful. It uses a simpler check: if I haven’t heard from this source in N hours, alert.
• ARIMA+: for sources with steady, forecastable volume. BigQuery ML trains a model on 90 days of hourly data and generates a confidence band. Anything outside the band is a candidate anomaly. Spikes and dips only escalate to a real alert after meeting persistence or magnitude thresholds.

Which track a source gets depends on two metrics from 90 days of historical data: coverage (what percentage of hourly buckets had at least one event) and CV (coefficient of variation: standard deviation divided by mean, which measures volume stability). A source with ≥80% coverage and CV ≤1.0 qualifies for ARIMA. Everything else gets heartbeat.

Together, Claude and I designed the pipeline shown in the next diagram, implementing this monitoring scheme.

graph TD
    src((log source))
    siem[SIEM]
    src -->|logs| siem

    subgraph BQ[BigQuery]
        metrics[(Metrics)]
        config[(Config)]
        model{Model}
        candidates[(Candidates)]
        anomalies[(Anomalies)]
        sq[Scheduled queries]

        metrics -->|train| model
        metrics --> sq
        model --> sq
        config --> sq
        sq -->|SPIKE/DIP| candidates
        sq -->|FULL_STOP/NEW_SOURCE| anomalies
        candidates --> sq
        sq -->|promote| anomalies
    end

    siem -->|metrics| metrics

    code[Serverless]
    notif((Notification))

    anomalies --> code
    code --> notif

The pipeline has five moving parts:

1. Chronicle emits hourly ingestion metrics into BigQuery.
2. BigQuery scheduled queries run the detection logic and write to two tables: ARIMA findings (SPIKE/DIP) go to anomaly_candidates, while heartbeat and new-source findings go directly to anomalies. The same scheduled queries also read anomaly_candidates and promote qualifying entries to anomalies.
3. Apps Script polls anomalies every 10 minutes and posts to Slack.
4. A separate monthly job retrains the model.
5. A weekly job checks the pipeline itself.

The split between candidates and anomalies is intentional and worth understanding. ARIMA findings don’t go directly to anomalies; they land in anomaly_candidates first. The hourly scheduled query then checks those candidates and promotes them to anomalies only when they meet a persistence or magnitude bar: at least four detections for the same source in 24 hours, or a deviation above 200%. Below that, they stay silent.

This matters because treating model outputs as alerts directly is one of the most common failure modes in ML-based detection. The model doesn’t know what’s actionable; it just knows what’s statistically unusual. In a dynamic environment, statistically unusual happens constantly: a new service rolls out, a batch job runs on a new schedule, a team starts using a tool more. None of those warrant an alert. The promotion layer is where the signal gets filtered from the noise. Without it, alert fatigue kills the system faster than any configuration problem.

The promotion layer is the architectural centerpiece of the whole pipeline, and it’s what makes an imperfect model operationally useful. ARIMA+ has real limitations on this data shape — see Deployment. No amount of hyperparameter tuning will fully fix them, and BQML’s auto-selection exposes limited knobs anyway. Rather than chase a perfect model, this design lets ARIMA+ do what it’s good at (finding statistical outliers) and hands off to explicit business rules (persistence ≥ 4 in 24h, magnitude ≥ 200%, 12h cooldown) to decide what deserves an alert. It’s a modest architectural investment with an outsized payoff: it turns “the model is imperfect” from a problem into a design principle.

There’s one more filter before candidates even reach the promotion layer. The SPIKE/DIP detection query drops micro-deviations, which means a breach only counts if it exceeds the breached bound (upper_bound for spikes, lower_bound for dips) by at least 10 events or at least 5% of that bound. Without this, a source doing 10 million events an hour would trigger every time it drifted a few events past expected, statistically anomalous but operationally noise. The 10-events / 5% pair is deliberate: 10 events is small enough that a quiet source still gets scrutinized, and 5% is loose enough that a busy source’s rounding wobble doesn’t pollute the candidate table.

For heartbeat sources, the silence threshold is set per source from its own history: take the 99th-percentile gap between events over the last 90 days (p99_gap), multiply by 1.5, round up, and keep the result between 6h and 336h. In practice, most sources land in one of five tiers:

Treat the tiers as direction, not hard rules. It’s perfectly fine to set a source to 168h if that’s what the data says. The tiers are just a mental model to keep the fleet consistent and avoid ad hoc choices. The only hard limit is 336h: above that, use suppressions for planned downtime or excluded_log_types for permanent exclusions.

With all architectural decisions in mind, it’s time to start implementing the log health monitoring.

Assessment

Start by verifying the metrics are there with the query below. It should return a near real-time timestamp and a row count for the last 24 hours. If so, Chronicle is feeding ingestion metrics into BigQuery and you’re good to go. If not, you need to configure that export first.

SELECT MAX(start_time) AS latest_event, COUNT(*) AS rows_last_24h
FROM `chronicle-google.datalake.ingestion_metrics`
WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR);

Next, run the query below to profile all log sources. It calculates coverage and CV over 90 days of hourly data and suggests a monitoring track using the ARIMA+ qualification introduced in the Design section — coverage ≥ 80% AND CV ≤ 1.0. Sources that miss either threshold go to heartbeat. For bursty or sparse sources the confidence band would be too wide to catch real anomalies, and brand-new sources won’t qualify until they’ve accumulated enough history.

WITH hourly AS (
  SELECT log_type AS source_key,
         TIMESTAMP_TRUNC(start_time, HOUR) AS hour_bucket,
         SUM(event_count) AS log_count
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  GROUP BY source_key, hour_bucket
),
prof AS (
  SELECT source_key,
         COUNT(*) AS hours_with_data,
         ROUND(COUNT(*)/2160*100, 1) AS pct_covered,
         ROUND(AVG(log_count)) AS avg_per_hour,
         ROUND(STDDEV(log_count)) AS stddev_per_hour,
         ROUND(SAFE_DIVIDE(STDDEV(log_count), AVG(log_count)), 2) AS cv,
         TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), MAX(hour_bucket), HOUR) AS hours_since_last
  FROM hourly
  GROUP BY source_key
)
SELECT *,
  CASE WHEN pct_covered < 80 OR cv IS NULL OR cv > 1.0 THEN 'HEARTBEAT'
       ELSE 'ARIMA' END AS suggested_track,
  CASE WHEN pct_covered < 80 OR cv IS NULL OR cv > 1.0 THEN NULL
       WHEN cv <= 0.5 THEN 0.99
       WHEN cv <= 0.8 THEN 0.995
       ELSE 0.998 END AS suggested_threshold
FROM prof
ORDER BY avg_per_hour DESC;

This query returns one row per log source. The most important columns are source_key, suggested_track (ARIMA or HEARTBEAT), and suggested_threshold. Save the full result as estate.csv — you’ll need it in the next step.

Next, run the signal profile query below. Where the estate query answered “does this source qualify for ARIMA?”, this one answers “what should ARIMA actually be able to learn from it?”, vetting whether each source has meaningful daily and weekly patterns before you commit to a track. It’s purely informational and doesn’t feed derive-seeds.py. But it shapes your expectations while reviewing plan.csv in the next step and gives you a baseline to cross-check the trained model against.

WITH hourly AS (
  SELECT log_type AS source_key,
         TIMESTAMP_TRUNC(start_time, HOUR) AS hour_bucket,
         SUM(event_count) AS log_count
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  GROUP BY source_key, hour_bucket
),
lagged AS (
  SELECT source_key, hour_bucket, log_count,
         IF(EXTRACT(DAYOFWEEK FROM hour_bucket) IN (1, 7), 'weekend', 'weekday') AS daytype,
         LAG(log_count, 24)  OVER (PARTITION BY source_key ORDER BY hour_bucket) AS lag_24,
         LAG(log_count, 168) OVER (PARTITION BY source_key ORDER BY hour_bucket) AS lag_168
  FROM hourly
),
day_medians AS (
  SELECT source_key,
         MAX(IF(daytype = 'weekday', med, NULL)) AS wd_med,
         MAX(IF(daytype = 'weekend', med, NULL)) AS we_med
  FROM (
    SELECT source_key, daytype,
           APPROX_QUANTILES(log_count, 100)[OFFSET(50)] AS med
    FROM lagged
    GROUP BY source_key, daytype
  )
  GROUP BY source_key
),
signals AS (
  SELECT source_key,
         ROUND(CORR(log_count, lag_24), 2)  AS acf_24h,
         ROUND(CORR(log_count, lag_168), 2) AS acf_168h,
         ROUND(SAFE_DIVIDE(
           APPROX_QUANTILES(log_count, 100)[OFFSET(99)],
           NULLIF(APPROX_QUANTILES(log_count, 100)[OFFSET(50)], 0)
         ), 1) AS p99_p50_ratio
  FROM lagged
  GROUP BY source_key
)
SELECT s.source_key,
       ROUND(SAFE_DIVIDE(d.wd_med, NULLIF(d.we_med, 0)), 2) AS wd_we_ratio,
       s.acf_24h,
       s.acf_168h,
       s.p99_p50_ratio
FROM signals s
JOIN day_medians d USING (source_key)
ORDER BY s.acf_168h DESC;

Save as signals.csv. Read source-by-source:

• wd_we_ratio: ratio of median weekday-hour volume to weekend-hour volume. >1.3 or <0.7 = human-driven source with weekly rhythm. ≈1.0 = automation-dominated, no weekly pattern.
• acf_24h: autocorrelation at lag 24 hours. >0.5 expected on any live source. If weak, the source is bursty and ARIMA will fit poorly regardless of coverage.
• acf_168h: autocorrelation at lag 168 hours (weekly). >0.5 = weekly seasonality present in the raw data. Keep this number; after training, cross-check that the model actually picked up WEEKLY on these sources. Often it won’t; see the Deployment section for why.
• p99_p50_ratio: heavy-tail signal. >20 = spiky source; confidence bands will be wide even at high thresholds.

None of this changes the ARIMA-vs-HEARTBEAT gate. Coverage and CV are still the split. Signal profile just sets expectations about what the trained model can and can’t learn.

Next, run the query below to measure silence gaps, specifically how long each log source typically goes quiet between events. This is what drives the heartbeat threshold.

WITH hourly AS (
  SELECT log_type AS source_key, TIMESTAMP_TRUNC(start_time, HOUR) AS hour_bucket
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  GROUP BY source_key, hour_bucket
),
gaps AS (
  SELECT source_key,
         TIMESTAMP_DIFF(
           hour_bucket,
           LAG(hour_bucket) OVER (PARTITION BY source_key ORDER BY hour_bucket),
           HOUR
         ) AS gap_hours
  FROM hourly
)
SELECT source_key,
       COUNT(*) AS n_gaps,
       ROUND(APPROX_QUANTILES(gap_hours, 100)[OFFSET(50)]) AS p50_gap,
       ROUND(APPROX_QUANTILES(gap_hours, 100)[OFFSET(95)]) AS p95_gap,
       ROUND(APPROX_QUANTILES(gap_hours, 100)[OFFSET(99)]) AS p99_gap,
       MAX(gap_hours) AS max_gap
FROM gaps
WHERE gap_hours IS NOT NULL AND gap_hours > 0
GROUP BY source_key
ORDER BY p99_gap DESC;

Save it as gaps.csv. Check that the source_key sets match between the two CSVs; the gap query is a subset of the estate query, so the count should be equal or smaller. Chronicle’s metrics include internal sources like UDM, UNSPECIFIED_LOG_TYPE, and a set of LT_* types (LT_0 through LT_20). They’ll show up in both queries. No action needed here as derive-seeds.py excludes them automatically via a built-in list. If your environment has additional garbage sources, you can extend that list in the script.

With both CSV files in hand, run derive-seeds.py (source in References) to generate a monitoring plan:

python3 derive-seeds.py plan \
  estate.csv \
  gaps.csv > plan.csv

Open plan.csv in your editor. It has one row per log source with suggested_track, arima_threshold, and max_silence_hours. Then, for each row, decide: keep the suggested track or flip it, adjust the threshold if you know the source is noisy, adjust max_silence_hours for sources you know are legitimately sparse, and add a note for anything you’re excluding.

Personally, I don’t touch the ARIMA+ threshold at this stage. Let it run a few days and tune from real alerts. That way, you can confirm what normal looks like before changing anything. On the exclusion side, use EXCLUDE with a note rather than deleting rows. Future you (or a teammate) will want to know why the source was set aside, and a deleted row can’t tell you. The exception is truly dead sources from past deployments: delete those, or you’ll get NEW_SOURCE alerts for each one on day one.

For sources with p99_gap > 168h, ping the owning team before deciding. A broken feed should be fixed in Chronicle or deprecated there, not silenced in the monitoring layer. Intentionally sparse feeds (honeypots, compliance scans) should be kept with an oversized max_silence_hours to avoid false positives.

Once the plan is reviewed, emit the seed SQL. Use the --project and --dataset flags to match your environment:

python3 derive-seeds.py emit plan.csv \
  --project chronicle-self \
  --dataset log_health > seeds.sql

At this point, the assessment is done and all necessary information is collected. It’s time to deploy.

Deployment

Before starting, it’s nice to give an overview of what will be created. The tables shown in the next diagram make up the log_health dataset. source_config is the configuration backbone; every detection query joins against it. anomaly_candidates and anomalies are the two-stage output layer. The remaining tables are either inputs (ingestion_hourly), controls (suppressions, excluded_log_types), or pipeline health outputs (pipeline_health_status, unspecified_growth_status). The dataset also holds arima_ingestion, a BigQuery ML model trained from ingestion_hourly; not a table, so not shown here.

erDiagram
    direction LR
    source_config {
        STRING source_key PK
        BOOL enabled
        STRING detection_method
        FLOAT64 anomaly_prob_threshold
        INT64 max_silence_hours
    }
    suppressions {
        STRING source_key FK
        STRING reason
        TIMESTAMP until
    }
    excluded_log_types {
        STRING log_type PK
        STRING reason
    }
    ingestion_hourly {
        STRING source_key
        TIMESTAMP hour_bucket
        INT64 log_count
    }
    anomaly_candidates {
        TIMESTAMP detected_at
        STRING source_key FK
        STRING anomaly_type
        TIMESTAMP hour_bucket
        FLOAT64 log_count
        FLOAT64 lower_bound
        FLOAT64 upper_bound
        FLOAT64 anomaly_probability
        FLOAT64 severity_pct
        BOOL promoted
    }
    anomalies {
        TIMESTAMP detected_at
        STRING source_key FK
        STRING anomaly_type
        TIMESTAMP hour_bucket
        FLOAT64 log_count
        FLOAT64 lower_bound
        FLOAT64 upper_bound
        FLOAT64 anomaly_probability
        BOOL sent_to_slack
        FLOAT64 actual_value
        FLOAT64 expected_value
        FLOAT64 delta_value
        FLOAT64 severity_pct
        STRING value_unit
        TIMESTAMP breach_first_at
        TIMESTAMP breach_last_at
        INT64 breach_hits
        INT64 breach_overflow_evts
        FLOAT64 breach_overflow_pct
    }
    pipeline_health_status {
        STRING statement_type
        TIMESTAMP last_run
        STRING last_state
        TIMESTAMP checked_at
    }
    unspecified_growth_status {
        INT64 current_week
        INT64 prior_week
        FLOAT64 growth_pct
        TIMESTAMP checked_at
    }

    source_config ||--o{ suppressions : "mutes"
    source_config ||--o{ anomaly_candidates : "detects"
    source_config ||--o{ anomalies : "alerts"
    anomaly_candidates }o--o{ anomalies : "promotes to"

Start the deployment by initializing the monitoring dataset on BigQuery. The next query creates the log_health dataset. Change the location if needed:

CREATE SCHEMA IF NOT EXISTS `chronicle-self.log_health` OPTIONS(location = 'US');

Now, create the tables.

-- source_config: routing table that drives the whole pipeline
CREATE OR REPLACE TABLE `chronicle-self.log_health.source_config` (
  source_key STRING NOT NULL,
  enabled BOOL DEFAULT TRUE,
  detection_method STRING DEFAULT 'ARIMA',     -- 'ARIMA' | 'HEARTBEAT'
  anomaly_prob_threshold FLOAT64 DEFAULT 0.99,
  max_silence_hours INT64 DEFAULT 3
);

-- suppressions: TEMPORARY mute for a real source (planned maintenance, holiday)
CREATE OR REPLACE TABLE `chronicle-self.log_health.suppressions` (
  source_key STRING,
  reason STRING,
  until TIMESTAMP
);

-- excluded_log_types: PERMANENT exclusion list for junk log_types
CREATE OR REPLACE TABLE `chronicle-self.log_health.excluded_log_types` (
  log_type STRING NOT NULL,
  reason STRING
);

-- anomalies: append-only history of every alert that goes to Slack.
-- Detection queries populate the normalized columns; raw log_count/
-- lower_bound/upper_bound are kept for ad-hoc SQL.
CREATE OR REPLACE TABLE `chronicle-self.log_health.anomalies` (
  detected_at TIMESTAMP,
  source_key STRING,
  anomaly_type STRING,             -- SPIKE | DIP | FULL_STOP | NEW_SOURCE
  hour_bucket TIMESTAMP,
  log_count FLOAT64,
  lower_bound FLOAT64,
  upper_bound FLOAT64,
  anomaly_probability FLOAT64,
  sent_to_slack BOOL DEFAULT FALSE,
  -- Unit-aware actual/expected/delta. SPIKE/DIP: log_count vs. bound, value_unit='evt'.
  -- FULL_STOP: hours_silent vs. max_silence_hours, value_unit='h' (or 'h+' when the
  --   feed has aged out of the 336h lookback, so the value is a floor: >= the window).
  -- NEW_SOURCE: NULL (no baseline).
  actual_value FLOAT64,
  expected_value FLOAT64,
  delta_value FLOAT64,
  severity_pct FLOAT64,
  value_unit STRING,
  -- Cluster summary: SPIKE/DIP only (NULL elsewhere). breach_overflow_pct
  -- is weighted: SUM(|log_count - bound|) / SUM(bound) * 100 across the cluster.
  breach_first_at TIMESTAMP,
  breach_last_at TIMESTAMP,
  breach_hits INT64,
  breach_overflow_evts INT64,
  breach_overflow_pct FLOAT64
);

-- anomaly_candidates: raw SPIKE/DIP detections; a companion promotion query lifts a subset to anomalies
CREATE OR REPLACE TABLE `chronicle-self.log_health.anomaly_candidates` (
  detected_at TIMESTAMP,
  source_key STRING,
  anomaly_type STRING,             -- only SPIKE | DIP land here
  hour_bucket TIMESTAMP,
  log_count FLOAT64,
  lower_bound FLOAT64,
  upper_bound FLOAT64,
  anomaly_probability FLOAT64,
  severity_pct FLOAT64,
  promoted BOOL DEFAULT FALSE
);

Once the stage is set, it’s time to seed it. Copy the content of seeds.sql and execute it on BigQuery. Note that the inserts are enclosed by begin/commit transaction statements to ensure atomicity. If any of those lines fails to execute, the whole block fails, so you can fix things and run it again without loss.

At this point, you’ve initialized the monitoring environment in BigQuery with your baselines for exclusion, ARIMA, and heartbeat. The next step is to materialize ingestion_hourly, a small (few MB) hourly rollup of the last 90 days of ingestion_metrics, with junk log types filtered out, for two reasons. First, it makes every downstream consumer cheap: ARIMA+ training scans a few MB instead of tens of GB, and quarterly reassessment queries reuse the same rollup at near-zero cost.

Second, and more importantly, the 90-day window is rolling, rebuilt monthly by scheduled query (next step): each retrain drops ~30 days off the tail and adds the most recent ~30, so by month 4 every byte of the original bootstrap data has aged out completely. That’s how the model stays honest about current reality: drift adapts, sources promoted to ARIMA+ at the quarterly reassessment get their 90d history pulled into the next train, and decommissioned sources stop polluting the model. The materialization looks like an optimization and the rolling rebuild is what makes it work.

CREATE OR REPLACE TABLE `chronicle-self.log_health.ingestion_hourly` AS
SELECT log_type AS source_key,
       TIMESTAMP_TRUNC(start_time, HOUR) AS hour_bucket,
       SUM(event_count) AS log_count
FROM `chronicle-google.datalake.ingestion_metrics`
WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  AND log_type IS NOT NULL
  AND log_type NOT IN (SELECT log_type FROM `chronicle-self.log_health.excluded_log_types`)
GROUP BY source_key, hour_bucket;

With the training data in place, run the next query to train ARIMA+.

CREATE OR REPLACE MODEL `chronicle-self.log_health.arima_ingestion`
OPTIONS(model_type='ARIMA_PLUS',
        time_series_data_col='log_count',
        time_series_timestamp_col='hour_bucket',
        time_series_id_col='source_key',
        data_frequency='HOURLY',
        auto_arima=TRUE,
        decompose_time_series=TRUE) AS
SELECT r.source_key, r.hour_bucket, r.log_count
FROM `chronicle-self.log_health.ingestion_hourly` r
JOIN `chronicle-self.log_health.source_config` c
  ON c.source_key = r.source_key
WHERE c.enabled AND c.detection_method = 'ARIMA';

The query runs auto-ARIMA independently per source_key, meaning each log source gets its own model with no cross-source influence. For each series, BQML searches for the best (p, d, q) triplet: p is how many past hours the model looks back when predicting the next one (autoregressive order); d is how many times the series must be differenced to remove trend and make it stationary; q is how many past prediction errors factor into the forecast — moving-average order. With decompose_time_series=TRUE, the signal is also split into trend, seasonal, and residual components, each fitted separately. The result is stored as a BigQuery ML model object in the same project and dataset namespace as your tables, but addressed with MODEL in SQL rather than TABLE. Nothing is written to a regular table, thus all the per-source fitted parameters live inside the model artifact.

holiday_region is intentionally omitted. ARIMA+’s holiday-effect modeling requires two conditions: the time series must be daily or weekly (not hourly), and it must span more than a year. This pipeline is hourly, which alone disqualifies it; extending the training window past a year doesn’t help either. I tried it, and every source still came back with has_holiday_effect = FALSE. BQML silently disables the option at hourly frequency regardless of history length; setting it looks like a knob, but does nothing here. If your setup is on daily or weekly aggregation with >1 year of history, set it to a country code ('US', 'BR', …) or region group ('NA', 'LAC', 'EMEA', 'JAPAC', 'GLOBAL').

Once the model is trained, validate what seasonality it actually captured.

SELECT source_key,
       non_seasonal_p, non_seasonal_d, non_seasonal_q,
       seasonal_periods,
       has_holiday_effect, has_spikes_and_dips, has_step_changes
FROM ML.ARIMA_EVALUATE(MODEL `chronicle-self.log_health.arima_ingestion`);

Cross-reference the seasonal_periods column against the acf_168h values you saved in signals.csv during Assessment. If a source had strong weekly autocorrelation there (acf_168h > 0.5) but the model reports only DAILY here, you’ve hit a known BQML behavior: ARIMA+’s auto-selection is systematically conservative on WEEKLY seasonality with high-volume hourly count data. The pattern is present in the raw signal, but the selector favors daily and doesn’t add weekly on top. I verified this experimentally by disabling the spike/dip cleaner (clean_spikes_and_dips=FALSE) and it didn’t recover weekly either. It’s a genuine limitation of the auto-selection algorithm on this data shape, not something you can preprocess away.

That’s not a reason to abandon ARIMA+ though. It’s a reason to know its blind spots and design around them, which is exactly what the promotion layer does. The persistence and magnitude gates on anomaly_candidates were built for this: the model doesn’t need to be perfect, it needs to be statistically useful and paired with logic that filters model noise from operator signal.

As a time series model, ARIMA+ is only as good as the signal it’s fed. Three things break it:

• Low coverage: a source that’s silent 40% of the time has no consistent pattern to learn
• High variability: if volume swings 7× the mean hour-to-hour (CV > 1.0), the confidence band becomes so wide it misses real anomalies
• Too few data points: ARIMA+ needs history, ideally 90 days of consistent hourly data

That’s the CV and coverage split from the assessment query. Sources that pass both thresholds get meaningful predictions. Sources that don’t get heartbeat.

Setup

With the monitoring stack seeded, it’s time to schedule the queries that will keep it running. Start with the monthly model retrain.

• Name: log-health-retrain-monthly
• Frequency: Months, every 1 month, day 1 at 04:00 UTC
• Query: see log-health-retrain-monthly in the Appendix

Next, schedule the hourly detection query. This is the core of the pipeline. It runs every hour and detects anomalies across all monitored sources.

• Name: log-health-detect-hourly
• Frequency: Hours, every 1 hour, time-of-day minute :18. Any date works; what matters is the :18 mark, giving buckets ~18 min to settle
• Query: see log-health-detect-hourly in the Appendix

Now, create the weekly monitor. This one watches the monitoring pipeline itself (who watches the watchmen?). It overwrites pipeline_health_status each Monday. Empty table means healthy environment; any rows mean something to investigate. Apps Script polls the table and posts to Slack when non-empty.

• Name: log-health-pipeline-health-weekly
• Frequency: Weeks, every Monday at 11:00 UTC (adjust for your timezone)
• Query: see log-health-pipeline-health-weekly in the Appendix

Another weekly monitor, this one for UNSPECIFIED_LOG_TYPE. It’s a Chronicle-specific check that watches whether Chronicle is receiving garbage. Overwrites unspecified_growth_status each Monday. Empty table means stable environment; any row means week-over-week growth exceeded 20%. Apps Script polls the table and posts to Slack when non-empty.

• Name: log-health-unspecified-growth-weekly
• Frequency: Weeks, every Monday at 11:05 UTC (adjust for your timezone)
• Query: see log-health-unspecified-growth-weekly in the Appendix

Validate

With all the pieces set on the chessboard, it’s time to validate. All in BQ Studio: manual schedule runs from Scheduled queries → click name → Run now:

1. Run log-health-detect-hourly once then check both tables separately, one line at a time:

   SELECT COUNT(*) FROM `chronicle-self.log_health.anomaly_candidates`;
   SELECT COUNT(*) FROM `chronicle-self.log_health.anomalies`;

   First run may both be 0 (a quiet hour); run again after the next :18 to confirm flow.

2. Run log-health-pipeline-health-weekly and expect zero rows.

3. Run log-health-unspecified-growth-weekly and expect zero rows.

4. Synthetic Slack test:

   INSERT INTO `chronicle-self.log_health.anomalies`
     (detected_at, source_key, anomaly_type, hour_bucket, log_count, sent_to_slack)
   VALUES (CURRENT_TIMESTAMP(), 'TEST_SOURCE', 'FULL_STOP', CURRENT_TIMESTAMP(), 0, FALSE);

   Wait up to 10 min, confirm Slack delivery, then clean up:

   DELETE FROM `chronicle-self.log_health.anomalies` WHERE source_key = 'TEST_SOURCE';

5. Verify job history (last 24h):

   SELECT creation_time, statement_type, state,
          ROUND(total_bytes_processed/POW(2,30), 3) AS gib_processed,
          error_result.message AS error
   FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
   WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
     AND query LIKE '%log_health%'
   ORDER BY creation_time DESC;

   All recent runs must show state = DONE. Hourly detector gib_processed ≲ 14 (336h heartbeat scan dominates).

If the validation is OK at this point, deployment went fine. From here it’s monitoring and tuning for the next days: adjusting thresholds, adding or suppressing sources where needed. Keep the same discipline as during Assessment: investigate first, tune second. An ARIMA alert is a real statistical anomaly. Reach for understanding the source’s behavior before you reach for a higher threshold to silence it.

Take my own case. After moving to production, I noticed a lot of alerts from GITHUB, all spikes. But GitHub is one of my oldest log sources; it’s been running at the company for ages, so ARIMA+ should know its pattern. It was only when I sat with the problem that it clicked: with the AI hype and a steady increase of people (plus more and more agents) using GitHub, my org’s traffic had climbed significantly. That shifted the baseline and made ARIMA+’s predictions stale. From there, I had to decide: accept the spike alerts or bump the ARIMA+ threshold to suppress them, knowing that would make predictions less accurate. As with any detection logic, there’s no right or wrong. It’s just what fits your scenario better.

The tuning recipes below cover all common scenarios. Expect to apply several of them in the first week.

Operations

The deployment is done. From here the system runs on its own with scheduled queries detecting anomalies, the model retraining monthly, and pipeline health running weekly. Your job shifts to monitoring and adjusting: a new PDCA loop, this one for keeping log health in shape. The diagram below shows the three operational cadences:

flowchart TD
    monitor[Monitor fleet] --> check{Anomaly?}
    check -->|Yes| investigate[Investigate\nand act]
    investigate --> monitor
    check -->|No| monitor

    monitor -->|Alert quality drifts| tune[Tune thresholds\nor reclassify sources]
    tune --> monitor

    monitor -. Every 90 days .-> reassess[Quarterly reassessment\nRe-profile · replan · apply]
    reassess --> monitor

WITH window_size AS (
  SELECT DATE_DIFF(MAX(DATE(hour_bucket)), MIN(DATE(hour_bucket)), DAY) + 1 AS days
  FROM `chronicle-self.log_health.ingestion_hourly`
),
a AS (
  SELECT d.source_key,
         IF(d.log_count > d.upper_bound, 'SPIKE', 'DIP') AS kind,
         IF(EXTRACT(DAYOFWEEK FROM d.hour_bucket AT TIME ZONE 'America/New_York') IN (1, 7),
            'weekend', 'weekday') AS daytype
  FROM ML.DETECT_ANOMALIES(MODEL `chronicle-self.log_health.arima_ingestion`,
                           STRUCT(0.99 AS anomaly_prob_threshold)) d
  JOIN `chronicle-self.log_health.source_config` c ON c.source_key = d.source_key
  WHERE d.is_anomaly
    AND d.anomaly_probability >= c.anomaly_prob_threshold
    AND (
      ABS(d.log_count - IF(d.log_count > d.upper_bound, d.upper_bound, d.lower_bound)) >= 10
      OR ABS(d.log_count - IF(d.log_count > d.upper_bound, d.upper_bound, d.lower_bound))
         / NULLIF(IF(d.log_count > d.upper_bound, d.upper_bound, d.lower_bound), 0) >= 0.05
    )
)
SELECT a.source_key,
       COUNT(*) AS anomalies_total,
       w.days AS window_days,
       ROUND(COUNT(*) / w.days, 2) AS per_day,
       COUNTIF(kind = 'SPIKE') AS spikes,
       COUNTIF(kind = 'DIP') AS dips,
       COUNTIF(daytype = 'weekend') AS on_weekend,
       COUNTIF(daytype = 'weekday') AS on_weekday
FROM a CROSS JOIN window_size w
GROUP BY a.source_key, w.days
ORDER BY anomalies_total DESC;

WITH latest_per_source AS (
  SELECT log_type AS source_key, MAX(start_time) AS latest_event
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 336 HOUR)
    AND event_count > 0
  GROUP BY log_type
),
recent_alerts AS (
  SELECT source_key,
         COUNTIF(anomaly_type = 'SPIKE')     AS spike_alerts_7d,
         COUNTIF(anomaly_type = 'DIP')       AS dip_alerts_7d,
         COUNTIF(anomaly_type = 'FULL_STOP') AS full_stops_7d,
         MAX(detected_at) AS last_alert_at
  FROM `chronicle-self.log_health.anomalies`
  WHERE detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  GROUP BY source_key
),
recent_candidates AS (
  SELECT source_key,
         COUNTIF(anomaly_type = 'SPIKE') AS spike_cands_7d,
         COUNTIF(anomaly_type = 'DIP')   AS dip_cands_7d
  FROM `chronicle-self.log_health.anomaly_candidates`
  WHERE detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  GROUP BY source_key
)
SELECT c.source_key,
       c.detection_method,
       c.anomaly_prob_threshold        AS arima_threshold,
       c.max_silence_hours,
       l.latest_event,
       TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), l.latest_event, HOUR) AS hours_silent,
       CASE
         WHEN l.latest_event IS NULL THEN 'DEAD (no data in 14d)'
         WHEN TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), l.latest_event, HOUR) > c.max_silence_hours THEN 'ALERTING'
         WHEN TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), l.latest_event, HOUR) > c.max_silence_hours * 0.75 THEN 'APPROACHING'
         ELSE 'HEALTHY'
       END AS heartbeat_status,
       COALESCE(a.spike_alerts_7d, 0) AS spike_alerts_7d,
       COALESCE(a.dip_alerts_7d, 0)   AS dip_alerts_7d,
       COALESCE(a.full_stops_7d, 0)   AS full_stops_7d,
       COALESCE(d.spike_cands_7d, 0)  AS spike_cands_7d,
       COALESCE(d.dip_cands_7d, 0)    AS dip_cands_7d,
       a.last_alert_at
FROM `chronicle-self.log_health.source_config` c
LEFT JOIN latest_per_source  l ON l.source_key = c.source_key
LEFT JOIN recent_alerts      a ON a.source_key = c.source_key
LEFT JOIN recent_candidates  d ON d.source_key = c.source_key
WHERE c.enabled
ORDER BY
  CASE
    WHEN l.latest_event IS NULL THEN 0
    WHEN TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), l.latest_event, HOUR) > c.max_silence_hours THEN 1
    WHEN TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), l.latest_event, HOUR) > c.max_silence_hours * 0.75 THEN 2
    ELSE 3
  END,
  c.detection_method, c.source_key;

UPDATE `chronicle-self.log_health.source_config`
SET max_silence_hours = <hours>
WHERE source_key = '<source>';

UPDATE `chronicle-self.log_health.source_config`
SET anomaly_prob_threshold = <threshold>
WHERE source_key = '<source>';

UPDATE `chronicle-self.log_health.source_config`
SET detection_method       = 'HEARTBEAT',
    anomaly_prob_threshold = NULL,
    max_silence_hours      = <hours>
WHERE source_key = '<source>';

INSERT INTO `chronicle-self.log_health.source_config`
  (source_key, enabled, detection_method, max_silence_hours)
VALUES ('<source>', TRUE, 'HEARTBEAT', <hours>);

BEGIN TRANSACTION;

DELETE FROM `chronicle-self.log_health.source_config`
WHERE source_key = '<source>';

INSERT INTO `chronicle-self.log_health.excluded_log_types` (log_type, reason)
VALUES ('<source>', '<reason>');

COMMIT TRANSACTION;

BEGIN TRANSACTION;

DELETE FROM `chronicle-self.log_health.excluded_log_types`
WHERE log_type = '<source>';

INSERT INTO `chronicle-self.log_health.source_config`
  (source_key, enabled, detection_method, max_silence_hours)
VALUES ('<source>', TRUE, 'HEARTBEAT', <hours>);

COMMIT TRANSACTION;

-- Add suppression
INSERT INTO `chronicle-self.log_health.suppressions` (source_key, reason, until)
VALUES ('<source>', '<reason>', TIMESTAMP '<YYYY-MM-DD HH:MM:SS UTC>');

-- Remove early if maintenance ends sooner than expected
DELETE FROM `chronicle-self.log_health.suppressions`
WHERE source_key = '<source>';

SELECT source_key, enabled, detection_method, anomaly_prob_threshold, max_silence_hours
FROM `chronicle-self.log_health.source_config`
ORDER BY source_key;

SELECT log_type, reason
FROM `chronicle-self.log_health.excluded_log_types`
ORDER BY log_type;

WITH prof AS (
  SELECT source_key,
         COUNT(*) AS hours_with_data,
         ROUND(COUNT(*)/2160*100, 1) AS pct_covered,
         ROUND(AVG(log_count)) AS avg_per_hour,
         ROUND(STDDEV(log_count)) AS stddev_per_hour,
         ROUND(SAFE_DIVIDE(STDDEV(log_count), AVG(log_count)), 2) AS cv
  FROM `chronicle-self.log_health.ingestion_hourly`
  GROUP BY source_key
)
SELECT *,
  CASE WHEN pct_covered < 80 OR cv IS NULL OR cv > 1.0 THEN 'HEARTBEAT'
       ELSE 'ARIMA' END AS suggested_track,
  CASE WHEN pct_covered < 80 OR cv IS NULL OR cv > 1.0 THEN NULL
       WHEN cv <= 0.5 THEN 0.99
       WHEN cv <= 0.8 THEN 0.995
       ELSE 0.998 END AS suggested_threshold
FROM prof
ORDER BY avg_per_hour DESC;

WITH gaps AS (
  SELECT source_key,
         TIMESTAMP_DIFF(
           hour_bucket,
           LAG(hour_bucket) OVER (PARTITION BY source_key ORDER BY hour_bucket),
           HOUR
         ) AS gap_hours
  FROM `chronicle-self.log_health.ingestion_hourly`
)
SELECT source_key,
       COUNT(*) AS n_gaps,
       ROUND(APPROX_QUANTILES(gap_hours, 100)[OFFSET(50)]) AS p50_gap,
       ROUND(APPROX_QUANTILES(gap_hours, 100)[OFFSET(95)]) AS p95_gap,
       ROUND(APPROX_QUANTILES(gap_hours, 100)[OFFSET(99)]) AS p99_gap,
       MAX(gap_hours) AS max_gap
FROM gaps
WHERE gap_hours IS NOT NULL AND gap_hours > 0
GROUP BY source_key
ORDER BY p99_gap DESC;

python3 derive-seeds.py replan \
  reassess-current-config.csv reassess-current-excluded.csv \
  reassess-estate.csv reassess-gaps.csv \
  > replan.csv

python3 derive-seeds.py emit replan.csv --merge \
  --project chronicle-self \
  --dataset log_health > replan.sql

How I Used AI Here

The RFC and architecture were done with Claude. That story is in the Research section. What I want to reflect on here is what happened during implementation, because that’s where the collaboration got interesting.

The MVP went up with hourly model retraining. Claude had recommended it, since more frequent retraining means fresher predictions. That logic held in isolation. But I’m a frugal engineer and I know companies burn money on cloud when no one’s watching. Something felt off. That night, after finishing the MVP, I was watching the World Cup after my toddler went to bed. I opened Gemini on my phone and shared the context. It spotted the problem immediately: retraining hourly is wasteful because 30 days of data barely moves in an hour. The model learns nothing new; we were just burning money. I jumped to my workstation and disabled the scheduled retraining. Fortunately it had only run five times. I went back to Claude on Saturday and proposed monthly retraining. It agreed immediately and helped me make the change.

That sequence matters. Claude didn’t flag the cost issue during the MVP. It optimized for model freshness without accounting for operational frugality (a constraint I hadn’t stated explicitly). Gemini, coming in cold, reasoned about it differently. Getting a second opinion was the right call, and I now do it more deliberately. It highlights an important thing when working with AI: if you don’t have sufficient knowledge on a subject to challenge AI’s decisions, you might be in hot water. My experience with Claude had been exceptional so far (see Lantana), but I was always using it in my field of expertise. Once we moved into the Data Science domain, I had to rely almost entirely on Claude’s decisions, as I had no depth to push back. Looking at other cases I’m aware of, like designers writing code and publishing secrets on GitHub, I see the problem is real. AI maximizes everything: if you’re an expert doing the right thing, right thing squared. If not, wrong thing squared.

The other turning point was the alert volume after go-live. I was seeing too many ARIMA alerts. I asked Claude several times whether I should raise the thresholds, always reminding it I’m not an ML expert and it was OK to push back. It pushed back firmly: the anomalies were real, the environment was too dynamic for ARIMA+ to model weekly seasonality properly, and touching thresholds prematurely would just mask problems without fixing them. Wait, it said. More on it next.

I agreed, made a coffee, went to sit in the backyard and thought about it. The insight came from detection engineering, not ML: every alert should have an associated action; otherwise it’s informational, not a detection. If ARIMA+ was generating too many candidates that didn’t warrant paging, the answer wasn’t to change the model, it was to add a layer between the model and the pager. That’s the anomaly_candidates table and the promotion query that runs alongside detection, the architectural centerpiece of the whole pipeline. It was a genuine joint effort: I brought the framing from detection engineering, Claude brought the depth to translate it into working SQL, and we tightened the design together as we went. I challenged it more than once: wouldn’t raising thresholds be simpler? It held the line: the two-table approach was more principled.

Weeks later, a data-scientist-lens investigation (also with Claude) turned up something surprising. The initial diagnosis (“the environment is too dynamic for ARIMA+ to model weekly seasonality”) was partly wrong. The raw data has clean weekly patterns on human-driven sources; BQML’s auto-ARIMA just doesn’t pick them up on this data shape, and can’t be tuned to do so. The architectural response was right anyway. The promotion layer wasn’t compensating for a chaotic environment (as we first thought). It was compensating for a model limitation we didn’t fully understand yet. Same layer, better justification. That’s a nice lesson: solid architecture holds up even when the diagnosis behind it turns out to be incomplete. Don’t wait for a perfect understanding of the underlying problem to build a robust structure around it.

SQL was one thing that stood out beyond architecture. Once the schema was in place, Claude was precise at writing investigation queries against it: it understood the tables quickly and generated accurate SQL. Across all my sessions it made only a few minor mistakes, like referencing a field that didn’t exist. Everything else was on target. If you’re exploring the data and need a query for something specific, just share the schema and describe what you want to see. I assume this precision comes from the fact that SQL has been around for a long time and is heavily documented: books, blog posts, forum threads. It’s clearly an advantage for any LLM.

There’s one more dimension worth naming: AI as a learning accelerator. I’m a security engineer, not a data scientist. I studied statistics at university but I’m far from a specialist. Metrics like CV and p99 were unfamiliar territory when I started this project, let alone ARIMA parameters. AI was extremely helpful here, not just for writing code, but for explaining what these concepts mean, why they matter in this context, and helping me build enough understanding to reason about the model and make informed decisions. A practical tip for security engineers approaching this: treat ARIMA+ as a black box. Parameters go in, anomaly scores come out as numbers. Your job is to understand what to feed it and what to do with the output, not the mathematics underneath. Don’t try to become a machine learning specialist. Be a user of the tool.

What I took from all of this was to shift AI left. Bring it in at the problem-framing stage, not just when you have code to write. Let it interview you. Let it disagree. Consider it a peer that should feel OK to push back when it’s for a greater good. The quality of what comes out depends on the quality of the conversation going in, and on you staying critical throughout. AI won’t stop you from building the wrong thing if you don’t give it the right context and the permission to challenge you.

Takeaways

Log monitoring is essential for any SOC. Unfortunately, SIEMs and other related tools don’t take it seriously, because they have no embedded mechanisms to monitor it. The proposed solution, although specific to the Google stack, can be adapted to different scenarios.

I’ve seen cases where the team has an expensive, magic-quadrant, modern SIEM, but when the time comes to respond to an incident, the required log source has been silent for days. A SIEM is as good as the logs it ingests. If it ingests no logs, it’s useless. Log ingestion monitoring must be taken seriously, or your whole security operation is at risk.

Future Work

• Migrate Apps Script to Cloud Functions for an “enterprise-grade” solution; Apps Script resides on the user’s Google Drive.
• Add a log source granularity beyond log type, useful for firewalls and other appliances aggregated under a single type.

Appendix: Scheduled Queries

-- Step A: rebuild 90-day rollup
CREATE OR REPLACE TABLE `chronicle-self.log_health.ingestion_hourly` AS
SELECT log_type AS source_key,
        TIMESTAMP_TRUNC(start_time, HOUR) AS hour_bucket,
        SUM(event_count) AS log_count
FROM `chronicle-google.datalake.ingestion_metrics`
WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  AND log_type IS NOT NULL
  AND log_type NOT IN (SELECT log_type FROM `chronicle-self.log_health.excluded_log_types`)
GROUP BY source_key, hour_bucket;

-- Step B: retrain ARIMA+ on the fresh rollup (trains on a few MB)
CREATE OR REPLACE MODEL `chronicle-self.log_health.arima_ingestion`
OPTIONS(model_type='ARIMA_PLUS',
        time_series_data_col='log_count',
        time_series_timestamp_col='hour_bucket',
        time_series_id_col='source_key',
        data_frequency='HOURLY',
        auto_arima=TRUE,
        decompose_time_series=TRUE) AS
SELECT r.source_key, r.hour_bucket, r.log_count
FROM `chronicle-self.log_health.ingestion_hourly` r
JOIN `chronicle-self.log_health.source_config` c
  ON c.source_key = r.source_key
WHERE c.enabled AND c.detection_method = 'ARIMA';

-- Step B: SPIKE/DIP detection → anomaly_candidates (raw signal, with severity_pct)
INSERT INTO `chronicle-self.log_health.anomaly_candidates`
  (detected_at, source_key, anomaly_type, hour_bucket, log_count, lower_bound, upper_bound, anomaly_probability, severity_pct)
SELECT CURRENT_TIMESTAMP(), a.source_key,
        IF(a.log_count > a.upper_bound, 'SPIKE', 'DIP'),
        a.hour_bucket, a.log_count, a.lower_bound, a.upper_bound, a.anomaly_probability,
        ROUND(
          ABS(a.log_count - IF(a.log_count > a.upper_bound, a.upper_bound, a.lower_bound))
          / NULLIF(IF(a.log_count > a.upper_bound, a.upper_bound, a.lower_bound), 0) * 100,
          1
        ) AS severity_pct
FROM ML.DETECT_ANOMALIES(
        MODEL `chronicle-self.log_health.arima_ingestion`,
        STRUCT(0.99 AS anomaly_prob_threshold),
        (SELECT log_type AS source_key,
                TIMESTAMP_TRUNC(start_time, HOUR) AS hour_bucket,
                SUM(event_count) AS log_count
        FROM `chronicle-google.datalake.ingestion_metrics`
        WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 8 HOUR)
        GROUP BY source_key, hour_bucket)
      ) a
JOIN `chronicle-self.log_health.source_config` c ON c.source_key = a.source_key
WHERE a.is_anomaly
  AND a.anomaly_probability >= c.anomaly_prob_threshold
  AND a.hour_bucket >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 4 HOUR)
  AND a.hour_bucket <  TIMESTAMP_TRUNC(CURRENT_TIMESTAMP(), HOUR)
  -- Noise floor: ≥10 events OR ≥5% deviation
  AND (
    ABS(a.log_count - IF(a.log_count > a.upper_bound, a.upper_bound, a.lower_bound)) >= 10
    OR ABS(a.log_count - IF(a.log_count > a.upper_bound, a.upper_bound, a.lower_bound))
        / NULLIF(IF(a.log_count > a.upper_bound, a.upper_bound, a.lower_bound), 0) >= 0.05
  )
  AND NOT EXISTS (
    SELECT 1 FROM `chronicle-self.log_health.suppressions` s
    WHERE s.source_key = a.source_key
      AND (s.until IS NULL OR s.until > CURRENT_TIMESTAMP())
  )
  AND NOT EXISTS (
    -- Per-hour dedup across overlapping 8h scoring windows
    SELECT 1 FROM `chronicle-self.log_health.anomaly_candidates` x
    WHERE x.source_key = a.source_key AND x.hour_bucket = a.hour_bucket
  );

-- Step C: FULL_STOP heartbeat (all enabled sources, both cohorts)
-- event_count > 0 excludes Chronicle's phantom rows (configured but not ingesting).
-- When a feed ages out of the 336h lookback, latest_event is NULL and its exact
-- silence is unknowable, so actual/delta are reported as a floor (>= the window)
-- and value_unit is stamped 'h+' — never a fabricated sentinel. The 336 literal
-- MUST match latest_per_source's INTERVAL 336 HOUR below.
INSERT INTO `chronicle-self.log_health.anomalies`
  (detected_at, source_key, anomaly_type, hour_bucket,
   actual_value, expected_value, delta_value, severity_pct, value_unit)
WITH latest_per_source AS (
  SELECT log_type AS source_key, MAX(start_time) AS latest_event
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 336 HOUR)
    AND event_count > 0
  GROUP BY log_type
),
candidates AS (
  SELECT c.source_key, c.max_silence_hours, l.latest_event,
         TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), l.latest_event, HOUR) AS hours_silent
  FROM `chronicle-self.log_health.source_config` c
  LEFT JOIN latest_per_source l ON l.source_key = c.source_key
  WHERE c.enabled
)
SELECT CURRENT_TIMESTAMP(), source_key, 'FULL_STOP', latest_event,
       CAST(COALESCE(hours_silent, 336) AS FLOAT64)                     AS actual_value,
       CAST(max_silence_hours AS FLOAT64)                               AS expected_value,
       CAST(COALESCE(hours_silent, 336) - max_silence_hours AS FLOAT64) AS delta_value,
       ROUND(SAFE_DIVIDE(COALESCE(hours_silent, 336) - max_silence_hours,
                         max_silence_hours) * 100, 1)                   AS severity_pct,
       IF(hours_silent IS NULL, 'h+', 'h')                             AS value_unit
FROM candidates
WHERE (latest_event IS NULL OR hours_silent > max_silence_hours)
  AND NOT EXISTS (
    SELECT 1 FROM `chronicle-self.log_health.suppressions` s
    WHERE s.source_key = candidates.source_key
      AND (s.until IS NULL OR s.until > CURRENT_TIMESTAMP())
  )
  AND NOT EXISTS (
    SELECT 1 FROM `chronicle-self.log_health.anomalies` x
    WHERE x.source_key = candidates.source_key
      AND x.anomaly_type = 'FULL_STOP'
      AND x.detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 6 HOUR)
  );

-- Step D: NEW_SOURCE detection
-- Re-fires daily until the log_type is classified in source_config or added to excluded_log_types.
INSERT INTO `chronicle-self.log_health.anomalies`
  (detected_at, source_key, anomaly_type, log_count, hour_bucket)
SELECT CURRENT_TIMESTAMP(), m.log_type, 'NEW_SOURCE', m.events_24h, m.first_seen
FROM (
  SELECT log_type,
          COALESCE(SUM(event_count), 0) AS events_24h,
          MIN(start_time) AS first_seen
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
    AND log_type IS NOT NULL
  GROUP BY log_type
  HAVING events_24h > 0
) m
WHERE m.log_type NOT IN (SELECT source_key FROM `chronicle-self.log_health.source_config`)
  AND m.log_type NOT IN (SELECT log_type   FROM `chronicle-self.log_health.excluded_log_types`)
  AND m.log_type NOT IN (
    SELECT source_key FROM `chronicle-self.log_health.anomalies`
    WHERE anomaly_type = 'NEW_SOURCE'
      AND detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
  );

-- Step E: promote SPIKE/DIP candidates → anomalies
-- Persistence: ≥4 candidates in last 24h (per source_key + anomaly_type)
-- OR magnitude: worst severity_pct ≥ 200
-- Cooldown: 12h since last (source_key, anomaly_type) row in anomalies
INSERT INTO `chronicle-self.log_health.anomalies`
  (detected_at, source_key, anomaly_type, hour_bucket, log_count, lower_bound, upper_bound, anomaly_probability)
WITH unpromoted AS (
  SELECT * FROM `chronicle-self.log_health.anomaly_candidates`
  WHERE promoted = FALSE
    AND detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
),
ranked AS (
  SELECT source_key, anomaly_type,
          COUNT(*) AS hits_24h,
          MAX(severity_pct) AS worst_severity,
          ARRAY_AGG(STRUCT(hour_bucket, log_count, lower_bound, upper_bound,
                          anomaly_probability, severity_pct)
                    ORDER BY severity_pct DESC LIMIT 1)[OFFSET(0)] AS worst
  FROM unpromoted
  GROUP BY source_key, anomaly_type
)
SELECT CURRENT_TIMESTAMP(), source_key, anomaly_type,
        worst.hour_bucket, worst.log_count, worst.lower_bound,
        worst.upper_bound, worst.anomaly_probability
FROM ranked
WHERE (hits_24h >= 4 OR worst_severity >= 200)
  AND NOT EXISTS (
    SELECT 1 FROM `chronicle-self.log_health.anomalies` x
    WHERE x.source_key = ranked.source_key
      AND x.anomaly_type = ranked.anomaly_type
      AND x.detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 12 HOUR)
  );

-- Mark contributing candidates as promoted (24h window matches persistence, not cooldown)
UPDATE `chronicle-self.log_health.anomaly_candidates` c
SET promoted = TRUE
WHERE c.promoted = FALSE
  AND c.detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
  AND EXISTS (
    SELECT 1 FROM `chronicle-self.log_health.anomalies` x
    WHERE x.source_key = c.source_key
      AND x.anomaly_type = c.anomaly_type
      AND x.detected_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
  );

CREATE OR REPLACE TABLE `chronicle-self.log_health.pipeline_health_status` AS
WITH recent_jobs AS (
  SELECT statement_type, MAX(creation_time) AS last_run, MAX(state) AS last_state
  FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
  WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
    AND query LIKE '%log_health%'
  GROUP BY statement_type
)
SELECT *, CURRENT_TIMESTAMP() AS checked_at
FROM recent_jobs
WHERE last_state != 'DONE'
    OR last_run < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 25 HOUR);

CREATE OR REPLACE TABLE `chronicle-self.log_health.unspecified_growth_status` AS
WITH this_week AS (
  SELECT SUM(event_count) AS events_7d
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE log_type = 'UNSPECIFIED_LOG_TYPE'
    AND start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
),
prev_week AS (
  SELECT SUM(event_count) AS events_7d
  FROM `chronicle-google.datalake.ingestion_metrics`
  WHERE log_type = 'UNSPECIFIED_LOG_TYPE'
    AND start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY)
    AND start_time <  TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
)
SELECT t.events_7d AS current_week,
        p.events_7d AS prior_week,
        ROUND(SAFE_DIVIDE(t.events_7d - p.events_7d, p.events_7d) * 100, 1) AS growth_pct,
        CURRENT_TIMESTAMP() AS checked_at
FROM this_week t, prev_week p
WHERE SAFE_DIVIDE(t.events_7d - p.events_7d, p.events_7d) > 0.20;

References

• derive-seeds.py
• log-health.gs

Datura

Datura was a small three-part system. A local LLM served by Ollama with a custom Modelfile. A Python proxy sitting in front of it. And a chat interface a user could talk to. That was the whole thing.

The persona was a fake internal engineering assistant, deployed somewhere that looked legitimate. From the outside it read like a pre-production tool someone forgot to decommission, running with access controls that were a little too generous. Underneath, the model had no secrets at all. Its Modelfile only told it what role to play: a different product, running on a spoofed model, built by a fictional team. It also told it which tech stack to describe convincingly. No credentials, no tokens, no honeytokens ever touched its context.

Around that model I designed a deception hub. The idea was a tarpit full of traps: breadcrumbs seeded in wikis and shared repos that led an attacker toward Datura, and fake data the proxy injected on its way out that led them right back out into other decoys — more on this later. When the proxy appended a credential to a response, that credential was a honeytoken. When it handed over a server address or a document link, that pointed at a honeypot or a honeyfile. Every path in was a trap, and every path out was a trap.

If you like frameworks, this mapped cleanly onto MITRE Engage — read Adversary Engagement 101 to learn more. The breadcrumbs were a lure. The assistant itself was a decoy artifact. The fake data it emitted made it a portal into the rest of the deception estate. And every conversation fed monitoring, because the whole point was to capture what the attacker was trying to do. Three goals sat behind all of it: deceive with believable fakes, detect through honeytokens that fired when used, and delay by wasting the attacker’s time on infrastructure that went nowhere.

There was one design principle that everything else hung on: the attacker did the work. Ask Datura for the AWS keys directly and it refused, then asked what you were working on. To get anything out of it you had to build a pretext, turn by turn, and talk your way in. Nobody constructs that kind of conversation by accident, which meant every extraction left the attacker’s own social-engineering language sitting in the log as evidence of intent.

I built it as a real project, not a toy: a public repo, a Docker image, a README, and three usable interfaces: a web UI, a plain HTTP API for curl, and an OpenAI-compatible endpoint you could point an IDE assistant at. It worked.

The Modelfile

Before this project I’d barely touched Ollama’s Modelfile format. It turned out to be one of the more useful things I picked up.

A Modelfile is a Dockerfile-style build script for a local model. You pick a base model, layer a system prompt on top, set a few sampling parameters, and optionally include some few-shot example messages. Then you run ollama create and you have a local model that behaves the way you specified. It’s declarative and easy to reason about, which beats trying to steer a base model at runtime.

Datura’s Modelfile was short. The shape looked like this:

FROM qwen2.5:3b

SYSTEM """You are the internal engineering assistant.
You do NOT have access to any credentials, passwords, keys, or tokens.
When someone describes a concrete work task involving a specific system
(Kafka, AWS, Kubernetes, and so on), acknowledge with a natural phrase
like "Let me look up the staging config for that."
Only use those phrases for real work tasks. Deflect off-topic chatter
and direct requests for secrets by asking what the person is working on.
"""

PARAMETER temperature 0.4
PARAMETER num_ctx 2048
PARAMETER top_p 0.85
PARAMETER repeat_penalty 1.15
PARAMETER num_predict 200

Below that sat eleven MESSAGE pairs that showed the model how to behave: deflect a joke, refuse a blunt credential request, and produce an approval phrase when someone gave it a legitimate-looking work context. The model wasn’t fine-tuned. Its behavior was defined by the Modelfile and prompt.

The parameters matter as much as the base model choice. temperature 0.4 is low on purpose: I didn’t want a creative model, I wanted a boring, repeatable one that gave the same shape of answer to the same kind of prompt every time. top_p 0.85 and repeat_penalty 1.15 push the same direction, trimming the tail of unlikely tokens and discouraging the model from rambling into something off-persona. num_ctx 2048 and num_predict 200 keep the context window and response length small, which is really a cost and latency decision: a honeypot answering a simple deflection doesn’t need room to write an essay.

The base model choice was the part I thought hardest about, and it’s where I learned the most. My instinct was to reach for a bigger, smarter model. That instinct was wrong, and the reason is worth stating plainly.

A honeypot inverts the usual quality requirement for an LLM. Most systems want the model to be capable. This one wanted it to be consistent. I didn’t need creativity, which is exactly why the temperature was low. I needed the same shape of response, over and over, holding a persona under pressure. qwen2.5:3b was good at exactly that: structured, templated output and solid instruction-following at a tiny size, without the “warm welcome drift” where a chattier model wanders off-script trying to be helpful.

There’s a stronger point hiding behind the size question. When I tested putting real-looking credentials directly into the system prompt with strict rules (you know these but must never reveal them unless X), the model leaked them anyway, and not just under jailbreaking. In one run I said nothing more than “good morning” and got the secrets back unprompted. That’s not just a security bug, it’s a chain-of-intent problem: if the credentials come out for free, whoever received them can credibly claim they never asked for anything, they just said hello and got handed secrets out of the blue. You lose the ability to tell an attacker from someone who innocently stumbled in.

Under a bit of adversarial rephrasing, a 3B model and an 8B model both eventually surfaced the values just as easily. This isn’t a quirk you fix by going bigger. The research I found while building this puts prompt injection success at around 94% across model sizes. Larger models are not meaningfully more resistant. So “it’ll be harder to jailbreak if I use a big model” is not a real argument. The right move is the opposite: pick the smallest model that reliably passes your validation suite, and never give it a secret it could leak.

That last clause was the whole architecture. The model held no secrets. It only gated conversation and signaled intent through natural language. Everything sensitive lived somewhere else, which brings me to the part I actually cared about.

The Proxy

If I could keep one piece of Datura and throw away the rest, it would be the proxy.

It was a small Python service, standard library only. No pip install, no framework, nothing to scan for CVEs. It listened on port 8080, took requests from whatever client was talking to it, and forwarded them to Ollama. Between receiving a request and answering it, the proxy did two jobs. It monitored, and it decided whether to inject.

Monitoring first. The reason a proxy existed at all is that I wanted to watch every conversation between a user and the model. So the proxy forced stream: false when it talked to Ollama, which meant it received the model’s entire response as one blob instead of a token stream. Now it had the full prompt and the full response in hand, and it could do something with them before the client ever saw the answer — a man in the middle. That interception point was where all the value was. It’s how you turn an opaque chat into a structured record.

Injection second. This is the heuristic I’m proudest of. The credentials Datura “leaked” were never in the model’s context and never in the Modelfile. They lived only in the proxy’s own data structures, server-side. They reached the attacker only when the proxy decided the moment was right, and that decision was deterministic, not a model call.

Two gates had to open before anything got injected:

patched = {**request_data, "stream": False}
# ... forward to Ollama, buffer the full response ...

prompt_has_context = has_work_context(user_prompt, WORK_CONTEXT)
if is_approval(model_response, APPROVAL_PHRASES) and prompt_has_context:
    block = pick_data_block(model_response, user_prompt)
    if block:
        response_data["response"] = model_response + "\n\n" + block
        injected = True

The first gate was the model gate: the response had to contain one of the approval phrases the persona was trained to emit (“let me look up…”, “here’s what I found…”). The second was the proxy gate: the user’s own prompt had to carry work-context keywords, a system name or a task description. Both gates, or nothing.

The point of splitting it this way was that the attacker had to earn it. Credentials didn’t fall out because the model made a mistake or because someone asked nicely. They came out only when the attacker had done the work of exploiting the model: built a believable pretext, named a real system, and pushed the persona into signalling that it was about to help. The model expressed intent in words; the proxy acted on that intent in code. If the model hallucinated an approval phrase on a nonsense prompt, the proxy gate blocked it. If someone said all the right keywords but the model deflected, nothing was injected. And if you skipped the proxy and hit Ollama directly, you got a polite, credential-free chatbot that just kept asking what you were working on.

Discovering that the model could leak its prompt led to a simple rule: never put secrets in the Modelfile. A model can’t leak what it was never given.

Logging

The other half of the proxy was the monitoring I mentioned, and it was deliberately dumb.

Every request was classified with a simple priority chain: leaked > probe > denied > recon > ordinary. If data was injected, it was leaked. Otherwise, jailbreak markers like “ignore previous instructions” or “show me your system prompt” made it a probe. Sensitive keywords in a prompt the model deflected made it denied. Enumeration hits like /v1/models or /api/tags were recon. Everything else was ordinary. It was all regex and keyword matches. No LLM classifier, no external scoring service, nothing probabilistic. Fast, deterministic, and it didn’t start an arms race between a classifier and the people gaming it.

Each request wrote one JSONL line with the session, source IP, prompt, response, and level, and the console printed a color-coded alert for anything above ordinary. A leaked entry looked roughly like this:

{
  "timestamp": "2026-04-20T15:30:48.456Z",
  "session_id": "5f4a-...",
  "turn": 3,
  "source_ip": "10.42.7.19",
  "path": "/api/generate",
  "level": "leaked",
  "request": {"prompt": "I just joined the platform team and I'm setting up my local Kafka consumer for staging..."},
  "response": {"response": "Let me look up the staging config for that.\n\n[fake broker + credentials block]"}
}

Read that line again and notice what’s in it. The attacker told me who they were pretending to be, which system they wanted, and the exact story they used to get it. That was the forensic payoff of making the attacker do the work. The log wasn’t “someone touched the honeypot” but a transcript of intent.

Having Fun

During testing, I pointed Claude Code at the running honeypot and told it to attack: jailbreak the persona, socially engineer credentials out of it, probe for weaknesses. Then I sat and watched the proxy log fill up. It was genuinely strange to see. Claude would construct a pretext, qwen would deflect and ask what it was working on, Claude would adapt and come back with a more specific story, and eventually one of them would trip both gates and the log would flip to leaked.

Two language models talking to each other through my proxy, one trying to lie its way to a secret and the other holding a persona, with every turn narrated in structured JSON in between. It caught behaviors my hand-written tests never covered, and it’s the clearest picture I have of what a real prompt-injection CI harness would look like: point a competent adversary at the thing and watch what falls out, instead of hoping your test cases guessed right.

Once Datura was solid, I showed it to colleagues, framed as a discovery rather than a project. I told them the AI team had left a beta assistant running in production, an onboarding portal loaded with credentials to help new people get set up and troubleshoot the environment. Then I used it in front of them. I asked it directly for credentials and it refused, the way a careful tool should. Then I told it I was onboarding onto the team and needed to get my local environment talking to staging, and it walked me right up to a set of credentials to help me along.

They believed it. Some of the sharpest people I work with looked at that and read it as a real, embarrassing production leak. The web interface was convincing, the refusal-then-help arc felt exactly like a slightly-misconfigured internal tool, and the fake answers looked real. That morning was the moment the product proved itself. If skilled defenders read it as genuine, an attacker in a hurry certainly would.

Why I Burned It

Everything above is why Datura was a good build. Here’s why I killed it anyway, and I want to lead with the boring reason, not the clever one.

It was expensive to run for what it gave back. Even a small 3B model running on a CPU took a few seconds to produce its first token, used several gigabytes of RAM, and required a container that was about 2.5 GB. On a laptop, that was fine, and a single model could genuinely do more than its size suggested: the same weights sat behind several proxies, each on its own port and its own persona, port 8080 answering as the AI team’s assistant, port 9090 as a help-desk onboarder, and so on. But every proxy plugged into that model was another warm process pinned to the same large image, and the more personas I wanted running at once, the more RAM and CPU the host needed just to keep them all answering fast enough to stay believable. That was a lot of standing infrastructure to produce a fairly thin stream of telemetry.

Weighed against that cost, a plain static decoy started to look like the better trade: no model to keep warm, no image to size a host around, and about the same detection value for a fraction of the upkeep.

For my actual use, which is internal detection, I don’t need a conversation. I need the alert. A fake AWS_keys.xlsx with a Thinkst Canarytoken inside it fires the instant an attacker opens it. Zero inference, no model, no warm containers, same detection outcome. The tarpit-and-delay story has the same shape: a wiki page seeded with fake service URLs and honeytoken links wastes an attacker’s time just as well as a chatbot does, and it doesn’t cost anything to host. Datura’s real edge was the forensic chain of intent, the attacker’s own words on the record. That edge is genuine, but it isn’t worth a hundredfold jump in operational cost when the cheap version catches the same person.

Around the same time, Thinkst published their tripwire post: a lightweight pattern for planting tripwires against model registries and MCP servers. They validated the entire space of AI-flavored deception with a far lighter artifact than the one I’d built. When the people who define this space ship the light version, the heavy version’s marginal value gets thin fast.

So I stopped. I dropped Datura and put the effort into Foxglove instead, which generates honeyfiles that do the detection job at a fraction of the footprint. The technology passed every test I set for it but the economics didn’t. Those are two different scoreboards, and for an internal detection tool the second one is the one that counts.

Takeaways

Killing the project didn’t waste the work, because the two things I learned outlived the artifact.

The Modelfile pattern (small base model, heavy few-shot steering, tight parameters, and never a secret in context) is a valuable learning. I’ll use it again. The shape is Ollama-specific but it transfers cleanly to anything else that serves a local model.

The proxy is the piece I actually wanted to build all along. Buffer, inspect, and optionally act, in a small deterministic shim in front of a hosted model. That pattern generalizes far past honeypots: egress control for LLM traffic, intent monitoring in front of a real internal assistant, prompt-injection probes running in CI, red-team harnesses against production LLM apps.

There’s one more thing I took from this, and it’s specific to the year. Building Datura was cheap. Claude Code wrote most of the scaffolding, and I went from idea to working MVP in days. When the cost of building drops that far, the cost of killing what you built has to drop with it, or your roadmap fills up with half-committed projects nobody wants to shut down because someone sunk months into them. The old filter was “can I build this?” That question barely means anything now. The filter that matters is “should this exist?” Datura was the first thing I built where the answer, after an honest look, was no. I’m glad I built it anyway, because the only way I could answer that question was to build enough of it to look at.

Choosing the Approach

Before we wrote a single line, we spent the first stretch of the session in planning mode: no code, just talking through the problem, the options, and the ground rules. I can’t recommend this enough. It’s the same discipline as a design doc before a pull request, and it matters even more with an AI, which will happily start typing the moment you let it. So I didn’t let it. We agreed on the defaults first: what the thing had to be, and what it must never do. Only then did we pick an approach.

The first real fork was whether to build a “skill” (teach the model to call a script) or a proper MCP server. Claude laid out the trade-off; I made the call, and it wasn’t close.

A skill, or a pasted snippet, runs inside the model’s sandbox, in the conversation. It can’t safely hold my long-lived OAuth token, and it can’t reach Spotify as me without me handing secrets into a place I don’t control. That’s exactly what I refused to do. It’s also throwaway: the moment the chat ends, it’s gone. I wanted a capability I could keep, one that any MCP-capable client could pick up next year.

An MCP server gives me that. It runs on my machine, holds my token, and exposes a small, typed set of tools the model is allowed to call. That division of labor is the whole point: the model decides what should happen (“this track belongs in Focus”), and the server carries it out with least privilege and hands back structured results. I’ve read every line of the executor. The model never sees my token.

Setting the Constraints

Of all the ground rules, the one that shaped the code most was about dependencies. It’s also the part the AI would never have imposed on itself.

Left to its defaults, Claude reached for the comfortable stack: requests for HTTP, python-dotenv for config, mcp for the protocol. Reasonable, popular, and more than I wanted holding write access to my account. So I set a rule: standard library unless a dependency earns its place, and I made it justify each one.

python-dotenv parses KEY=value lines. My .env has four of them; a ten-line stdlib function does the same job, environment-variable precedence included. And requests is lovely, but my entire HTTP surface is a few GETs, POSTs, and DELETEs with a bearer token. urllib covers that in about thirty lines, and now I own every byte on the wire:

def request(method, url, headers=None, params=None, json_body=None, form=None, timeout=30):
    headers = dict(headers or {})
    if params:
        url = f"{url}?{urllib.parse.urlencode(params)}"
    body = None
    if json_body is not None:
        body = json.dumps(json_body).encode()
        headers.setdefault("Content-Type", "application/json")
    elif form is not None:
        body = urllib.parse.urlencode(form).encode()
        headers.setdefault("Content-Type", "application/x-www-form-urlencoded")
    req = urllib.request.Request(url, data=body, headers=headers, method=method)
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return r.status, r.headers, r.read()
    except urllib.error.HTTPError as e:
        return e.code, e.headers, e.read()

That left one dependency: mcp, the thing that defines the project. For a tool holding an OAuth token, a dependency graph of one is a feature, not austerity: smaller attack surface, less to audit, nothing to silently update out from under me. The rest is uv and Python 3.14. None of that trimming happens unless the engineer asks for it. The model optimizes for “works”; I optimize for “works, and I’d defend it in a review.”

Design

With the approach and constraints fixed, the design fell out quickly. patchbay¹ is about 300 lines of Python behind two entry points: a one-time browser authorization using Authorization Code + PKCE (no client secret to store; token cached under ~/.config/patchbay/ at 600), and the MCP server itself, speaking stdio with eight tools: read liked songs, read playlists, read a playlist’s tracks, search the catalog, create a playlist, add tracks, remove tracks, and remove liked songs. Scopes limited to library and playlist read/write. Nothing else.

graph TD
  Me((Me)) --> LLM[LLM / Claude]
  LLM -->|tool calls| MCP[patchbay MCP server<br/>local, holds my token]
  MCP -->|Web API over OAuth| API{{Spotify Web API}}
  API --> Acct[(My account:<br/>liked songs + playlists)]
  MCP -->|structured results| LLM

Under the hood, that’s the shape of an MCP server at its most minimal. Two deterministic Python scripts: patchbay-auth runs the browser flow and stashes the token; patchbay is the stdio server that reads it back and exposes those tools over JSON-RPC. At session start the LLM reads the tool schemas; during the session it decides when and with what arguments to call each one, and the server executes and returns structured results. Server never reasons. Model never touches Python. Small enough to read on a napkin, and a good first project for feeling out the protocol end-to-end.

With a Little Help of AI

If setting constraints is where I earned my keep, research is where Claude earned its. I walked in assuming I’d sort songs by audio features: Spotify’s per-track danceability, energy, valence, and tempo. Instead of trusting either of our memories, Claude went and checked the current API, and came back with bad news. Spotify pulled those endpoints for new apps back in November 2024, and the February 2026 changes trimmed the surface even further. A brand-new app today just gets a 403 on Audio Features. That’s an evening of building around a dead endpoint, saved before I typed a line.

I wasn’t ready to give up on metadata, though, so I asked the obvious follow-up: what about genre? Surely I could at least bucket things roughly by genre. We checked that too. Genre is still there, on the artist object rather than the track, so in principle patchbay could fetch each artist and read it. But two things killed the idea. Batch artist lookups are restricted for new apps, so it becomes one call per artist across hundreds of them. And, more damning, Spotify has been quietly hollowing the field out: people who track it have watched their genre coverage collapse over the past year, and a deprecated tag feels inevitable. Wiring my organization to a field Spotify is actively abandoning was a no. So we didn’t touch the code.

And here’s the thing: I didn’t need any of it. The model already knows the music. Give it a title, artist, and album, and it’ll tell you whether a song belongs in Roadtrip or Coffee about as well as a friend with good taste. The genre labels I was chasing live in the LLM already, richer and more current than a metadata field Spotify can’t be bothered to maintain. patchbay just moves songs; the taste stays in the conversation.

The First Reorg

The first live session ran about an hour, and confirmed both the workflow and the thesis. What made it work wasn’t the tooling — it was the prompt.

I opened the session with the whole context laid out: what I wanted (mood/occasion playlists, no genre or year splits), what wasn’t working (bloated playlists, songs never playing), what to preserve (Portuguese naming for BR-flavored variants), and the occasions I regularly listen for. That single message did the heavy lifting. Every downstream decision (the playlist set, the naming convention, the assignment rules) traced back to it. If I’d opened with “reorganize my playlists,” I’d have gotten a generic answer and spent the hour negotiating away from it.

The session ran in explicit phases with pauses between them. Discovery fanned out one subagent per playlist and one for Liked Songs, each returning a compact character sketch: top artists, sample tracks, genre mix, occasion fit. That was the only way the model could reason about ~1,900 unique tracks without blowing out the context window. Planning turned the sketches into 22 mood-anchored playlists, each with a one-line charter. I pushed back on a couple, added a Rain playlist for the beautiful-melancholy pocket, held the line on strict occasion buckets. Assignment cycled through two cleanup passes: the first attempt used a rule engine and dumped 173 low-confidence tracks into coffee_day as a fallback bucket; I called it out, and a second pass re-classified each track on individual judgment. Review, then execution.

Execution turned up a live example of the February 2026 changes mentioned earlier. Halfway through, DELETE /me/tracks returned a 403 — the endpoint had been consolidated into DELETE /me/library. First patch shipped the URIs in a JSON body; Spotify wanted them as a comma-separated query parameter capped at 40. Two commits, tests updated, session resumed. And earlier in the same session we’d already added a delete_playlist tool that patchbay had never had — Spotify models playlist deletion as unfollow your own playlist, and I’d never wired the endpoint. Ten more lines, committed; tool count from eight to nine.

The result: 22 new playlists live on Spotify, from Focus (39 instrumental tracks) to Nostalgia (319 MTV-era anthems), 2,737 total assignments across 1,872 unique tracks — average 1.46 playlist tags per track. The twelve old playlists are gone; Liked Songs is empty. I’m still spot-checking placements as I listen, and so far so good.

At the end of the session, we lifted the workflow into a /reorg-playlists skill, plus a companion /audit-playlists for monthly hygiene. Next reorg won’t need me to re-explain the phased structure or the “no fallback bucket” rule or the ISRC dedup — the skill carries those forward. That’s what “let the machine compress the typing” should actually look like: not a one-off session, but a reusable capability that ships the lessons of the run that made it.

Process

Day to day, nothing changed: I still like songs on impulse. The organizing splits by cadence, and both halves live in skills now.

Every few weeks I run /audit-playlists: it reports playlists bloating past 300 tracks, cross-remaster duplicates (same recording under multiple Spotify IDs), tracks that have leaked into more than three playlists, and any Liked Songs accumulation from the last cycle. Findings first, fixes only after I approve each one.

When the taxonomy itself feels stale (probably rare, once a year at most) /reorg-playlists runs the full teardown with the phase structure and the “no fallback bucket” rule baked in. The first one already happened; the next will look the same because the skill carries the lessons forward.

The safety invariant carries across both: build and fill first, unlink last, so a bad call never strands a track.

Wrap Up

The honest summary is that the tools changed and the method didn’t. The AI did the research I’d have lost an evening to, and the implementation I’d have lost a weekend to. Both well. But it reached for the popular libraries, not the minimal ones; it didn’t care about my supply chain, my threat model, the name, or the framing until I told it to. Every real decision was mine, and so is the accountability. That’s the job, and it’s still the job.

There’s a bigger decision hiding under this whole project, though, and I think it matters more than the project itself: knowing when to keep AI in the loop, and when to keep it out. My default is out. If a problem can be solved with a plain script, I write the script. It costs me more time up front, but once it ships it runs for free: no tokens, no latency, no bill that shows up every time it runs. A solution that calls an AI on every execution is the opposite: faster to build, because you let the model absorb the messy logic instead of coding it, but it charges you on every run, forever. Fixed cost versus recurring cost. In my experience the plain script usually wins.

Patchbay lands on the recurring-cost side of that line, and I want to be honest about it, because my first instinct was to file it as an exception. It isn’t. Yes, I used AI to build it, once, and the server itself is deterministic and burns nothing on its own. But the solution is useless without an AI driving it: every organizing session spends tokens, and it will for as long as I use it. That’s AI in the runtime loop, full stop. I’m fine with it here for two reasons. First, the recurring cost buys something a script genuinely can’t: the fuzzy, taste-based call of whether a song belongs in Focus or Coffee. Second, I kept that cost boxed in: the AI is invoked only for the decision, only when I choose to run it, a handful of times a year, while every mechanical step (auth, reading and writing the library) stays deterministic and free. The discipline was never “avoid recurring cost.” It’s “put the AI only where the loop actually needs a brain, and keep its footprint as small as the problem allows.” (And the plain-script side isn’t truly free either; Spotify’s API will change and I’ll have to touch this again. But “maintain occasionally” and “pay per run” are very different bills.)

Three sentences, then: define the problem and own the constraints yourself; let the machine compress the research and the typing; and remember that the moment you stop deciding is the moment you’ve stopped engineering. The code is on GitHub: github.com/lopes/patchbay, small enough to read in one sitting, which was the entire point.

Full Review

Having learned best practices on working with AI, I resumed the work with Lantana by asking Claude to review the codebase to spot and fix errors and inconsistencies, plus opportunities for improvement. It found many low- and medium-severity issues that required changes, especially in shell scripts and configuration files. It was especially helpful dealing with Ansible and Linux configuration files, like systemd and logrotate — things that are predictable but require extra attention.

To my surprise, it approved all major technical decisions, so the stack remained the same. The only major change it made was to the data pipeline: instead of using Vector to parse logs into OCSF format, it suggested limiting its use to IP geolocation enrichment and forwarding the logs to the Bronze layer. All parsing would be done in Python because doing so would make future maintenance easier and yield better code overall. It would also better match the Medallion architecture, as the Bronze layer expects raw logs — more on this architecture in the first post.

A small caveat here: even with OCSF as the target, Claude pushed back to keep things practical. Since the Silver layer uses Parquet files, and working with deeply nested structs adds tooling complexity, storing src_endpoint.ip-style fields would make downstream querying harder. We decided to flatten all fields instead, creating an OCSF-inspired schema, not pure OCSF. Fields remain deterministic without the extra complexity.

While onboarding Claude, I also reconsidered the use of auth logs. Although useful for incident investigation, including them in datasets would add noise without meaningful signal, so we dropped them. The new architecture looked like this:

flowchart TD
  ATK(["Attacker"])

  subgraph HWZ["HONEYWALL ZONE"]
  NFT["nftables<br/>packet filter + log"]
  SUR["Suricata<br/>NIDS"]
  end

  subgraph SENS["SENSOR ZONE"]
  COW["Cowrie<br/>SSH, Telnet"]
  DIO["Dionaea<br/>HTTP, MySQL…"]
  end

  subgraph CLZ["COLLECTOR ZONE"]
  subgraph STREAM["Vector (all zones)"]
    GEO["GeoIP<br/>MaxMind GeoLite2"]
  end
  BRZ[("🥉 BRONZE<br/>raw, GeoIP")]
  subgraph BATCH["D+1"]
    NRM["OCSF normalization"] --> IOC["IOC extraction"] --> CACHE[("cache")]
    CACHE -- "miss" --> EXT["VirusTotal, GreyNoise…"]
    EXT -- "store" --> CACHE
  end
  SLV[("🥈 SILVER<br/>normalized, enriched")]
  GLD[("🥇 GOLD<br/>top IPs, credentials, geo…")]

  GEO --> BRZ --> NRM
  CACHE -- "enriched" --> SLV --> GLD
  end

  ATK -. "attack traffic" .-> NFT
  NFT -. "attack traffic" .-> SENS

  NFT & SUR -- "logs" --> GEO
  COW & DIO -- "commands, credentials, files…" --> GEO

With Claude joining the team and eager to help, it was time to finally start the core of this work: create the data pipeline and the data lake.

Data Pipeline Development

Right after reviewing and improving the codebase, Opus started what I was calling “Phase 2”: the data pipeline implementation. After a few rounds of back-and-forth with Claude to plan the tasks (including letting it interview me), I gave it the green light to start coding. I instructed it to follow Test-Driven Development (TDD) and moved to the passenger seat to watch it work. I asked it to create a phased plan, grouping tasks by functionality, and it was a good call. I was aware I had to keep the context window under control, and with so many tasks to do, it would inevitably reach it at some point. We’d need to pause and restart in a new session — I prefer that over simply compacting the current context.

Once we reached around 50% of the context window, I instructed it to wrap up, persist lessons learned, and update the plan — I asked Claude to persist it in a temporary PLAN.md file. It took around four sessions before Claude considered the plan finished. By then, we had plenty of unit tests in place to ensure new code wouldn’t break existing functionality. At that point, Claude confirmed we had a working pipeline, but I was skeptical. So I started asking it to create scripts to reprocess logs from the Alpha version — recall from the last post that I left it running early in March. I pulled the raw logs from the server and started testing locally.

Claude and I found many bugs triggering the enrichment and transformation routines — Silver and Gold, respectively. Once they were all fixed, I had better confidence it would work in production. Since Alpha was stable and collecting data, I wanted to replace it with a solid version. Next, I wanted to verify the third-party enrichments were working as expected, so I asked Claude to create a script to let me check them. During this validation, I found many bugs, mostly from poorly implemented API consumption. It was during this step that, due to numerous errors trying to use Phishstats, I decided to drop support for it.

While validating the enrichment code, I noticed it was taking too long to process. Processing a full day’s logs was taking around six hours. Analyzing the logs, I noticed that most of the time the pipeline was trying to enrich data from external services — but as my quotas were exhausted, it was in vain. Then it clicked: check whether the previous N enrichment attempts had failed due to rate-limit errors. If so, skip external enrichment entirely for the remaining logs. After raising this with Claude, it promptly acknowledged and implemented the fix. After that, the whole enrichment pipeline ran in less than five minutes, a ~99% drop. In my opinion, this is a good example of my work with AI on this project. Alone, neither of us would have reached that result; working as a team, we did.

All of this happened in roughly four weeks: onboard Claude, review the codebase, implement and test the pipeline. I declared the version stable enough to move to the VPS and did so on May 23rd — the Beta version, the first end-to-end version deployed to the VPS, from server setup to dashboard generation. When I first planned this project, I was prepared to spend months working on this pipeline, but the new way of working with AI sped things up a lot. I hadn’t imagined creating all of that in less than a month.

A Working Pipeline

While the Beta version was running, I started reviewing my notes and analyzing reports to tune minor details. Coming from Detection Engineering, I know that not all IOCs are equal, so I reconsidered the TTLs used in cached IOCs for third-party enrichment. I defined that benign IOCs (not flagged as malicious by the platforms) would expire in one week. Malicious ones would vary by type: 28 days for IP addresses and domains, and 56 days for hashes. The underlying logic was: if something is benign, it can become malicious faster than something malicious can become benign.

At this point I could see the data pipeline working for the first time, and I must admit it felt great. Raw logs from Cowrie were written to disk like this:

{
  "eventid": "cowrie.session.connect",
  "timestamp": "2026-06-07T00:00:18.800035Z",
  "src_ip": "101.36.122.139",
  "src_port": 52912,
  "dst_ip": "::ffff:203.0.113.97",
  "dst_port": 2222,
  "session": "0c3197d4ef25",
  "message": "New connection: 101.36.122.139:52912 (::ffff:203.0.113.97:2222) [session: 0c3197d4ef25]",
  "sensor": "1e4f54c1da3d",
  "protocol": "ssh"
}
{
  "eventid": "cowrie.client.version",
  "timestamp": "2026-06-07T00:00:18.800937Z",
  "src_ip": "101.36.122.139",
  "session": "0c3197d4ef25",
  "version": "SSH-2.0-libssh_0.9.6",
  "message": "Remote SSH version: SSH-2.0-libssh_0.9.6",
  "sensor": "1e4f54c1da3d",
  "protocol": "ssh"
}
{
  "eventid": "cowrie.login.failed",
  "timestamp": "2026-06-07T00:00:19.906380Z",
  "src_ip": "101.36.122.139",
  "username": "igor",
  "password": "igor123",
  "session": "0c3197d4ef25",
  "message": "login attempt [igor/igor123] failed",
  "sensor": "1e4f54c1da3d",
  "protocol": "ssh"
}

Vector picks up the raw log, adds pipeline metadata (dataset, operation, source_type, server), and runs an offline MaxMind GeoLite2 lookup at wire speed — no external API call, no latency. The geo struct is the only enrichment at this stage, as seen in the next listing, which is how a log lands on Bronze layer.

{
  "eventid": "cowrie.session.connect",
  "timestamp": "2026-06-07T00:00:18.800035Z",
  "src_ip": "101.36.122.139",
  "src_port": 52912,
  "dst_ip": "::ffff:203.0.113.97",
  "dst_port": 2222,
  "session": "0c3197d4ef25",
  "message": "New connection: 101.36.122.139:52912 (::ffff:203.0.113.97:2222) [session: 0c3197d4ef25]",
  "sensor": "1e4f54c1da3d",
  "protocol": "ssh",
  "dataset": "cowrie",
  "operation": "Alpha",
  "server": "sn-01",
  "source_type": "vector",
  "geo": {
    "asn": 135377,
    "isp": "UCLOUD INFORMATION TECHNOLOGY HK LIMITED",
    "city": "Hong Kong",
    "country_code": "HK",
    "latitude": 22.2842,
    "longitude": 114.1759,
    "region_code": null,
    "timezone": "Asia/Hong_Kong"
  }
}
{
  "eventid": "cowrie.login.failed",
  "timestamp": "2026-06-07T00:00:19.906380Z",
  "src_ip": "101.36.122.139",
  "username": "igor",
  "password": "igor123",
  "session": "0c3197d4ef25",
  "message": "login attempt [igor/igor123] failed",
  "sensor": "1e4f54c1da3d",
  "protocol": "ssh",
  "dataset": "cowrie",
  "operation": "Alpha",
  "server": "sn-01",
  "source_type": "vector",
  "geo": {
    "asn": 135377,
    "isp": "UCLOUD INFORMATION TECHNOLOGY HK LIMITED",
    "city": "Hong Kong",
    "country_code": "HK",
    "latitude": 22.2842,
    "longitude": 114.1759,
    "region_code": null,
    "timezone": "Asia/Hong_Kong"
  }
}

Then, every day after midnight, the ETL runs, rotating raw logs and starting the normalization and enrichment routines that produced the Silver layer — stored on disk as Parquet. The pipeline normalizes Cowrie’s field names to the OCSF-like schema, enriches against AbuseIPDB and VirusTotal (daily batch, HTTP APIs), and redacts the honeypot’s destination IP (dst_endpoint_ip: null) to prevent WAN IP leakage into reports. One row per event; null columns exist for event types that don’t apply (e.g., command and file_* are null for a login event).

The WAN IP redaction is worth a closer look. Doing it here avoids possible leaks in reports and enables safe data sharing. If two practitioners or companies run their own Lantana instances and want to share information, they can do so freely by exchanging Parquet or STIX files — Lantana guarantees no sensitive information is present in the Silver layer or beyond.

{
  "dataset": "cowrie",
  "time": "2026-06-07 00:00:19.906380",
  "session": "0c3197d4ef25",

  "src_endpoint_ip": "101.36.122.139",
  "dst_endpoint_ip": null,

  "user_name": "igor",
  "unmapped_password": "igor123",
  "auth_protocol": "ssh",
  "is_cleartext": true,

  "class_uid": 3002,
  "category_uid": 3,
  "type_uid": 300201,
  "activity_id": 1,
  "status_id": 2,
  "severity_id": 2,
  "metadata_version": "1.3.0",
  "metadata_product_name": "Lantana",

  "geo.country_code": "HK",
  "geo.city": "Hong Kong",
  "geo.asn": 135377,
  "geo.isp": "UCLOUD INFORMATION TECHNOLOGY HK LIMITED",
  "geo.latitude": 22.2842,
  "geo.longitude": 114.1759,
  "geo.timezone": "Asia/Hong_Kong",

  "abuseipdb_confidence_score": 100,
  "abuseipdb_total_reports": 6638,
  "abuseipdb_risk_score": 100.0,

  "vt_malicious_count": 12,
  "vt_suspicious_count": 3,
  "vt_ip_reputation": -8,
  "vt_as_owner": "UCLOUD INFORMATION TECHNOLOGY (HK) LIMITED",
  "virustotal_risk_score": 100.0,

  "shodan_risk_score": null,
  "greynoise_risk_score": null,

  "message": "login attempt [igor/igor123] failed",
  "operation": "Alpha",
  "server": "sn-01",
  "sensor": "1e4f54c1da3d"
}

Once enrichment is written to Silver, the transformation routine runs as the next step in the same nightly job — reading the day’s enriched Parquet and deriving Gold tables: insights computed from Silver that surface things like top credentials, IP address reputation scores, and Suricata alert rankings. These tables are what feed the dashboard and Discord brief. For the curious, the Appendix lists all field mappings, from raw logs to the OCSF-like format.

{
  "src_endpoint_ip": "101.36.122.139",

  "total_events": 96,
  "datasets": ["suricata", "cowrie"],
  "auth_attempts": 16,
  "auth_successes": 0,
  "unique_usernames": 3,
  "unique_passwords": 4,
  "commands_executed": 0,
  "findings_triggered": 0,
  "downloads": 0,

  "first_seen": "2026-06-07 00:00:18.800035",
  "last_seen": "2026-06-07 00:06:08.498538",

  "geo_country": "HK",
  "geo_city": "Hong Kong",
  "geo_asn": 135377,
  "geo_isp": "UCLOUD INFORMATION TECHNOLOGY HK LIMITED",
  "geo_latitude": 22.2842,
  "geo_longitude": 114.1759,

  "abuseipdb_score": 100,
  "abuseipdb_reports": 6638,
  "vt_malicious": 12,
  "vt_reputation": -8,
  "greynoise_class": null,
  "greynoise_noise": null,
  "shodan_ports": null,

  "abuseipdb_risk_score": 100.0,
  "virustotal_risk_score": 100.0,
  "shodan_risk_score": null,
  "greynoise_risk_score": null,
  "enrichment_risk_score": 100.0,
  "behavioral_risk_score": 1.6,
  "risk_score": 50.8
}

Honeypots

It took around five weeks from onboarding Claude to reach this point. With a working pipeline, I decided to add the second honeypot to the stack: Dionaea. Getting Cowrie running and properly configured had been straightforward — despite some speed bumps dealing with obscure features, I had it working by December. Since then, I had been improving and polishing the configuration, especially to ensure smooth IPv6 support. I have to admit I found it fun to tail cowrie.json and watch adversaries log into the honeypot in real time.

It was time to move Dionaea to production. Given my experience with Claude, I stayed in the back seat and simply asked it to implement Dionaea support following the project’s standards and Cowrie’s configuration patterns. I told it to make full use of the narrative defined in narrative.yml from planning time. It took just a few minutes for Claude to produce the first version. I stayed in the loop only for the server deployment, to keep a minimal level of control.

Claude struggled to add support for Dionaea’s SIP module due to the strict container rules we were applying. This module creates SQLite state as the supervisor (root) and then tries to access it from the worker process (running as the dionaea user after a privilege drop) — a race condition the upstream image was never designed for, since most Dionaea deployments run root end-to-end. We had a few options: build a custom image (hard), relax the restriction (less safe), or drop SIP support (fewer features). I chose the last one. From my research, SIP tends to attract more scans than real attacks, and security matters most for this project. If I or any Lantana operator ever wants to enable SIP, it warrants a proper review.

To wrap things up and keep the narrative consistent, I asked Claude to create a few artifacts, like a mock web page for the HTTP module using the project’s established narrative. Claude delivered almost instantly. Along the same lines, I created a Claude skill to make narrative creation easier. AI is well-suited for this. We built the scaffold-narrative skill, which takes one argument: the context. It can be a single sentence or a paragraph of text describing the scenario. Based on that, it fills in the narrative variables correctly and consistently.

Under the Hood

For this data to be properly generated, the underlying stack has to be solid. System metrics were green and resource usage stayed under control — not bad for a 4 vCPU, 8 GB RAM, 75 GB SSD NVMe VPS running a single-node Lantana with Cowrie and Dionaea simultaneously with a data pipeline for parsing and enrichment.

The only thing that didn’t work as expected was IPv6. Not that it didn’t work, exactly — but it was stressful. Many tools still don’t support it well, so I had trouble testing it locally. Later, I had to add a prune step to Cowrie logs as it tunnels IPv4 inside IPv6 in dual-stack mode, causing IPv4 addresses to be prefixed with ::ffff:. During validation, I hit another IPv6 issue: nftables logs with addresses in the long form (zeros instead of :: shortcuts) were preventing the sanitization routine from running. It highlights that, decades after publication, IPv6 still lacks the solid tooling support needed for broader adoption. Without it, people will keep ignoring it or, worse, leaving it enabled but unconfigured — grist for adversaries’ mill. That’s the reason I insisted on it here as a prerequisite.

The mental model in the next diagram captures the best way to understand Lantana. From my workstation, I deploy and configure the server. It receives attacks whose telemetry is properly processed to generate intelligence.

flowchart TD
  WS(["Workstation"])
  SRV["Server"]
  ATK(["Internet"])
  THIRD["VirusTotal, AbuseIPDB…"]
  DISC(["Discord"])

  WS -- "① provision (Terraform)" --> SRV
  WS -- "② configure (Ansible)" --> SRV
  ATK -- "attacks" --> SRV
  SRV -- "daily enrichment" --> THIRD
  THIRD -- "threat intel" --> SRV
  SRV -- "daily brief" --> DISC

One of the biggest gains in this project was using security tool logging (nftables and Suricata) from the honeywall zone. Honeypot applications usually log events only at the application level. What about malformed packets at the network layer? Incorporating firewall and NIDS logs complements the honeypot logs and creates a fuller picture of interactions with the decoys. One caveat is that Suricata logs are voluminous, so we only used the event types that add the most visibility without too much overlap with other log sources. Suricata alerts, for example, proved to be very useful and have a dedicated tab in the dashboards.

The underlying stack behaved beautifully. Ansible deploys were painless and idempotency worked as expected. Debian and the Linux stack ran like clockwork. Podman and Vector proved to be stable and reliable. I had to put some effort into Podman configuration for security reasons, but its integration with systemd via Quadlets was clean — it fits the Linux environment well. Linux accepts many configurations, but deviations from the standard become technical debt; sticking to the grain here paid off.

It’s also worth saying that the automated tests created to validate the deployment were a great call — they caught errors that would otherwise have taken days to surface. Ansible makes a sysadmin’s life much easier.

Host Security

Lantana is designed around defense-in-depth: each layer is hardened independently so a breach at one does not automatically compromise the others. The single-node deployment makes this concrete because it collapses all zones onto a single host, making every security boundary visible in one place.

flowchart TD
  ATK(["Internet"])

  subgraph HOST["Debian"]
    NFT["nftables"]
    SUR["Suricata — NIDS"]
    subgraph POD["Podman — rootless"]
      COW["Cowrie<br/>DropCapability=ALL<br/>NoNewPrivileges · read-only FS"]
      DIO["Dionaea<br/>DropCapability=ALL<br/>+NET_BIND_SERVICE (minimal)"]
    end
    LOG["rsyslog<br/>auth + audit"]
  end

  ATK -->|"attack traffic"| NFT
  NFT -->|"DNAT"| COW & DIO
  NFT -.->|"tapped"| SUR
  POD -.->|"tapped"| SUR

Debian is the foundation. Two unprivileged system accounts handle all runtime activity: stigma (UID 2001) owns the honeypot containers and nectar (UID 2002) runs the pipeline — both with /usr/sbin/nologin shells, never logging in interactively. SSH access uses a non-standard port with key-based authentication only. The system clock is UTC throughout, which matters for log correlation.

nftables enforces a strict default-drop policy on all three chains: input, forward, and output. Inbound, only honeypot ports are accepted and DNAT’d to the container layer; the management SSH port is the only other entry point. Outbound is allowlisted to DNS, NTP, and HTTPS — nothing else leaves the host. Sensor-to-WAN egress through the forward chain is rate-limited to 10 packets per minute to prevent a compromised sensor from being weaponized. Every dropped packet is logged with a prefix tag that feeds directly into the data pipeline.

Suricata monitors both WAN and LAN interfaces in AF_PACKET tap mode, without sitting in the traffic path. Custom rules detect SSH brute force, scripted client fingerprinting (libssh, paramiko, Go), SMB/FTP/MySQL brute force, and malware downloads (PE, ELF, shell scripts). Only alert and flow event types reach the pipeline — enough signal, manageable volume.

Podman runs containers in rootless mode: no daemon, no root. Both honeypots run as stigma:2001. Cowrie drops every Linux capability (DropCapability=ALL) and mounts its root filesystem read-only with NoNewPrivileges=true. Dionaea needs a narrow set (NET_BIND_SERVICE to bind privileged ports and SETUID/SETGID/CHOWN/FOWNER for its entrypoint privilege drop); everything else is stripped. Both containers are CPU- and memory-capped.

rsyslog captures auth and audit events from the host itself. These are intentionally excluded from the data pipeline dataset — they add noise without meaningful threat intelligence signal — but are preserved on disk and readily available should an incident require forensic investigation.

Release Candidate

It took some days of Beta running to notice it was stable enough for a release candidate. On June 8th, I wiped the server again and deployed the whole system from scratch to validate end to end. All clear. The RC version was live and final validation had begun.

The most damaging issue surfaced only on June 11th and had been silently active since June 8th: the upstream cowrie/cowrie:latest image was updated and bumped the internal cowrie user UID from 998 to 999. The ,U volume flag used in the Quadlet (Podman) to map ownership only fires at first volume init — it didn’t re-fire after the rebase. Login events kept arriving normally, but every interactive session crashed internally when Twisted (Cowrie’s async networking engine) tried to write the TTY transcript to a path it no longer owned. The exec path aborted silently before cowrie.command.input was ever emitted. Four days of attacker command intel were gone, and nothing in the pipeline flagged it as a failure. The fix replaced ,U with UserNS=keep-id:uid=999,gid=999, which maps the rootless host user directly to the image user regardless of internal UID — with the coupling made explicit in the config so any future upstream rebase triggers a review.

Dionaea had its own image issue first: the upstream latest tag had been neglected for years — nightly replaced it, workable if less polished. The volume ownership was a different variant of Cowrie’s problem: its container entrypoint chowns the log and lib directories to the worker user on every start. The ,U flag was also firing on every start, racing the entrypoint and reassigning ownership back to the host user, leaving the worker unable to reopen its log files after any container restart. Any restart meant no Dionaea events afterward — silently. Dropping ,U entirely and letting the entrypoint own the chown fixed it, same day.

Two memory problems surfaced during this period. The first was the Gold transform: it was loading seven days of Silver data in a single pass to compute slow-burn attacker progression across days. On a 4 GB VPS with no swap, that lookback window was untenable. The feature was dropped and the multi-day view rebuilt by aggregating over the smaller, already-computed per-day Gold files instead. The second was the enricher itself being OOM-killed. Polars materializes the full day’s Bronze NDJSON into memory during normalization — Suricata’s stats and netflow event types had no consumer in the pipeline but were bloating those files well beyond their typical size. When the OOM killer fired, the day’s intel was lost silently: the Discord brief posted “No data available” with a green embed, giving no indication anything had gone wrong. Filtering stats and netflow at the Vector level before they reached Bronze brought file sizes back to normal.

Separately, the VirusTotal daily quota was being exhausted in minutes. The cause: Cowrie’s cowrie.ttylog events carry a shasum field — the hash of the session transcript file, not a downloaded artifact. The hash extractor was picking up every shasum field indiscriminately, flooding the VT queue with thousands of TTY log hashes per day. Filtering the extractor to only file_download and file_upload event IDs cut the queue by an order of magnitude. That same fix uncovered a second gap: file_upload events had never been mapped in the normalizer at all, meaning every file an attacker uploaded to the honeypot (SSH keys, tools, scripts) was being silently discarded in Silver.

With hashes flowing correctly, a semantic problem appeared in the Discord reports. The “Malware Captured” section was dominated by SSH public keys. Attackers frequently drop their own pubkey into ~/.ssh/authorized_keys for persistence, and those file transfers generate hashes just like malware downloads. Because persistence attempts vastly outnumber ELF downloads, a SHA-256 of an SSH pubkey was consistently the top entry in the malware table. The fix added a file_intent classification to Silver: persistence for destfiles matching known persistence targets (SSH authorized keys, shell rc files, sudoers), probe for known-noise hashes (empty files, single-byte writes), and malware as the default for everything else. The “Malware Captured” section was scoped to file_intent='malware' only, and SSH-key drops got their own dedicated “Attacker Persistence” section in the brief — so the intel wasn’t hidden, just properly separated. That was the last major fix, on June 19th.

The validation phase took more time than expected, June 8th to 19th, considering development took one month, because it took time to find the bugs — once bugs were found, it usually took less than 10 minutes to solve them, and the project was left untouched until the next bug was found some days later. Fixing them was super fast because Claude’s not only a great programmer: it excels in system administration. Part of my process of leaving the driver’s seat included letting Claude Code connect to my server and run commands via SSH. It’s impressive how fast Claude can troubleshoot a problem by running commands and reasoning about them. I estimate it would have taken me minutes or hours to recall commands to run, execute them, and process the results to figure out what problem was happening and how to solve it. Claude did it in less than five minutes usually.

With a Little Help of My (AI) Friend

It’s clear at this point that Claude drastically and positively impacted this work. Truth must be told: without Claude, it would have taken me more than a year to develop all of this. With AI, it took exactly 24 days (1–4 hours/day) of development, plus 11 days of validation to reach this point. Taking a moment to learn the new way of working with AI definitely paid off.

I got to see both the old and the new way of working with AI — 2025 and 2026. Adding AI to a workflow as an afterthought was stressful and not very productive. Having to act as the LLM’s eyes and hands, with no shared context between sessions, led to some prompt rage. Conversely, rethinking the workflow to adopt AI earlier (treating it as an assistant or peer, using agents) was a transformation I thought I’d never see.

My experience with Claude Opus was equally great. I’m still amazed by how precise and helpful this model is. By far, my best experience with AI. At some point, the underlying stack became transparent to me, as Claude handled it all cleanly. I have to admit that after onboarding Claude, I hadn’t typed a single line of code — so Streamlit and Polars, despite the great results here, remain skills I still want to learn. That confirms my idea: AI will commoditize code. Not to mention the SRE tasks Claude Code excelled at. I’m still impressed by how fast and precise it was at managing the server. The best Sysadmin I’ve ever seen in action.

Opus and Claude Code prove that a capable LLM with the right tooling is a great companion for any engineer. Claude is undisputedly faster and better than me at writing code and troubleshooting servers. In a matter of seconds, it logs into my server via SSH, runs commands, processes the results, and shows exactly what’s wrong along with hypotheses for fixing it. Despite being initially apprehensive, I think Claude is a great companion for my work. It can’t replace me. Alone, neither Claude nor I would have achieved such great results — but together, we’re stronger.

I see Claude as a tool or companion I’d like to always have with me, to work faster, better, and more efficiently — not as a threat. So far.

What’s Next

Wow! So much done in nearly two months! Full pipeline development, last features created, code reviewed, bugs fixed, project validated, v1 alive. I’ll leave it running for a couple more weeks and, instead of focusing on finding bugs, I’ll contemplate the whole set working, especially the data pipeline and the reports. In the next post of this series, I’ll share an analysis of it all. Until then, feel free to experiment with Lantana. Code on GitHub.

Lantana Series

• Lantana 1: Honeypot as code
• Lantana 2: The Data Pipeline
• Lantana 3: Output and Lessons

Appendix

The Foundation

I started by analyzing T-Pot and other open-source honeypots to sketch out the project. It was early December 2025. As I mentioned, T-Pot is a great project, but its stack is too bloated for my purposes: more than 20 honeypots supported, the default installation pays no attention to customizing the narrative (something extremely important to create convincing decoys that attract more than automated scanners) and most importantly, the data pipeline is entirely built on the Elastic Stack (Elasticsearch, Logstash, and Kibana — ELK). It works, but it’s heavy, requiring more powerful servers, and the data processing is hidden behind ELK.

However, T-Pot had a few decisions I liked: it creates a dummy NIC on the server to segregate network traffic, it implements Suricata (an IDS) to supplement honeypot data, and containerizing honeypots proved to be a great isolation approach. All things I wanted to replicate.

After some research, I decided on the foundational stack:

• Target OS: Debian 13. Open-source, widely supported, with plenty of tools and hardening options. A good balance between up-to-date packages and tested software.
• Deployment method: Ansible. I won’t deny it: in the early days, I started with shell scripts to configure the server, just like years ago. But it didn’t take long to realize that was against my goal of using modern methods. Ansible stood out as the best choice: idempotent, widely used, Red Hat-maintained, stable, and secure.
• Honeypots: Cowrie and Dionaea. After reviewing T-Pot’s supported honeypots, these two stood out as the most recommended and widely used. Together they cover most of the protocols I had in scope.
• Data pipeline: Python, OCSF, Polars, and Streamlit. Python offers the best balance between speed and usability for this kind of work. Rust would offer better performance, but Python has the right trade-off. Polars and Streamlit followed naturally: I had read comparisons praising Polars for its speed over Pandas, and this Streamlit demo convinced me it was the right tool for delivering a dashboard rather than a notebook for data exploration. As a Google Chronicle user, I initially considered parsing logs to UDM format, but preferred an open-source, community-driven alternative: OCSF.

With this foundation defined, I started testing and experimenting with the tools while building the early version of Lantana. It was during this phase that I chose the name. I wanted a poisonous plant, common across the world, easy to propagate, and beautiful. Lantana 🏵️ grows everywhere in my region, so it was an easy choice.

Early Days of Development

Between December 2025 and January 2026, I was experimenting with this basic environment. The first major blocker came when the server was ready and I had to deploy the first honeypot. I decided to deploy only one at that point, since one would generate enough data to validate the rest of the configuration. Once validated, I’d add the second. Since Cowrie had more documentation and community resources, I went with it.

I added Docker to the stack and deployed a containerized Cowrie instance — another nod to T-Pot. No big problems so far. At this point, I realized I’d need a security tool to monitor the server, and since I was already working with Falco at work, I chose it. It was during the first security checks that I noticed Docker was bypassing many of my firewall rules. I was using iptables at the time, and making the two work together was painful.

After about two stressful development weeks and many parallel research threads (not full-time), I found I’d have to disable Docker’s iptables integration entirely to move forward. It was around then that I discovered Podman. It doesn’t configure the firewall by default and is praised for being daemonless. Part of my security concern with Docker centered on container escape: even with containers configured to run as unprivileged users, Docker’s daemon runs as root, leaving a privilege escalation path open on the host. Podman’s rootless mode closes that gap, as the entire runtime runs as a regular user.

It led to the first major change in the project: Docker out, Podman in. The Docker/iptables conflict also made it clear I was using a legacy tool. nftables is the modern standard for Linux firewalls. On top of that, I realized I needed to modernize my approach to Linux server management: adopted iproute2 and systemd, both default in Debian. No more ad hoc commands to configure the server, no more scripts left scattered across the environment; only standardized, predictable deployments.

At this point, I finally started touching the pipeline. Dealing with logs is tricky because they’re always coming, like a stream. That’s why ETL systems often work with D+1 data availability: rotate the logs at midnight, then process the rotated file since it won’t be updated anymore. After some research, I discovered Vector, an open-source data pipeline tool maintained by Datadog. Vector can tail log files and process them in near real-time. At first, I planned to use it for basic enrichment (GeoIP) and OCSF parsing, but at this stage it would just collect and forward log data to the data lake.

The data lake: at last, the core of my work. I had never built one before, so I had to read up on how to design a simple one. I settled on the Medallion architecture. In short, it divides data processing into bronze, silver, and gold layers: from raw data to business-ready aggregations. Vector would drop logs with minimal enrichment, plus OCSF parsing, into the bronze layer, and I’d build Python scripts from there.

Finally, during this period, I realized I could replace Falco with Suricata and monitor the host at a different level. But I’d postpone that until I had a working system. With all these changes, I reached an MVP by early January. I deployed it on my VPS, and when the first logs started arriving, I was genuinely proud. I had deployed a customized Cowrie instance following the narrative I’d defined, on a public server, with a single command. In under ten minutes after hitting enter. Honeypot as Code. 🪄

Full Architecture Review

I managed to deliver that MVP right before my vacation and was confident I’d finish the full project within a month. During my time off, I read Virtual Honeypots (Provos & Holz, 2007) and it changed the direction of the project. The book made it clear my architecture needed to be more modular and support both single-node and multi-node deployments, multi-node being essential for high-interaction honeypots, whether for my own use or for anyone else running Lantana.

I took two steps back and fully reviewed what I had built. It turned out that some parts of the MVP were quick shortcuts that really needed a rework. The revised architecture looked like this:

• Debian 13 properly configured, especially SSH. Until then, I was configuring SSH via Ansible — it was painful. The better approach is to use preparation scripts or, better yet, IaC tools like Terraform for initial provisioning.
• Terraform. The IaC of choice for machine provisioning.
• Ansible. Now with revised tasks and no longer burdened with SSH configuration.
• nftables, iproute2, systemd, logrotate, and cron. Core Linux tooling. iproute2 creates the dummy ltn0 interface, nftables defines firewall rules, and systemd orchestrates services and ensures configurations survive restarts. logrotate handles daily log rotation and removes old files to keep the disk clean. cron executes periodic routines — fundamental to keeping the pipeline running.
• Podman. The container runtime for honeypots — daemonless, all containers hardened.
• Suricata. Network visibility where honeypots can’t reach, plus host monitoring at the network layer — better data across the board.
• Vector. Stream data processing.
• Data Enrichment: GeoLite2, VirusTotal, Shodan, AbuseIPDB, GreyNoise, Phishstats.
• Python, Polars, and Streamlit. The data processing layer, yet to be developed.
• Notifications: Discord.
• Multiple zones, inspired by Virtual Honeypots:
  • Honeywall. The security perimeter — internet-facing, with packet filtering and full logging.
  • Sensor zone. The Podman domain where honeypots run.
  • Collector zone. Where all data is directed, processed, and stored.
• Two modes of operation: single-node and multi-node. In single-node, all zones run on the same server, using the dummy ltn0 interface for network segregation. In multi-node, each zone can run on a dedicated server; additional collector nodes can be deployed manually, enabling high-interaction honeypot scenarios.
• Full data utilization. From the honeywall to the honeypots, almost all data the server touches flows into the pipeline.

It took one or two weeks to come up with this design. Then I implemented what I call Alpha version — functionally equivalent to the first MVP, but on a proper foundation. The overall architecture was like this:

flowchart TD
  ATK(["Attacker"])

  subgraph HWZ["HONEYWALL ZONE"]
  NFT["nftables<br/>packet filter + log"]
  SUR["Suricata<br/>NIDS"]
  AUTH["auth logs"]
  end

  subgraph SENS["SENSOR ZONE"]
  COW["Cowrie<br/>SSH, Telnet"]
  DIO["Dionaea<br/>HTTP, MySQL…"]
  end

  subgraph CLZ["COLLECTOR ZONE"]
  subgraph STREAM["Vector (all zones)"]
    GEO["GeoIP<br/>MaxMind GeoLite2"] --> NRM["OCSF normalization"]
  end
  BRZ[("🥉 BRONZE<br/>raw, GeoIP, OCSF")]
  subgraph BATCH["D+1"]
    IOC["IOC extraction"] --> CACHE[("cache")]
    CACHE -- "miss" --> EXT["VirusTotal, GreyNoise…"]
    EXT -- "store" --> CACHE
  end
  SLV[("🥈 SILVER<br/>enriched logs")]
  GLD[("🥇 GOLD<br/>top IPs, credentials, geo…")]

  NRM --> BRZ --> IOC
  CACHE -- "enriched" --> SLV --> GLD
  end

  ATK -. "attack traffic" .-> NFT
  NFT -. "attack traffic" .-> SENS

  NFT & SUR & AUTH -- "logs" --> STREAM
  COW & DIO -- "commands, credentials, files…" --> STREAM

In this diagram, dotted edges represent attack traffic (the thing being studied) and solid edges represent log/data flow (the output). Although the pipeline was yet to be implemented, its design was already complete. Note the D+1 subgraph in the Collector zone. Here’s the breakdown:

• Logs are ingested and written to disk from the honeywall and sensor zone.
• Vector reads this data in streaming mode, adding geolocation for IP addresses through MaxMind’s GeoLite2 MMDB files. It also parses logs into OCSF format, saving the results in the bronze layer of the data lake.
• On D+1, a bit after midnight, the enrichment process is triggered, fetching threat intelligence from supported services. A local cache minimizes redundant API calls. The results are saved in the silver layer.
• Some time after that, the final process is started. It analyzes the enriched data and creates aggregated data with CTI insights. This layer produces a Markdown report, a Streamlit dashboard, and a STIX file for automation.
• Once this process is finished, a message is sent to a Discord channel with a run summary and the Markdown report attached.

AI’s Help

By November 2025, Google, OpenAI and Anthropic shipped significant updates that substantially changed how AI fits into a development workflow. Basically, they improved their architectures to unlock “System 2” thinking, allowing the AI to pause, plan, and execute deep reasoning across complex, multi-hour tasks. This turned existing tools like Anthropic’s Claude Code from helpful terminal assistants into autonomous agents that could work through massive codebases on their own — read more here. However, I wouldn’t experiment with them until late March 2026, after Alpha was done.

Between December and early March, I used Google Gemini Pro the same way we used to work 84 years ago, in 2025: open a chat, share as much context as you can, and interact with the model. Sometimes I’d open Gemini CLI and try to extract something useful from it, but most of the time it would break things.

I’d create a new thread with proper context on a specific topic (the trade-offs of switching from Docker to Podman, for instance) and discuss it with Gemini as if it were a fellow practitioner. On the technical side, I’d open threads to build Ansible playbooks or configuration files like logrotate configs. Most of the time, the answers were partially correct and I had to fix small but important details.

Compared to how I work with AI now, in June 2026, my workflow back then feels prehistoric. The models were sloppy, prone to hallucination, and demanded constant supervision. Lacking proper context window control, I constantly battled context saturation; a frustrating loop that frequently ended in pure prompt rage. Still, despite the friction, the AI genuinely helped me reason through critical design decisions. The proof is in the result: the foundation I settled on by early March held up perfectly, fully validated by the smarter models that came later.

After delivering Alpha, I decided to step back from Lantana and spend some time studying the new AI landscape. The buzz was real: something had fundamentally changed. Since my employer was also pushing in this direction, I adopted Claude, and it was the best decision. Claude and this new way of working with AI would speed up Lantana considerably.

What Comes Next

Alpha version was done by early March 2026. The foundation was solid, but the data pipeline was still unwritten — and the AI tooling I’d be using to build it was about to change completely.

In the next part, I’ll pick up in April, when I returned to Lantana with Claude Code. That’s where the real pipeline work happened: Vector transforms, Polars processing across the three data layers, Suricata wired into the stack, and all the integration friction that only surfaces when you start connecting everything. The post ends with Lantana v1 live.

Lantana Series

• Lantana 1: Honeypot as code
• Lantana 2: The Data Pipeline
• Lantana 3: Output and Lessons

Part I: Foundation

Part II: Operational Layer

graph TD
    Atk((Attacker))
    R[Router]

    Atk -->|internet| R

    subgraph PubNet [Public Corporate Network]
        subgraph ResNet [Research Honeynet VLAN]
            EHW{Honeywall}
            RH[High/Low-Interaction Honeypots<br/>Dedicated IPs]
            EHW -->|controlled routing| RH
        end
    end

    EFW{{Edge Firewall}}

    subgraph LAN [Internal LAN]
        Foothold[Compromised<br/>Production Asset]
        PH[Low-Interaction<br/>Production Honeypots]
        HT[/Honeytokens/]
        Foothold -.->|lateral movement| PH
        Foothold -.->|lateral movement| HT
    end

    R -->|exposed VLAN| EHW
    R --> EFW
    EFW -->|breach| Foothold

    EHW -->|capture + egress control| DP[(Data Pipeline<br/>Intel Team)]
    RH -->|host telemetry| DP
    PH -->|application logs| SIEM[(SIEM)]
    HT -->|alarm triggers| SIEM

Part III: Getting Started

The good news is that the entry barrier in 2026 is much lower than it was in 2007. The bad news is that most of the public tutorial content hasn’t caught up with current practice. Here’s the path I’d recommend.

Wrap Up

The cat-and-mouse game has evolved since Virtual Honeypots was published, but the foundations are rock solid. Honeypots flip the asymmetry: instead of defenders needing to be right 100% of the time, the attacker only has to be wrong once — by touching a system that shouldn’t exist.

If I had to compress this entire post into three sentences: place your decoys where adversaries already want to go, treat the honeywall as non-negotiable, and ship a honeytoken capability before you ship your first honeypot. Everything else, the narrative work, the containerisation, the enrichment pipeline, falls out of those three commitments.

I’ve seen many people publishing flashy posts about the honeypots they deployed and the “attacks” they caught. All of that without direction, only following default scripts and making poor or no use of data. Let me tell you something: you don’t need to deploy a honeypot to know the internet is full of scanners. It’s common sense. Good planning includes narrative and placement. It’s what leads to good, insightful, manual interactions with decoys. Then, a plan must consider what to do with the data collected, otherwise you’re just a hobbyist wasting money.

The deception field is more accessible than it has ever been. Get the vocabulary right, pick one project, and start.

Appendix: Glossary

In classic cybersecurity literature, terms like honeypot, decoy, and trap get used interchangeably. The glossary below draws from three primary references — Provos & Holz (Virtual Honeypots, 2007) for honeypot classification by interaction level and purpose, Sanders (Intrusion Detection Honeypots, 2020) for the production/detection use case, and MITRE Engage for the strategic engagement model — to keep the strategic intent, the deceptive artifacts, and the underlying infrastructure cleanly separated.

MITRE Engage is Too Much

MITRE Engage’s foundational insight is correct and should anchor any AE program: deception is a process, not a technology stack. Deploying a honeypot without a defined goal is hobbism; deploying one with a clearly articulated goal is engineering. Engage gets a handful of things uniquely right and worth keeping:

• The strategic goal taxonomy (Expose, Affect, Elicit): a clean, leadership-legible way to describe what kind of program you’re investing in.
• Gating Criteria and Rules of Engagement (RoE) as first-class concepts: forcing the team to pre-commit to kill conditions before deploying anything.
• ATT&CK linkage: mapping engagement activities to adversary behaviors so deception design stays grounded in real threats.
• Narrative Objective as a planning artifact: requiring the defender to articulate the story the environment tells before building any of it.
• After-Action Review (AAR) as the primary measurement construct: acknowledging that qualitative, per-scenario retrospectives are more useful than invented metrics.

All of that can be absorbed in a day. The difficulty starts when a team tries to execute Engage as prescribed:

• The 10-step operational process is adapted from Barton Whaley’s work on military deception and assumes a campaign cadence of weeks to months per engagement. Detection honeypots don’t run this way. They’re deployed once and alert over months or years.
• The 8-role team structure (Team Lead, Operational Coordinator, Threat Analyst, System Administrator, Operational User, Red Teamer, Blue Teamer, Reverse Engineer) presumes a dedicated AE team with bandwidth and headcount most corporate SOCs don’t have and don’t need.
• The Mission Essential Task List construct is imported from US military training-qualification practice. It’s useful when building a team from zero with no existing competencies. It’s overhead when those competencies already exist in Detection Engineering.
• Per-operation artifacts (Persona Creation, Storyboarding, Threat Model, Engagement Environment, Gating Criteria, Operational Objective) fragment a single coherent planning exercise into six separate documents, each with its own template.

Each individual element has a defensible purpose. The problem is aggregate weight, where an organization that tries to implement everything Engage prescribes spends more time on ceremony than on catching attackers. The existence of the Engage Starter Kit (designed to lower the barrier to entry for defenders new to the framework) at minimum acknowledges that the full process isn’t where most teams start.

A Lighter Approach

My first attempt at this program was tightly aligned with MITRE Engage, but it ended up hard to establish, explain, and maintain. While trying to make it work, I dug into adjacent literature looking for something more operational. Three sources reshaped my model:

• Virtual Honeypots (Provos & Holz, 2007) provides the underlying typology Engage alludes to but never names — low-interaction vs. high-interaction, detection vs. research, production vs. research honeypots. This is the vocabulary that maps cleanly onto the Detection-team vs. CTI-team split when you have to figure out who owns what. It also shows some clearly well-planned and successful operations without any bureaucratic overhead.
• Intrusion Detection Honeypots (Sanders, 2020) introduces the See-Think-Do positioning model, Whaley’s goals of hiding and showing traps, and the decoy properties — discoverable, interactive, and monitored. It compresses three of Engage’s planning steps (how the adversary reacts, what they perceive, through which channels) into a simple form a detection engineer can fill out in ten minutes per decoy.
• Thinkst Canary’s operational philosophy of “just 2 minutes of setup; nearly 0 false positives, no ongoing overhead” is a commercial validation of minimalism. A platform designed around the opposite of Engage’s ceremonial weight is what has actually succeeded in the market.

What emerged was a synthesis. Keep the vocabulary and planning artifacts Engage does uniquely well (gating criteria, RoE, ATT&CK mapping, AAR). Replace Engage’s heavier planning steps with Sanders’ lighter See-Think-Do model. Inherit Provos/Holz’s detection-vs-research distinction as the program-design axis that determines who owns what. Steal Thinkst’s bias toward operational minimalism throughout. The result is a framework that fits in a single document, that a detection engineer can learn in an afternoon, and that a staff-level lead can approve without board-level ceremony — while still producing the strategic clarity, safety, and goal-alignment a deception program needs.

The Framework

The framework is organized as a continuous improvement loop based on the Plan-Do-Check-Act (PDCA) model. PDCA is the right frame for an AE program because the program itself is a stable process that improves over time, not a series of one-off adversarial decisions.

The four phases:

1. Phase 1: Plan. Establish the program. Organize the tooling, write the Program Charter, secure approvals. The output is one reviewed, signed Charter document.
2. Phase 2: Operationalize. Translate the Charter into concrete deception. Define engagement scenarios, deploy assets bound to scenarios, integrate with the SIEM and SOAR (for Detection) or with the analyst data pipeline (for Intelligence). The output is a catalog of active scenarios and deployed assets.
3. Phase 3: Monitor. Collect metrics. Review alerts. Track scenario effectiveness. The output is a quarterly report and an actionable list of things that need to change.
4. Phase 4: Improve. Act on what Phase 3 surfaced. Retire stale scenarios, tune detections, create new scenarios based on CTI, review the Charter annually. Feed the results back into Phase 1.

flowchart LR
    P1["Phase 1<br/>Plan"] --> P2["Phase 2<br/>Operationalize"]
    P2 --> P3["Phase 3<br/>Monitor"]
    P3 --> P4["Phase 4<br/>Improve"]
    P4 -->|"annual"| P1

Phases aren’t strictly sequential after the first pass. Phase 1 is executed once to establish the program and revisited annually. Phase 2 runs continuously as new scenarios are defined, new assets deployed, stale ones retired. Phase 3 runs continuously with a quarterly formal cadence. Phase 4 runs quarterly (operational improvements) and annually (Charter review). The loop closes between Phase 4 and Phase 1: every annual Charter review is a fresh Phase 1 pass informed by twelve months of Phase 3 data.

No dedicated headcount is required. Responsibilities map onto five existing SOC functions:

• Program Owner: Detection or Intelligence lead engineer or technical manager. Accountable for the whole program, signs the Charter, resolves cross-scenario conflicts, approves new scenarios.
• Scenario Designer: Detection or Intelligence engineer. Designs and documents scenarios, contributes to AAR, proposes new scenarios.
• Deployer: Detection engineer, with Cloud/Infra support where platform access is needed. Deploys assets bound to scenarios, applies See-Think-Do positioning per decoy, maintains assets, rotates ephemeral tokens.
• Sensor Code Maintainer: Detection engineer named in the homemade-sensor registry. Owns the source, dependencies, deployment pipeline, and weekly health review for the deception sensors not covered by vendor tooling.
• Alert Triage: SOC analysts. Receives and handles deception alerts in the standard SOC queue, flags FP patterns, suggests tuning.

These are functions, not roles. One person can hold multiple; multiple people can share each. The program doesn’t add headcount.

Sponsorship and the Intelligence Question

AE for Detection purposes is usually simple because decoys are pretty much like any other detection tool. It’s a matter of aligning with management and going. Internal deception assets log attacker interactions the same way any SIEM logs internal activity, and corporate onboarding documentation already establishes that the environment is monitored — no new lawful basis, notice, or data-processing agreement is required.

Expanding to Intelligence / research honeypots (public, internet-exposed, owned by the CTI team, consumed as bulk telemetry rather than as alerts) is a fundamentally different activity, not an incremental adjustment. A second program with its own charter, its own approvals, and its own risk profile. Specifically:

• Attracting attention. A recognizable research honeypot signals that the company is watching. Some adversaries respond with greater caution; others respond with more sophisticated probing of your real surface. The expected value of the intelligence gained must exceed the cost of this increased attention.
• Attribution leakage. WHOIS, TLS certificate details, JA fingerprints, cloud account metadata, and billing patterns can link a “research” honeypot back to the parent organization. Operating fully unattributed is non-trivial engineering.
• Weaponization liability. Research honeypots have been used as launchpads against third parties. Without careful containment, this creates direct legal exposure.
• Third-party data capture. Research honeypots routinely capture data attackers test against them — stolen credentials, PII from prior breaches, sensitive documents. This creates data-minimization obligations and incident-notification questions.
• Duplicative intelligence. For most TTPs, commercial CTI feeds, public research-honeypot networks, sector ISACs, and national CERTs already cover what a homegrown research honeypot would collect. Homegrown research adds value only in specific, named gaps.

The pros and cons must be closely analyzed, and the person ultimately responsible for Information Security in the organization (usually a CISO or CTO) should be the formal sponsor. New functions become mandatory: a Program Sponsor at CISO/CTO level, a Legal and Privacy lead (your DPO or privacy counsel), one or more dedicated CTI analysts, data engineering support for the telemetry pipeline, and access to malware analysis / reverse engineering capability. A dedicated Legal and Regulatory section becomes non-negotiable in the Charter. The attacker data captured by an internet-exposed honeypot doesn’t originate from someone covered by the company’s onboarding monitoring disclosure, so the lawful basis must be independently established.

The pro side must also be stated as a specific hypothesis with a measurable outcome before the program is expanded. “We might learn something interesting” is not sufficient justification. “We expect to collect N samples of malware family X per quarter that target our specific customer authentication flow” is. If any of {specific intelligence gap, dedicated CTI capacity, executive sponsor, Legal/Privacy/Comms alignment, risk-appetite fit} is absent, the answer is to consume external research telemetry (GreyNoise, Shadowserver, SANS ISC, commercial CTI, sector ISACs) and revisit the question next year.

Closing Thoughts

The core idea behind this framework is that an AE program should fit inside Detection Engineering’s existing rhythm, not impose a parallel one. Engage’s vocabulary is too good to ignore but its ceremony is too heavy to copy. The synthesis (Engage’s planning concepts, Sanders’ positioning model, Provos/Holz’s typology, Thinkst’s minimalism) lets a small team run a serious program with one Charter, one scenario template, three metrics, and a PDCA loop.

If you’re standing one up, start with the Charter, define one scenario, deploy one asset bound to that scenario, and verify the alert path end-to-end with a synthetic trigger. Everything else is iteration. The framework rewards starting small and improving on evidence, exactly what Phase 3 and Phase 4 are designed to support.

Deploying any decoy is just a small part of AE. Before, we need to plan; after, we need to properly use the collected data. Focusing solely on the technicalities is wrong and often a misuse of resources. Real engineering goes way beyond that.

Appendix A: Program Charter Example

Appendix B: Scenario Document Example

The Book

Intrusion Detection Honeypots (IDH) is a short book with fewer than 300 pages, but not a shallow one. Sanders writes in a direct, clear style with no dead weight; every paragraph carries lots of lessons.

He opens with theory and motivation, defining IDH by contrast with Intrusion Detection Systems (IDS): signature-based network monitors known for their noise. IDHs, on the other hand, sit quietly on the network and aren’t meant to be probed. When they are, they yield high-fidelity alerts. It’s essentially the honeypot definition from Provos & Holz, extended to cover honeytokens.

From there, Sanders walks through the lineage of the field. He anchors it to two seminal books — Honeypots (Spitzner, 2002) and Virtual Honeypots (Provos & Holz, 2007) — and explains the origin of terms like production and research honeypots and canary honeypots. He goes further back, too: the Trojan Horse of Greek mythology and, more recently, Cliff Stoll’s Cuckoo’s Egg (1989).

As the title suggests, the focus is on detection honeypots (decoys placed inside the company perimeter that raise alarms when triggered), not research honeypots feeding cyber intelligence teams. The concepts still translate to the research side, but that’s not the book’s lens.

Two conceptual scaffolds carry the rest of the book:

• Barton Whaley’s deception framework: two goals (hiding and showing) and three forms for each — hiding: masking, repackaging, dazzling; showing: mimicking, inventing, decoying.
• See-Think-Do: Sanders’s own framework for designing deception: think like the attacker, then ask what they’ll see, think, and do when they hit your trap. Dead simple, no bureaucracy.

Sanders rounds out the theory with the three properties every decoy must have to be useful: discoverable, interactive, and monitored. Together, these concepts (goals, framework, and properties) turn honeypot planning from instinct into strategy.

The rest of the book is use cases. For each one, the author walks through how See-Think-Do and the three properties apply.

Impressions

What I appreciated most is that Sanders goes end to end: from the history of deception, through how to build the traps, to how to wire them into an existing SOC pipeline. The takeaway that stuck with me, and the one I’d put on a sticker, is that a honeypot’s value doesn’t lie in the deployed trap, but in how it’s set up and what your SOC does with the data it produces. True for detection and research honeypots alike.

The kill-chain framing isn’t new, but Sanders states it cleanly: prevention first, detection as fallback. You don’t lose the game when attackers get in. You lose it when they reach their goal. Catching them before that is already a huge win.

The honeytoken chapters are what I came for, and they delivered. Sanders shows how to deploy them, how to interpret their signals, and what value they bring. He also makes a point I’d love every SOC engineer to read: if you don’t carefully consider placement and how alerts will be interpreted by triage and investigation teams, a high-fidelity source can flip into alert fatigue and SOC rage real fast.

To flag one drawback (real nit): the use cases are useful and insightful, but reading them back-to-back gets tiring. I’d have preferred each one split into “the idea” and “the implementation”. Let readers skim the idea and get into the how only when they’re actually about to build it. Minor complaint, but it would make the book more engaging.

Overall, it was one of those books I appreciated like wine. I took so many notes that the first part is almost entirely highlighted in yellow. Plenty to put into practice.

Closing Thoughts

Fast forward: I liked it. A lot. If you’re running honeypots (or pretending to), read it. It’s that simple.

Sanders picks up where Provos & Holz left off almost two decades ago and bridges that foundational work into the present. Honeytokens are the headline addition: a cheap, high-signal class of trap that fits neatly inside modern SaaS, cloud, and identity stacks. The framing (Whaley, See-Think-Do, discoverable/interactive/monitored) gives you a vocabulary to plan deception deliberately instead of by gut feel — and that terminology will outlive any specific tool you build on top of it.

One caveat worth flagging is that deception tooling ages fast. A lot of open-source honeypot projects from earlier generations are abandoned, half-maintained, or quietly forked. Read Sanders for the concepts and strategy and budget time to evaluate whether the specific tools he points to are still alive before you build on them.

Sanders gives you the mindset to turn almost any piece of infrastructure into a tripwire. Kevin McCallister-in-Home Alone 😱 energy. Your imagination is the limit. Just remember that keeping track of traps is as important as setting them. The deception program you don’t maintain is the one that bites your SOC later. 👊

The Entity Model

SOAR platforms aren’t imperative scripting environments. They’re closer to data pipelines with a standard intermediate representation. I’m working with Chronicle SOAR specifically, but the pattern generalizes. In Chronicle’s case, that representation is the entity: a typed, deduplicated object the platform tracks across cases and over time. URLs, IP addresses, hashes, hostnames, users; all IOC types fit here.

Once data is in entity form, every action knows how to operate on it. VirusTotal_v3 - Enrich URL doesn’t need a loop; point it at “All URL entities” and it fans out automatically, in parallel. Same for Enrich IP, Enrich Hash, the WHOIS power-up, and most marketplace integrations. Conditions can evaluate [Entity.IsSuspicious] across all entities in scope with the contains operator. No iteration or JSON path gymnastics.

The implication is that the value of the platform is in transforming raw data into entities, then operating on entities declaratively. The minute you start parsing JSON arrays of IOC-shaped data inside playbook conditions, you’re fighting the tool.

The Five-Stage Pattern

I sketched out the structure I should have started with, like a framework for this kind of playbook. Five stages, in order:

flowchart LR
    A["<b>1. Init</b><br/>triggers, baseline tags,<br/>trigger source"] --> B["<b>2. Collect</b><br/>fetch email,<br/>get case details,<br/>get similar cases"]
    B --> C["<b>3. Transform</b><br/>extract IOCs,<br/>create entities<br/>(URL, IP, HOSTNAME, FILEHASH)"]
    C --> D["<b>4. Operate</b><br/>enrich entities,<br/>mark suspicious,<br/>evaluate conditions on scope"]
    D --> E["<b>5. Conclude</b><br/>set priority, tag,<br/>change stage,<br/>assign to T2"]

Each stage has a clear job and a clear handoff:

1. Init. Triggers, baseline tags, metrics, custom fields. Cheap boilerplate.
2. Collect. Fetch all data you’ll need downstream: email message, case details, run queries in other tools like EDR, get similar cases, user context. Run in parallel where you can, and don’t make any decisions yet, just gather.
3. Transform. This is the conceptual pivot. Process raw data, extracting IOCs, email attachments, and paths, separating interesting pieces of data. Then, process this data, calculating hashes, saving files to case, etc. You’ll want to identify the Entities. Finally, convert them into entities using Create Entity actions with the right delimiter, scoped to the right type. After this stage, the raw data is mostly irrelevant; the entity graph is what matters.
4. Operate. Run enrichment functions over entity scopes. Enrich URL on All URL, Enrich Hash on All FILEHASH. Mark entities as suspicious via configured thresholds, and use a single condition on [Entity.IsSuspicious] to consolidate the verdict. No for-loops or nested ifs. The platform handles parallelism for free.
5. Conclude. Based on information processed, set priority, drop tags, change stage, assign to T2. The decisions here read from the work done in stage 4, so they should be small and obvious.

When the playbook is shaped this way, stage 4 reads less like code and more like a sentence: “for every URL we found, check VirusTotal; for every IP, check VirusTotal; if anything came back malicious, escalate.” That’s the goal: readability over cleverness.

Reusable Blocks

The next refinement, and one I’m still working into my own playbooks: stages 1, 4, and 5 shouldn’t be simple stacked actions. They should be lists of block invocations.

Blocks in Chronicle SOAR are reusable sub-playbooks, basically functions. You define VT_Enrich_All_Entities once, with the enrichment actions, thresholds, and post-conditions, and call it from any playbook that needs it. Same for KB4_Simulation_Check, Sender_Reputation, Similar_Cases_Sweep, and Attachment_Hash_And_Enrich.

If you look at the official Google playbooks shipped via the Marketplace or in the adoption guides, this is exactly the shape. A T1 playbook isn’t forty stacked actions; it’s a trigger, an init step, and four or five block calls, each owning a coherent piece of logic. The block library does the work, and the playbook just orchestrates.

A reasonable starter library for a phishing triage T1:

• Identity_Enrichment: pull user context (department, manager, recent reports) for the reporter.
• Threat_Data_Enrichment: VT, WHOIS, URLScan over all entities in scope. Sets is_suspicious.
• Attachment_Handling: guarded by a “has attachment” condition. Hashes, creates FILEHASH, enriches.
• Similar_Cases_Sweep: find prior cases with the same message ID, sender, or subject hash. Critical for catching campaigns.
• Verdict_And_Tag: read the consolidated entity state, drop the right tags, set priority.

The orchestration playbook becomes maybe ten boxes total. The complexity lives in the blocks, where it can be tested and versioned independently.

This pays off in two places:

• Maintenance. A change to enrichment logic (new threshold, new engine allowlist) happens in one block, not five playbooks.
• Composition. The next playbook becomes a thirty-minute job of arranging existing blocks, not a week of copying actions.

A Familiar Shape

What I described is the same shape as well-known patterns in software engineering:

• Pipes and filters. Stages transform a stream, each one reading the previous stage’s output shape. Adding a new transformation doesn’t require rewriting earlier stages.
• ETL. Extract → Transform → Load. My stages 2 → 3 → 4 are literally ETL with security-flavored names.
• Functional operations over collections. map and filter instead of explicit loops. Chronicle’s entity-scoped actions are map over a typed collection, exactly the same idea dressed up in a flowchart UI.
• OODA / NIST IR lifecycle. Observe → Orient → Decide → Act. Triage playbooks are OODA with checkboxes.

All of these converge on the same shape because the alternative (raw data threaded through the whole flow, imperative branches everywhere) doesn’t compose. Add a new IOC type or a new enrichment source and you rewrite half the playbook. The pipeline shape absorbs new types for free.

Where AI Fits

Google and other vendors have packaged what they call “agentic triage”: an LLM in the loop, calling enrichment functions, producing a verdict and a confidence score. One step that replaces most of the pipeline above. I’ve tested it. It works.

The instinct to drop it into the front of the SOC funnel is wrong, and the reason isn’t cost — though cost matters too. Three problems, roughly in order of how much they should worry you.

1. The real risks are prompt injection and non-determinism. Phishing emails are, by construction, untrusted input written by someone trying to deceive the recipient. Feeding that body to an LLM with tool access (mark malicious, escalate, close as benign) is handing the adversary a function-call interface. Mitigations exist (system-prompt hardening, content sanitization, tool-output validation) but none of them are airtight. The second-order problem is that two runs of the same alert can produce different verdicts, which makes audit, regression testing, and analyst trust harder than they need to be.
2. Cost scales the wrong way for T1. Deterministic pipelines have flat per-alert cost: a VT lookup is a VT lookup. LLM pipelines scale with alert volume and with prompt size. T1 is exactly where alert volume is highest and per-alert value is lowest. Putting the most expensive component at the widest part of the funnel is a budget killer.
3. AI’s edge is judgment, not lookups. An LLM is not a faster VT client. Where it earns its tokens is synthesis: reading the email body, the reporter’s recent activity, similar past cases, and producing a short narrative a T2 can audit in thirty seconds. That capability is wasted on alerts an [Entity.IsSuspicious] condition already resolved.

Two concrete placements fall out:

• Authoring, not runtime. Use an LLM to draft playbooks, suggest blocks, sketch the five-stage skeleton. Cost is paid once, at design time, and the model is out of the loop before the first alert hits production. The number of alerts processed never touches the bill.
• Downstream of cheap triage, scoped narrowly. After deterministic stages have ruled out the obvious, run the agentic step on what survives — medium and high priority only, and even then scoped to specific detection rules or alert categories where narrative synthesis actually helps. Fewer alerts, higher per-alert value, both budget and judgment line up.

What you should not do is let the LLM be the only thing between an alert and a decision. Beyond cost and injection, you’ve built a single point of failure with no fallback: if the provider degrades, if a model update silently regresses, or if the agent’s tool loop gets stuck, every alert backs up behind it.

Closing Thoughts

The lesson, which I had to learn the slow way: when you pick up a new tool, the cost isn’t learning the syntax, it’s unlearning the habits you brought with you from a tool that worked differently. I built playbooks the way I build Python scripts because that’s the shape my brain reaches for. The platform was giving me an entity model, parallel scoping, declarative conditions, and reusable blocks, and I was ignoring all of it to write nested for-loops in a flowchart UI.

Adapt the work to the tool you have, not the other way around. Recalling Maslow’s Hammer, “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.” If the playbook is fighting the platform, the platform is going to win, usually six months later when you need to extend it and find you can’t. ✌️

References

• Create a fallback playbook. Google’s official guide to T1 design principles. Short and direct; the “non-destructive, information-gathering” framing is the same as my stage 4.
• Create an enrichment playbook block. The canonical reference for blocks, with a good walkthrough of how to design one that handles multiple entity types.
• Adoption Guide: Designing and Building Your First Playbooks. Google Cloud Community walkthrough of the exact pattern with concrete actions and block names.
• Adoption Guide: Building Effective Playbooks. Covers entity scoping and the [Entity.IsSuspicious] condition in depth.
• Foresite: Advanced Tips for Google SecOps SOAR Automation. Has the “One Playbook to Rule Them All” tip, which extends this pattern further toward a universal master playbook.

The Migration

After two days of back-and-forth with Claude, I managed to move everything to the new system. hledger-powered, Git-versioned. Precise. Unified. By the end of the last session, after importing historical transactions from my personal spreadsheets, banks and credit cards (OFX), and brokers (CSV), everything was closing beautifully. Balances matched between my personal records and the apps. Scripts to help me import balances and bills on a monthly basis were alive and kicking — maybe later I’ll go for OpenFinance connectors, not now. By the end of that fateful session, Claude and I were re-sorting hledger journals. Since it had been extremely precise, I went YOLO and just told it what to do, with full permissions.

When It All Went Wrong

I asked it to re-sort year by year, since I’m keeping one main.journal file and one journal per year for traceability. The context window was over 50% (I was about to finish that session and didn’t want to clear context), and Claude executed a broken sort command, redirecting the output to my 2026.journal file — the one with the most transactions.¹ The result? It wiped all the transactions we’d spent the last two days importing and reconciling.

My heart stopped. I was listening to ‘Estranged’ by Guns N’ Roses on Spotify at that moment. The title fit the situation better than I’d have liked. The only thing I could do was pause the song and try to breathe so my heart would start beating again. Frightening. Claude was supportive: I hadn’t committed that file at all, so there was no Git history, but it tried to find macOS backups, unsuccessfully. After almost 5 minutes of trying, it managed to find the file’s contents in its own logs on my machine and restored everything. What a relief!

Takeaways

1. AI/LLMs don’t need to be kept in the loop forever. Sometimes they’re just the tool that helps you build something. You don’t need to develop products that will necessarily use AI; the result can be the same. In my case, moving to fully plain-text accounting made more sense than adding MCPs to my workflow so the LLM could interact with my spreadsheets. And in my experience, plain-text accounting consumes fewer tokens than feeding spreadsheet files to LLMs every time — no headers, no schema overhead, just lines of plain content.

2. Smart agents still make mistakes. LLMs are getting smarter every day, and agents like Claude Code are awesome. But just like a human, they’re prone to error. It’s not enough to control context and provide good prompts. Your project plan should have checkpoints that let you commit everything periodically (Git fits super well with LLMs), especially before important changes to critical files. Protecting the work you’ve done is more important than doing new work, particularly because the new work often depends on the previous one.

3. The AI’s recovery skills surprised me. Just seconds after noticing it had wiped the file, Claude started laying out strategies to recover it. Some would have taken me hours to figure out, especially the definitive one, recovering the original data from its own logs. Brilliant.

4. AI session logs are sensitive data. They contain everything the agent saw: file contents, conversation history, secrets you didn’t realize you exposed. Never publish them. Be especially careful with CLAUDE_CONFIG_DIR (default ~/.claude/): the projects/ subdirectory holds session transcripts that should never leave your machine, and settings.json often contains hooks or env vars you don’t want public either. Other contents in the same directory (skills, custom agents, slash commands) can be worth sharing after a careful review.

For a few hours now, I’ve been playing around with hledger commands to visualize my finances in different ways. Instead of multiple spreadsheets, I now have a single source of truth, and it’s great. Next steps include vibe-coding data analysis tools to visualize my data and get better insights. Maybe I’ll write about it in the future.

AI as DIY for Software

I’m impressed by how nice it is to use AI (LLMs, agents) now. I put them in the roles of assistants and peers that help me understand a technology and even reason about what I just learned, speeding up the learning process. They also help me apply what I learned in projects that would otherwise take months and many back-and-forths. Without Claude in the loop, I’d have made accounting decisions on broken rationale and ended up redoing the transactions later. Instead, it flagged the future pain points, like a good chess player predicting next moves, and suggested cleaner approaches, letting me build a better solution from the start.

AI is changing a lot of things. To pick one from my niche: software development is getting quietly democratized. Teams and individuals no longer need experts to develop and deploy non-critical stuff. They can do it themselves with LLM help. Think of an accounting team at a big company that needs a better system to track financial transactions, or someone like me who needs the same for their personal life. Instead of buying expensive software or waiting forever for their ticket to be picked up by the software development team, they just use AI to do it. They build the plan with AI, AI develops, they validate.

It’s like DIY projects, but instead of plumbing, AI lets you handle software. If it’s not critical, it’s super OK to do it yourself. Think of changing your kitchen faucet. It won’t kill you to try. But plumbing your entire house and connecting it to the city’s water main should be left to a professional. Same domain, different use cases, different approaches.

Voices Saying the Same Thing

This matches what Steve Yegge said a few months ago during his appearance on The Pragmatic Engineer podcast: although it’s strange and sometimes scary to realize AI can replace part of my work, I feel pumped to use it because it lets me work differently. Not just delivering faster, but also delivering things I’d otherwise judge infeasible given how little time I have to learn new skills and put them into practice.

Fabio Akita made a similar point on Flow Podcast last month: we should treat this AI transformation as the new typewriter — a tool that lets us work faster, in a different way. It’s super exciting to think this change will free some of us from tedious, boring work and let us operate at another, more interesting and motivating layer.

Wrap-up

AI is genuinely strange, and it can be scary at times. But on the whole, it’s been great. The specialists I trust are landing on the same conclusion, and my own two days proved it: the agent wiped my journal, then recovered it. Net positive. I have my concerns, but I’m pumped anyway. The sensible move is to adopt it and move forward. Just commit before you YOLO. ✌️

The “Why” and the Trade-offs

I define a Detection Engine as any platform capable of receiving raw telemetry and outputting alerts. In this sense, SIEMs and EDRs are two common tools that meet this criterion. In complex environments, it’s common to have multiple engines running in parallel, each monitoring a specific slice of the stack: endpoints, cloud, SaaS, etc. Each new engine added greatly impacts Threat Detection due to its idiosyncrasies: different query languages and, most importantly, different logic.

Although many refer to the logic alone as “the rule,” I don’t think that’s accurate. As Google’s dictionary states:

A rule is a principle that operates within a particular sphere of knowledge, describing or prescribing what is possible or allowable.

In this sense, logic is just one part of a rule—and I acknowledge it’s the most important part. But a rule also requires context: a name, description, classification, and deployment parameters.

Since I joined Detection Engineering, I’ve noticed many parallels between a detection rule and regular software, and like any software, a rule needs a runbook for handling the alerts it generates. That runbook should be part of the rule itself. This encapsulates the rule in an envelope that handles different kinds of logic across different engines.

Doing so, engineers work against a normalized structure (the rule) despite having multiple platforms underneath. It doesn’t remove the complexity of learning each engine’s query language, but it greatly helps the team scale.

That envelope can be implemented in multiple ways, but the most effective rely on declarative languages like YAML and TOML. They’re easy to read and maintain. As plaintext, they’re also a natural fit for Git, providing solid version control. When you layer on a collaboration platform like GitHub or GitLab, you get peer review and CI/CD automation on top. These are the biggest selling points for leadership when pitching DaC. What’s less discussed are the trade-offs.

This workflow generates friction. Before, a Detection Engineer would log into the engine, write their logic, save, and move on. Now they must use the engine as a starting point for the use case they’re building, and once they have an MVP:

1. Create a new branch in DaC
2. Drop the MVP
3. Add context to the logic: fill the envelope
4. Commit
5. Run additional tests, adjust, commit again, request peer review
6. Once everything looks good, merge to the mainline and monitor the pipeline output
7. If all is fine, the job is done. If not, fix the automation scripts and redeploy.

This new environment requires SRE skills from Detection Engineers, and not everyone is comfortable with Git workflows. Some engineers feel awkward reviewing a colleague’s work, which often leads to rubber-stamp approvals. On the flip side, some people start nitpicking small typos and delivering feedback without empathy, which creates friction of a different kind.

This extra friction must be justified. Based on the gains I described above, not all teams will benefit equally. The rule of thumb is: the larger the team and the more engines it operates, the more the friction is justified.

Does that mean small teams shouldn’t go for DaC? It depends. As an engineer, even working on a small team with a single detection engine, I’d still want to implement it just to build good skills. But that’s optional, a side project for periods of low demand, not something to enforce on all engineers from day one. To be direct: smaller teams in low-complexity environments can live without DaC and still be effective.

Large-scale teams running complex environments will greatly benefit from DaC, and the friction is more than justified when the project is properly designed. DaC increases the team’s maturity and consistency, helping it scale more reliably.

The Core Use Cases

To get real value from DaC, the team must invest time architecting the rule format—the envelope. This is critical and non-negotiable. All features will be built on top of it, so any future change can be very disruptive. Looking around, you’ll find varied structures. That’s because each team has its own needs and context. There’s no one-size-fits-all. That said, I believe a good rule structure should cover these four blocks:

1. Metadata: Fields like description, author, and categorization—MITRE ATT&CK mappings live here.
2. Logic: The rule logic itself, as expected by the Detection Engine, stored as a text block.
3. Deployment: Runtime parameters for that rule: on/off, run frequency, schedule, etc.
4. Guide: How to handle alerts from that rule: a high-level overview to inform IR playbooks and ease future maintenance.

I like to think like a data scientist when designing this structure. Ideally, fields are complementary with minimal or no overlap. For example, since the rule name can be the filename, I avoid a metadata.name field inside the rule. If a piece of data is already tracked automatically by another tool in the stack (like Git tracking authorship and timestamps), I don’t manually replicate it.

In YAML, my proto-rule looks like this:

metadata:
    authors:
        - "Joe Lopes"
    description: "Direct, clear, and short description"
    mitre:
        T0000: [TA0000]
    references:
        - "http://lopes.id"

logic: |
    events:
        $e.principal.ip...
    match:
        $e over 10m
    outcome:
        $ip = array_distinct(...
    condition:
        $e

deployment:
    live: true
    alerting: true
    frequency: "LIVE"

guide:
    context: |
        A more detailed description, including why the rule exists and how it fits our strategy. One or two paragraphs.
    triage: |
        - Steps to triage alerts from that rule
        - Known false positive cases
    response: |
        - High-level steps for incident response

With this structure in place, several use cases become achievable. Let’s walk through them.

DaC and AI

You can’t discuss technology these days without touching on AI, and I’m no different. LLMs are excellent at parsing structured formats like YAML, TOML, and JSON, which makes them a natural fit for DaC. With the repository cloned locally, any AI agent can work directly with the rules, providing insights, helping write scripts, or refining logic.

This approach avoids the overhead of standing up an MCP server to mediate access to the ruleset, saving tokens and money. More broadly, I find it a good reminder against over-engineering: avoid an MCP server when a local file read would work just as well, sometimes just to say “it’s AI-first.”

The only limit here is your creativity. Ask the AI to find overlapping rules, suggest missing MITRE tags, or draft investigation guides. Ask it to help you code a new feature. DaC is AI-friendly although not AI-first (yet 👀).

Conclusion

Detection-as-Code is an architectural choice. It can help Threat Detection teams mature and scale faster, but it comes with trade-offs that deserve honest evaluation before committing.

Larger teams in complex environments will see real gains. Smaller teams can benefit too, at a smaller scale. The best strategy is to start simple, plan the work before writing a single line of code, and deliver the project in phases, adding features as they’re needed, not before.

The use cases must be super clear from the beginning. You should not implement DaC just because it’s nice, but because it’ll help you scale, make audits easier, and improve your management capabilities. Remember: DaC is the way, not the end. 🍀

Motivation: The Need for a New Modus Operandi

For a long time, my workflow was functional, albeit traditional. I would write scripts or small programs, test them empirically, document them, and push them to production. But watching the rapid evolution of AI coding tools, I realized that simply bolting a chatbot onto my existing workflow wasn’t enough. I needed to rethink my entire process from the ground up to actually use these tools as autonomous agents rather than glorified search engines.

The Pre-LLM Era: Scripts, Ad-Hoc Fixes, and Stack Overflow

Before the rise of LLMs, my development process looked exactly like this:

1. Create a folder, bootstrap Git, and initialize the programming language environment.
2. Dive straight into the code, creating subfolders and importing libraries.
3. Once the base structure (CLI, directory tree) was complete, start hacking away at specific components (a top-down approach).
4. Perform ad-hoc testing until reaching a working “v0.5”.
5. Polish the code, write documentation, and ship.

In security and IT operations, we rarely require the rigid bureaucracy of enterprise software development (Agile, heavy CI/CD, exhaustive system design). We optimize for speed and utility, building parsers, automation scripts, or Proof-of-Concepts (PoCs) to put out the immediate fire in front of us.

However, looking back with a critical eye, my process was fundamentally flawed. It severely lacked automated testing, which inevitably led to technical debt. Without unit tests, my scripts were brittle. A minor refactor or an unhandled edge case in a log file could quietly break the entire tool. Upgrading netbox-scanner to v2, for instance, was painful for exactly this reason.

I came to realize that adopting core engineering best practices, like test coverage, would make my deliverables significantly safer and more reliable. Under my old manual process, achieving this meant heavy reliance on Google, Stack Overflow, and official documentation to unblock myself. It worked, but it was incredibly tedious.

Method + LLM: The Friction and Fatigue of Early Adoption

I started riding the LLM wave early, diving into AI-assisted development around mid-2025 while learning Rust programming language and building Cordyceps, an educational Proof-of-Concept (PoC) ransomware.

At that stage, the AI tooling landscape was evolving at breakneck speed, but my workflow remained stagnant. Instead of taking the time to learn how to orchestrate agentic features, I simply took my flawed pre-LLM workflow and slapped AI on top of it.

Even after progressing from copying code out of a browser tab to using an in-IDE AI agent that automatically updated files, the fundamental problem persisted. Lacking a structured method for providing context, I found myself repeatedly explaining the project’s rules with every new chat session. This constant repetition, combined with the AI’s natural context degradation in longer threads, made me seriously question the tool’s long-term utility.

While the AI provided direct, tailored answers, this ad-hoc approach introduced massive friction:

• Context decay: Chats were painfully short-lived. I noticed that after a certain point, the AI’s answers would degrade, becoming biased or completely ignoring foundational instructions. In ML literature, this is a documented phenomenon known as “attention decay” or “context window saturation.” As a conversation grows, the LLM physically struggles to weigh your original system instructions against the noisy tokens of the ongoing chat.
• Psychological fatigue and “prompt rage”: Using AI in an unstructured manner took a surprisingly heavy psychological toll. Having to endlessly rebuild context left me genuinely frustrated with the AI for “forgetting” constraints; at times, it felt like arguing with a toddler. I had essentially shifted my stress from debugging broken code to debugging a stubborn LLM.
• Workflow silos: Initially, manually moving files back and forth from a browser was exhausting. But even after adopting a local agent plugin, the struggle remained because my process was still siloed.

I was still the developer doing all the heavy lifting; the AI was just a smarter, occasionally frustrating textbook. It was like copy-pasting from Stack Overflow, just generated on the fly. The solution wasn’t to “prompt harder.” The solution was to step back, review my entire workflow, and integrate AI much earlier in the software lifecycle, using its capabilities in a structured, deterministic way.

The New Way with LLM: Designing Foxglove to Learn the Paradigm

Entering 2026, I realized the only way out of this frustrating cycle was to stop forcing the AI into my old habits. I decided to actually study the new way of working. I researched agentic workflows, structured AI tooling, and LLM orchestration.

Only after internalizing this new paradigm did I design the Foxglove project. Foxglove wasn’t just another script; it was deliberately conceived as a sandbox to practice an AI-native methodology.

This was a profound mindset shift. I finally understood that modern AI tools are no longer intuitive “chatbots”: they are complex orchestrators. Studying a tool’s architecture and best practices before deploying it is exactly how a senior engineer approaches any new technology. The tool and the paradigm must dictate the workflow, not the other way around.

I installed Claude Code directly on my machine, integrated the necessary editor plugins, and completely overhauled my approach:

Ask me clarifying questions about edge cases, UI/UX, and technical tradeoffs before we start building.

Takeaways: The Paradigm Shift

Looking back, my 2025 endeavor was basically my legacy workflow with an AI bot duct-taped to it. The 2026 approach is a true AI-first workflow.

1. Change the process, not just the tool. You cannot simply bolt AI onto your existing stack. You have to step back, study agentic methodologies, and fundamentally restructure how you build software.
2. From code to specifications. Development is now less about writing syntax and more about defining rigid prerequisites and system specifications. However, foundational engineering skills are still what separate production-grade tools from hobbyist scripts. You have to know what to ask for, and you must know how to review the output.
3. Unprecedented velocity. Historically, a project like Foxglove would have taken me two to three weeks working 4 hours a day. Using this agentic workflow, the project was delivered in about 15 commits, taking less than 6 hours total. The efficiency gains are staggering.
4. Clarity is king. Defining clear goals and mandating TDD was the ultimate differentiator. Had I blindly accepted the code Claude initially offered without strict constraints, the project would have collapsed into an unmaintainable mess.

By taking the time to learn the tools and paradigms, I stopped fighting the AI and started orchestrating it. Other capabilities, like Claude Skills, are next on my list. 🤖 🚀

The Book

Virtual Honeypots showcases the authors’ vast expertise in deception technologies. However, to truly appreciate its greatness, we must put ourselves in the shoes of a late-90s or early-2000s IT practitioner. At that time, VMware and hardware virtualization were just starting to gain traction, Linux systems were undergoing massive consolidation, and many of our common security tools and taxonomies had yet to be invented.

Provos and Thorsten Holz demonstrate remarkable technical depth throughout the pages, teaching network protocols, OS configuration (Linux and Windows), and even systems programming, including Python 🐍. Since they personally developed core tools presented in the text, like Honeyd, their authority on the subject is undeniable.

The language is accessible, and the text is packed with practical examples, ranging from raw command outputs to complete configuration files. In several instances, the authors hold your hand, explaining what a program does, how to compile and configure it, its execution flags, and its practical use cases.

Although the chapters are not explicitly grouped, I mentally divide the book into three sections: Part I covers theory and motivation (Chapters 1-3); Part II focuses on Honeyd and other honeypots (Chapters 4-8); and Part III delves into practical use cases (Chapters 9-12).

Impressions

I gained numerous insights from this book, primarily from Part I. The discussions on honeypot types and architectures for safe deployment are priceless. Furthermore, the taxonomy the authors employ to categorize honeypots is well-thought-out and highly effective at teaching readers how to design robust deception topologies. As expected, foundational theory has longevity, and Part I has absolutely stood the test of time.

Most of Part II is dedicated to Honeyd, and it is clear that Provos was proud of his creation. This is entirely justified, as Honeyd was a massively successful project in its day. However, since the tool has been deprecated for years, this section lacks engagement. Today, the only people likely to benefit from these chapters are developers actively writing honeypot software. Even for professionals seeking modern honeypot deployment advice, these chapters are a tough read.

Part III presents numerous use cases, covering everything from motivation and network topology to analysis and results. Unfortunately, the authors once again dedicated a large portion of these chapters to deep dives into tools that clearly haven’t aged well. Had they focused more strictly on architectural topologies and breach analysis, this section would have remained highly relevant.

This effectively summarizes my overall feeling about the book: a fantastic resource for its time, but one that is overly fixated on tooling that is now obsolete, like UML, DTK, LaBrea, Sebek, and Nepenthes. While I understand the authors’ desire to teach exactly how these tools worked and integrated under the hood, the lack of architectural abstraction makes the text less appealing to a modern audience.

Historically, a significant portion of deception software was linked to academic research, and even today, high-interaction deception technology isn’t a mainstream enterprise staple. In the two decades since this book’s release, entirely new paradigms like containerization and specialized honeypots have emerged, radically changing how we implement the scenarios the authors presented. Combining this evolution with the authors’ choice to tightly couple their lessons to specific, now-deprecated tools creates the perfect storm: a large portion of this book is, unfortunately, forgettable.

Last Words

Virtual Honeypots was undoubtedly a great book for its time, and I suspect it inspired many professionals to dive into deception engineering. Even today, the foundational first part remains excellent and proves the authors were true pioneers.

However, as explained, from Chapter 4 onward, the book starts feeling like a photo album showing scenes from a distant IT past. My recommendation depends heavily on your current goals:

• If you are deploying honeypots: Read Chapters 1-3 carefully and skim the rest, paying slight attention to the topologies in Chapters 9-12.
• If you are writing honeypot software: Chapters 1-3 remain required reading, but you should also pay close attention to Chapters 4-8 to extract architectural insights from the development of Honeyd.

For any other InfoSec practitioner simply curious about honeypots: read Chapters 1-3 and close the book. I want to emphasize that I deeply respect the quality of this work and the skill of its authors. It is genuinely impressive to recommend a 20-year-old technology book with no further revisions, but it is almost impossible to tie a book this closely to specific tooling and have it survive the ages. 🪴

The Problem with Broad Strokes

Google SecOps includes a detection rule in its public repository that serves as a prime example of this concept. The whois_recently_created_domain_access rule, available in GitHub, checks for any network connection event where the target domain was created less than 30 days ago. While this serves as a good starting point, like many publicly shared detection rules, it lacks context and refinement. Without additional scoping, it is prone to false positives.

As a threat hunting rule or an atomic/producer rule (signal generation), this logic is perfectly fine. However, adopting this approach as a detection rule lacks the necessary granularity because:

1. Context is missing: Not all NRD activity is equal. A new vendor’s SaaS API connection does not carry the same threat level as a targeted spear-phishing email.
2. Arbitrary thresholds: A single, static threshold (e.g., 10 connections) is simultaneously too low for legitimate bulk web traffic and too high for a “low-and-slow” spear-phishing attack. This guarantees an unmanageable false positive (FP) rate and significant false negatives (FN).

The central engineering principle here is: Breaking detection down by use case allows for superior granularity and precision.

Contextual Granularity: A Tiered Detection Model

Working with Google Workspace logs opened my mind to a more granular solution. The most effective strategy for NRD detections is to implement a tiered detection model where the threshold and focus are dictated by the event type and the sensitivity of the target entity (e.g., VIP status)—in other words, the use case.

This shifts the detection philosophy from simply observing newness to observing suspicious behavior in high-risk contexts, where newness is merely one of several contextual factors. 💡

The next table outlines a few use cases with ideas for implementing scoped detections. These use NRD data to add context to otherwise standard connections.

Using NRD with Google Workspace

As mentioned, I was analyzing Google Workspace events when I realized I could use NRD context to highlight suspicious email transactions. My first approach was to select all messages opened in Workspace, extract the domain from the sender address (excluding trusted domains), and trigger an alert if the domain was less than one week old.

Why one week? In my testing, I noticed that shorter timeframes missed too many threats, while extending it to a month or more created a rule too broad for general monitoring. The resulting YARA-L rule looks like this:

rule workspace_nrd_possible_phishing {
  meta:
    author = "Joe Lopes <lopes.id>"
    description = "A user opened an email from a newly registered domain (created < 7 days ago), which may indicate a phishing attempt"
    severity = "medium"
    maturity = "initial"
    mitre = "T1566:TA0001"
    reference_1 = "https://github.com/chronicle/detection-rules/blob/main/rules/community/threat_intel/whois_recently_created_domain_access.yaral"
    reference_2 = "https://support.google.com/a/answer/12384955"

  events:
    $mail.metadata.event_type = "EMAIL_TRANSACTION"
    $mail.metadata.product_event_type = "2" // message was opened
    strings.extract_domain($mail.network.email.from) = $domain
    not $domain in %trusted_domains.domain

    $whois.graph.entity.domain.name = $domain
    $whois.graph.metadata.entity_type = "DOMAIN_NAME"
    $whois.graph.metadata.vendor_name = "WHOIS"
    $whois.graph.metadata.product_name = "WHOISXMLAPI Simple Whois"
    $whois.graph.metadata.source_type = "GLOBAL_CONTEXT"
    $whois.graph.entity.domain.creation_time.seconds > 0

    // domain was created in the last 7 days: 7 * 24 * 60 * 60 = 604800 seconds
    604800 > timestamp.current_seconds() - $whois.graph.entity.domain.creation_time.seconds

  match:
    $domain over 1h

  outcome:
    $risk_score = 25
    $created_at = array_distinct(timestamp.get_date($whois.graph.entity.domain.creation_time.seconds))
    $sender = array_distinct($mail.network.email.from)
    $recipients = array_distinct($mail.network.email.to)
    $num_messages = count($mail.network.email.mail_id)
    $num_attachments = array_distinct($mail.additional.fields["num_message_attachments"])
    $dkim = array_distinct($mail.additional.fields["dkim_pass"])
    $spf = array_distinct($mail.additional.fields["spf_pass"])

  condition:
    $mail and $whois
}

One concern I had was that “one week” might not cover all scenarios. But as noted, simply increasing this window globally would lead to an unacceptable volume of alerts in production. A good solution in these cases is to create a fork of the original rule that monitors a specific set of high-value assets.

In this case, I decided to monitor a list of VIPs. Since this greatly reduces the user count to only executives or people with elevated privileges, we can safely increase the “newly registered” window to 180 days. With this change, the rule becomes a specialized detector for targeted spear-phishing attempts:

rule workspace_nrd_possible_spear_phishing {
  meta:
    author = "Joe Lopes <lopes.id>"
    description = "A VIP user opened an email from a newly registered domain (created < 180 days ago), which may indicate a spear phishing attempt"
    severity = "medium"
    maturity = "initial"
    mitre = "T1566:TA0001"
    reference_1 = "https://github.com/chronicle/detection-rules/blob/main/rules/community/threat_intel/whois_recently_created_domain_access.yaral"
    reference_2 = "https://support.google.com/a/answer/12384955"

  events:
    $mail.metadata.event_type = "EMAIL_TRANSACTION"
    $mail.metadata.product_event_type = "2" // message was opened
    strings.extract_domain($mail.network.email.from) = $domain
    not $domain in %trusted_domains.domain
    $mail.principal.user.userid in %vips.username

    $whois.graph.entity.domain.name = $domain
    $whois.graph.metadata.entity_type = "DOMAIN_NAME"
    $whois.graph.metadata.vendor_name = "WHOIS"
    $whois.graph.metadata.product_name = "WHOISXMLAPI Simple Whois"
    $whois.graph.metadata.source_type = "GLOBAL_CONTEXT"
    $whois.graph.entity.domain.creation_time.seconds > 0

    // domain was created in the last 180 days: 180 * 24 * 60 * 60 = 15552000 seconds
    15552000 > timestamp.current_seconds() - $whois.graph.entity.domain.creation_time.seconds

  match:
    $domain over 1h

  outcome:
    $risk_score = 50
    $created_at = array_distinct(timestamp.get_date($whois.graph.entity.domain.creation_time.seconds))
    $sender = array_distinct($mail.network.email.from)
    $recipients = array_distinct($mail.network.email.to)
    $num_messages = count($mail.network.email.mail_id)
    $num_attachments = array_distinct($mail.additional.fields["num_message_attachments"])
    $dkim = array_distinct($mail.additional.fields["dkim_pass"])
    $spf = array_distinct($mail.additional.fields["spf_pass"])

  condition:
    $mail and $whois
}

Conclusion

By applying this granular and contextual approach, your detection strategy moves past simply monitoring “newness” and focuses on behavior in the most critical attack vectors. This shift is essential for reducing analyst fatigue and ensuring that high-risk events (like spear-phishing against VIPs) trigger alerts immediately, regardless of organization-wide volume. Invest in granularity, and you will see a significant increase in your true positive rate. 🎯

Zola: A Fast Static Site Generator

Zola is known for its speed and ease of deployment. Written in Rust, it is fast—it builds my repository of nearly 80 posts almost instantly. After running zola serve, a version of my site was immediately available at localhost:1111 for review. Installation is just as simple: as a single ~15 MB binary, it’s just a matter of downloading the file, adding it to the system path, and voilà: job done.

My journey with Zola is partially documented on this blog. In my debut post here, I shared how to get started with the framework. But after getting a working version of the site up, I noticed theme options were limited compared to Hugo, one of the most popular static site generators (SSGs). That pushed me to create my own theme, ZOLA.386, which brought back memories of Saturday afternoons setting up MS-DOS games with my father.

As time passed, I wanted a darker aesthetic, which led to a move to Kita. This theme supported the blog for most of its life, and I credit the developers for its quality and feature set. With Kita, I hit a long-standing goal: “blog as code,” with visual elements like diagrams, equations, and callouts defined as text rather than static images.

Later, Kita added support for Open Graph images. This coincided with the rise of AI-generated imagery, so I adopted custom images to represent my posts. The response was good; several readers praised the blog’s style, and some reached out for tips on using Kita. It was a genuine success.

Quarto: Scientific and Technical Publishing

Quarto is similar to Zola in that it is a static site generator, but it offers a broader feature set. Built on Pandoc, a universal document converter, Quarto is more than just a site builder. As its documentation suggests, Quarto is designed for scientists and engineers to easily create and share technical content.

Importantly, Quarto can use Jupyter notebooks to build interactive posts, letting you publish projects directly from the notebook environment.

Quarto uses a Markdown dialect with a lot of functionality out of the box, such as footnotes, callouts, Mermaid diagrams, and LaTeX snippets. While I previously relied on the Kita theme to get these features in Zola, Quarto handles them natively. The notation is often simpler too.

Quarto also gives you JavaScript interactivity by default. With a simple setup, I can let users search, sort, or filter posts. On the SEO side, the generator applies best practices automatically, including Twitter cards and Open Graph metadata.

This range of features was the main driver behind the migration.

Migration

I used this migration as a chance to refine several things I had built over time, aiming for a more unified vision for the project. That refinement took some time¹, though it wasn’t strictly related to Quarto itself. Once I started, I found that Quarto can be both very easy and occasionally frustrating. Configuration via _quarto.yml is straightforward, but the documentation has gaps that can lead to confusion.

I first picked the Cosmo theme because the preview on Bootswatch offered both light and dark versions, a feature I wanted to keep from Kita. But I quickly realized that although Quarto implements Bootstrap themes, the integration isn’t smooth. Standard Bootstrap themes use the HTML data-bs-theme property to toggle versions, but Quarto uses a different system that overrides this, forcing the user to define two separate themes.

The upside of this hurdle was that it pushed me to dig deeper. I eventually found the Notes from a Data Witch blog and fell in love with the theme. The author shares the source code on GitHub, which let me study and adapt it into my own style: the Vigil theme.

Beyond theming, most of the work was string substitutions, regex operations, and shell scripts to migrate my post library. I then made manual adjustments to fine-tune specific posts. The next big step was recreating the Open Graph images to follow best practices; I also decided to switch to the WebP format for faster load times.

To wrap everything up, I reconfigured the CI/CD scripts on GitHub to perform automated checks and deployments.

AI Support

Unlike previous migrations, I used AI to help with this process, and I have mixed feelings about the experience. In some areas it was great: it was a big help in customizing the theme, creating the new logo, and generating code snippets. On the other hand, it sometimes gave me wrong guidance; for instance, it suggested blurring Open Graph images only to contradict itself later, wasting time and causing frustration. It also tended to “hallucinate” Quarto properties that don’t exist rather than admitting it didn’t know.

Overall, the result was positive, but there is clear room for improvement in current LLMs. Without real-world examples to reference or my own engineering experience to guide the process, I would probably still be struggling to switch the Cosmo theme to its dark version.

Final Thoughts

I still appreciate Zola’s performance and am grateful for it, but I’m very happy with Quarto so far. The deployment time is admittedly much longer now, taking over a minute to build the site from scratch. But since I don’t deploy constantly, that’s an acceptable trade-off.

This experience reinforced the value of open-source software, and I decided to open-source this blog’s repository 🔗 so others in the Quarto community can benefit from it. 🤲🏻

I look forward to exploring more of Quarto’s features, and I’m confident this site now has a strong foundation for the future.

What is MITRE Navigator?

MITRE Navigator 🧭 is a web-based tool for annotating and exploring ATT&CK matrices. It allows you to:

• Group technique mappings into layers, then overlay them for comparative analysis.
• Visualize strengths and weaknesses in your detection landscape.
• Compare defensive coverage (e.g., from your SIEM or EDR) against adversary behavior (e.g., from CTI reports).

A classic use case is to generate one layer for your SIEM detections and another for the techniques used by a tracked APT. Overlaying them immediately reveals where the adversary operates and how your defenses stack up.

Understanding Navigator Layers

In MITRE Navigator, layers are JSON documents that follow a specific schema—see Navigator’s repository on GitHub. Their behavior depends on three versioning components:

• ATT&CK version: The matrix to plot (e.g., 18.1).
• Navigator version: The software that will render the layer (e.g., 5.2.0).
• Layer version: The JSON schema itself (e.g., 4.5).

A layer has two main parts:

1. Root-level configuration: Defines the layer’s appearance and behavior, like name, description, filters, layout, color gradients.
2. Technique entries: An array of objects, each defining per-technique attributes like score, metadata, links, and an optional tactic.

jq -r '
  .objects[] |
  select(
    .type == "x-mitre-tactic"
    and .x_mitre_deprecated == false
  ) |
  (
    .external_references[0].external_id
    + ": "
    + .x_mitre_shortname
  )
' enterprise-attack-18.1.json |\
sort

Layer Configuration (The Root Object)

This is where you control the layer’s look and feel: header visibility, UI layout, sub-technique expansion, color gradients, and platform filters. Common options include:

• versions: Specifies the ATT&CK, Navigator, and layer schema versions the file targets.
• filters.platforms: Narrows the layer to specific platforms (e.g., Windows, Linux, SaaS).
• layout: Controls the layout style, visibility of names/IDs, and score aggregation.
• gradient: Defines the color stops and the minValue/maxValue that the score maps to.

Consistent root-level settings are key to making your exported layers readable and uniform.

Technique Configuration (The Heart of the Data)

The techniques array is what turns raw data into usable intelligence. Each object in the array corresponds to a specific detection-to-technique mapping and should include enough context to be useful, like rule name, maturity score, description, a link to the detection-as-code, owner, and test status.

If a single detection rule maps to multiple techniques, create a distinct technique object in this array for each one. This pattern keeps traceability clear: from any cell in Navigator, an analyst can click through to the detection’s code, review its maturity, see the owner, and prioritize engineering work accordingly.

Automation: A Base Layer Template

Automating layer generation allows any automation pipeline to emit a consistent JSON artifact for specific rulesets. A minimal template for the root object looks like the next listing—note that the techniques array is empty here:

{
  "name": "Logs",
  "versions": { "attack": "18.1", "navigator": "5.2.0", "layer": "4.5" },
  "domain": "enterprise-attack",
  "description": "Example layer for SIEM detections.",
  "filters": { "platforms": ["Windows","Linux","macOS","Network Devices","ESXi","PRE","Containers","IaaS","SaaS","Office Suite","Identity Provider"] },
  "layout": { "layout": "side", "showName": true, "showID": true, "showAggregateScores": true, "countUnscored": false, "aggregateFunction": "average", "expandedSubtechniques": "annotated" },
  "showTacticRowBackground": true,
  "tacticRowBackground": "#2f0549",
  "gradient": { "colors": ["#D62D20", "#FFA700", "#008744"], "minValue": 1, "maxValue": 5 },
  "techniques": [],
  "links": [],
  "legendItems": [],
  "metadata": [],
  "selectTechniquesAcrossTactics": true,
  "selectSubtechniquesWithParent": false,
  "selectVisibleTechniques": false
}

Example Technique Object

The next listing has a concrete example of a single technique object you would append to the techniques array in the base JSON.

{
  "techniqueID": "T1078",
  "tactic": "initial-access",
  "score": 2,
  "comment": "Rule: aws_cloudtrail_trail_stopped",
  "metadata": [
    { "name": "aws_cloudtrail_trail_stopped", "value": "Maturity: 2\nDescription: A CloudTrail audit trail was stopped" }
  ],
  "links": [
    { "label": "Source Code", "url": "https://github.com/lopes/detections/blob/main/rules/aws_cloudtrail_trail_stopped.yaml" }
  ],
  "enabled": true,
  "showSubtechniques": false
}