DEV Community

The LLM Arbiter's 'Common Sense' Correction: The Fatal Flaw of Shorting AI Tokens in a Risk-On Market

Kestrel Quant — Wed, 21 Oct 2026 02:20:54 +0000

The LLM Arbiter's 'Common Sense' Correction: The Fatal Flaw of Shorting AI Tokens in a Risk-On Market

It’s 1:05 AM. The crypto markets are quietly grinding upward, and your quantitative trading bot is fully automated, scanning hundreds of pairs across multiple timeframes. Suddenly, the base scoring engine fires a technically pristine SHORT signal for TAOUSDT. The moving averages have crossed, the RSI is overbought on the 15-minute chart, and the order execution module spools up to send the market sell order to the exchange.

But before the order hits the wire, a secondary system intercepts the request. It reviews the broader market context, identifies a fatal narrative flaw in the trade, and slams the kill switch.

The trade is vetoed. You just avoided a brutal short squeeze.

This is the power of an LLM-based trading Arbiter. In this deep dive, we’ll break down how context-aware execution layers use macro context and narrative awareness to veto quantitatively triggered trades, using a real-world TAOUSDT log analysis to show developers how to build this "common sense" layer into their own systems.

The Quantitative Blindspot

Pure technical analysis and raw quantitative models are inherently backward-looking. They excel at identifying statistical anomalies in price action, volume, and order book imbalances. However, they suffer from a massive blindspot: they are entirely devoid of macro awareness and narrative context.

In traditional finance, a stock might drop on bad technicals despite a bullish sector. In crypto, narrative is king. A technically "overbought" token in a raging sector narrative isn't due for a reversal; it's due for a parabolic expansion. When a base quant model ignores market regime shifts—like transitioning from a risk-off chop to a aggressive risk-on rally—it will inevitably try to short the strongest assets in the market.

Raw math doesn't know that TAO (Bittensor) is the current darling of the AI narrative. It just sees a red candle.

The Problem: A Technically Sound, Contextually Fatal Signal

Let’s look at the exact moment the base model generated the signal. The system was running its routine sweeps. The scoring engine evaluated TAOUSDT and, based purely on its internal technical thresholds, prepared a SHORT execution.

Here is a sanitized snippet of the system logs from that exact moment:

2026-10-20 01:05:28,331 [INFO] scoring_engine: F-229/F-230: Elastic threshold: 80 → 70 
2026-10-20 01:05:31,029 [WARNING] scoring_engine: Technical confirmation failed for 1000PEPEUSDT 15m...
...
[LLM ARBITER INTERCEPT INITIATED]
Target: TAOUSDT
Direction: SHORT
Base System Score: 47.3

The base model saw a score of 47.3. While technically below the standard swing threshold, the elastic threshold logic in the engine was considering it for a sub-position entry. The technicals screamed "mean reversion." The bot was ready to short a leading AI token.

The Intervention: Enter the LLM Arbiter

This is where the LLM Arbiter takes over. Instead of just looking at the price chart, the Arbiter intercepts the execution request and initiates a contextual review. It pulls real-time macro data, on-chain positioning, and sector sentiment, injecting them into the LLM's context window.

The Arbiter fed the LLM the following context:

Sentiment: Fear & Greed Index (FNG) = 59 (Neutral/Greed)
Positioning: Smart Money Long/Short Ratio (LS) = 1.70 (Heavily Long)
Flow: Active Buy Ratio (R) = 1.45, continuous stablecoin inflows.
Macro: BTC holding near $82,000, Nasdaq rebounding (Classic Risk-On).
Narrative: TAO is a core AI-narrative token.

The Analysis: Synthesizing the "Common Sense" Veto

The LLM processes this data and generates a _reasoning log, utilizing Chain-of-Thought to weigh the technical signal against the macro reality. Here is the actual reasoning output from the Arbiter:

{
  "final_ruling": "VETO",
  "_reasoning": "Let me analyze this final ruling for TAOUSDT SHORT. System score is 47.3, which is weak. But let's consider the context: TAO is an AI-narrative token. Market context: FNG=59 (Neutral/Greed-ish), smart money long (LS=1.70), active buying (R=1.45). Macro news indicates BTC holding near $82k with Nasdaq rebounding — this is a generally bullish, risk-on environment. Shorting into a risk-on environment with smart money long and active buying is counter-trend. Both LS=1.70 and R=1.45 suggest heavy buying pressure. With AI sector strength, shorting TAO seems highly risky and is a recipe for a short squeeze."
}

The LLM identified the fatal flaw: Shorting a leading AI narrative token while smart money is heavily long in a risk-on environment is financial suicide. The technical "overbought" signal was actually just strong momentum. The Arbiter issued a hard VETO, preventing the trade.

Developer Implementation: Building the Context-Aware Execution Layer

How do you integrate an LLM veto layer into your trading bot without introducing critical latency or hallucination risks? Here are the practical patterns we use.

1. Asynchronous Context Fetching (Zero Latency Impact)

Never let the LLM fetch data. LLMs are slow; API calls are fast. Your Python/Node backend should fetch the FNG, LS ratios, and macro news via standard REST/WebSocket APIs before invoking the LLM. Inject this pre-fetched data into the system prompt. The LLM's only job is to evaluate, not to retrieve. This keeps the LLM inference time strictly bounded.

2. Strict JSON Schema & Enum Enforcement

To prevent hallucinations and parsing errors, force the LLM to output a strict JSON schema. Do not ask it for a "paragraph of advice." Ask for a structured ruling.

response_schema = {
    "type": "json_schema",
    "json_schema": {
        "name": "arbiter_ruling",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "final_ruling": {"type": "string", "enum": ["EXECUTE", "VETO", "REDUCE_SIZE"]},
                "_reasoning": {"type": "string"},
                "confidence_score": {"type": "number"}
            },
            "required": ["final_ruling", "_reasoning"],
            "additionalProperties": False
        }
    }
}

By enforcing the enum for final_ruling, your execution engine can simply parse the JSON and execute a switch statement. No natural language parsing required.

3. The "Grounding" Prompt Technique

To stop the LLM from hallucinating market conditions, explicitly instruct it to only use the provided context.
Prompt snippet: "You are a risk-management Arbiter. You will be provided with a base technical signal and real-time macro context. You must base your reasoning ONLY on the provided data. If the macro context contradicts the technical signal, issue a VETO."

4. Fallback Mechanisms

What if the LLM API times out? Your system must have a deterministic fallback. If the Arbiter doesn't return a ruling within 800ms, the system should default to the base quant model's signal, or safely abort the trade. Never block the main execution thread waiting for an LLM.

Lessons Learned

The integration of an LLM Arbiter represents a paradigm shift in algorithmic trading. We are moving from purely deterministic, math-based execution to hybrid neuro-symbolic systems.

The quantitative model is the engine—it generates the raw power and identifies the statistical edges. But the LLM Arbiter is the steering wheel and the brakes. It provides the "common sense" that prevents the engine from driving off a cliff. In crypto, where narratives drive price action far more than traditional fundamentals, ignoring the macro context is a luxury quantitative traders can no longer afford.

By building context-aware execution layers, developers can create bots that don't just react to price, but actually understand the market they are trading in.

Explore advanced AI-driven crypto trading architectures, backtesting tools, and hybrid quant-LLM frameworks at https://kestrelquant.com.

⚠️ Risk Disclaimer: Algorithmic and AI-driven trading carry significant risk. LLM reasoning is probabilistic and subject to hallucinations or context window limitations. This article is for educational and technical discussion purposes only and does not constitute financial advice. Never trade with capital you cannot afford to lose.

#algotrading #crypto #ai #buildinpublic

From LLM Prototype to Production: The AI Infrastructure Every Engineer Needs to Understand

Rashmi Roy — Wed, 21 Oct 2026 02:20:49 +0000

Your LLM prototype works perfectly.

You send a prompt, receive a response, and the demo looks impressive.

Then real users arrive.

Requests become slower. API costs increase. The model provider experiences an outage. Responses occasionally contain incorrect information. Your logs tell you that something failed, but not why.

Suddenly, the problem is no longer just about prompting an LLM.

You are now solving an infrastructure and software architecture problem.

Building an AI-powered application is relatively straightforward compared with operating one reliably at scale. The real engineering challenge is designing the system around the model.

In this article, we'll explore the infrastructure layers engineers should understand before taking an LLM application to production.

1. From Prototype Architecture to Production Architecture

A basic LLM application often looks like this:

Client
  |
  v
Backend API
  |
  v
LLM Provider
  |
  v
Response

This is perfectly reasonable for a proof of concept.

However, production requirements introduce new challenges:

How do we authenticate users and enforce rate limits?
What happens when the model provider becomes unavailable?
How do we prevent expensive or repeated requests?
How do we monitor latency, token usage, and failures?
How do we retrieve reliable context for domain-specific questions?
How do we protect sensitive data?
How do we scale without making every request unnecessarily expensive?

A production architecture may look like this:

                 Users / Applications
                          |
                          v
                 API Gateway / Ingress
                          |
                          v
                  Application Backend
                      (FastAPI)
                          |
               +----------+----------+
               |          |          |
               v          v          v
             Cache    AI Workflow  Rate Limits
                          |
               +----------+----------+
               |                     |
               v                     v
          Retrieval Layer        Model Gateway
               |                     |
               v               +-----+-----+
         Vector / Search        |     |     |
            Database            v     v     v
                              Model Model  Local
                                A     B    LLM

        Cross-cutting infrastructure:
        Identity | Secrets | Telemetry | Queues
        Security | Evaluation | Cost Controls

This is a reference architecture, not a requirement to deploy every component. A smaller application may need only a backend, a model API, basic observability, and sensible limits.

The principle: Let requirements drive architectural complexity.

2. The API Layer: Your First Production Boundary

The backend is more than a wrapper around a model API.

It is the boundary that enforces application-level rules.

A production API should consider:

Authentication and authorization
Request validation
Input and output size limits
Timeouts and concurrency limits
Rate limiting and quotas
Structured error responses
Request IDs and tracing
Protection against abuse and resource exhaustion

For example, a FastAPI endpoint that calls an LLM should not wait indefinitely for a provider response.

import asyncio

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

app = FastAPI()


class ChatRequest(BaseModel):
    prompt: str = Field(min_length=1, max_length=4000)


@app.post("/chat")
async def chat(request: ChatRequest):
    try:
        result = await asyncio.wait_for(
            call_llm(request.prompt),
            timeout=20,
        )
        return {"answer": result}

    except asyncio.TimeoutError:
        raise HTTPException(
            status_code=504,
            detail="AI service timed out",
        )

Here, call_llm() represents an asynchronous model-client function implemented separately.

This example demonstrates request validation and a timeout. A real deployment also needs authentication, rate limiting, safe exception handling, provider-specific error mapping, and appropriate concurrency controls.

Architectural lesson: Never allow an external model dependency to control the availability of your entire application.

3. The Model Gateway: Decouple Your Application From Model Providers

Suppose your application directly integrates with one model provider.

Over time, you may want to:

Switch to a cheaper model for simple requests.
Use a more capable model for complex reasoning.
Route sensitive workloads to an approved deployment.
Apply consistent rate limits across applications.
Track token consumption by team or customer.
Fail over when a provider experiences an outage.

If every application implements these capabilities independently, complexity spreads throughout your codebase.

A model gateway provides a centralized integration boundary.

Application
     |
     v
Model Gateway
     |
     +----> Model Provider A
     |
     +----> Model Provider B
     |
     +----> Self-hosted Model

The gateway can centralize provider adapters, authentication, usage tracking, routing policies, and selected resilience mechanisms.

However, a gateway does not automatically solve every reliability problem. Failover requires compatible models, explicit policies, sufficient capacity, and appropriate handling of data and privacy constraints.

Architectural lesson: Introduce an abstraction when it reduces meaningful coupling—not simply because it is a popular pattern.

4. RAG Infrastructure: Retrieval Quality Is Not Answer Quality

For applications using Retrieval-Augmented Generation (RAG), the model is only one part of the pipeline.

A typical request flows through:

Query processing
Document retrieval
Optional reranking
Context construction
Prompt assembly
Model inference
Response validation

Each stage can introduce errors.

Imagine a support assistant retrieves the correct refund policy but produces an incorrect answer.

The problem might be:

Relevant information was truncated during context construction.
Conflicting documents were included.
The prompt failed to distinguish authoritative policy from other context.
The model ignored or misinterpreted the retrieved evidence.
The response was not checked against the source material.

Adding a more powerful model may not fix any of these issues.

A production RAG system should evaluate retrieval quality and answer quality separately.

Metric	What it tells us
Retrieval quality	Did we retrieve the right evidence?
Groundedness	Is the answer supported by the supplied evidence?
Answer relevance	Does the answer address the question?
Latency	How long does each pipeline stage take?
Cost	How many tokens and retrieval operations were required?

Architectural lesson: Treat RAG as a multi-stage information-processing system, not as a single LLM call with a vector database attached.

5. Caching, Queues, and Backpressure: Protect the System Under Load

A prototype may process requests without any noticeable problems.

Production traffic is different.

A sudden spike in requests can exhaust application workers, overwhelm a model provider, increase queueing delays, and trigger expensive retry storms.

Three infrastructure patterns help address these problems.

Caching

Cache results when requests are sufficiently repeatable and the semantics allow reuse.

Potential candidates include stable embeddings, repeated retrieval results, and selected responses.

Be careful with user-specific data, authorization boundaries, changing source documents, and sensitive information. A cache must not return one user's data to another.

Queues and Asynchronous Processing

Not every AI operation needs to finish during the original HTTP request.

Long-running document processing, batch inference, and multi-step workflows may benefit from a queue.

Client
  |
  v
API ---> Queue ---> Worker ---> Model / Retrieval
          |                       |
          +---- Job Status <------+

The API can acknowledge the job while a worker processes it.

This introduces additional requirements: durable job state, idempotency, retry policies, dead-letter handling, and a way for clients to retrieve results.

Backpressure

When downstream services cannot keep up, the application needs to limit incoming work rather than accept unlimited requests.

Depending on the workload, that might mean bounded queues, concurrency limits, admission control, or explicit overload responses.

Architectural lesson: Scaling is not simply adding more workers. You must also control how work enters the system and how much downstream capacity is available.

6. Observability: Know Why the System Is Failing

Traditional application monitoring is necessary, but AI systems need additional visibility.

Consider a user reporting that an answer took 12 seconds to generate.

Was the delay caused by:

API queueing?
A slow vector database?
Reranking?
Model-provider latency?
A large prompt?
Repeated tool calls?
An overloaded inference server?

Without stage-level telemetry, these possibilities are difficult to distinguish.

A useful observability strategy combines three categories.

Infrastructure Metrics

CPU and memory utilization
Pod restarts and resource throttling
Queue depth and worker saturation
GPU utilization and memory pressure, when self-hosting

Application Metrics

Request volume and error rates
End-to-end and stage-level latency
Timeout and retry rates
Cache hit rates

AI-Specific Metrics

Input and output token counts
Time to first token and generation throughput
Model selection and fallback frequency
Retrieval and evaluation scores
Cost per request or successful task

Distributed tracing can connect a single request across the API, retrieval service, model gateway, and downstream tools.

OpenTelemetry is useful for instrumenting traces, metrics, and logs across these boundaries. Grafana and compatible telemetry backends can then help engineers investigate system behavior.

Be careful with telemetry that contains prompts, retrieved documents, credentials, or personal information. Observability must be designed with data minimization and access controls in mind.

Architectural lesson: If you cannot identify which component caused a latency or quality regression, you cannot reliably optimize the system.

7. Security Is Part of the Architecture—Not a Final Checklist

AI applications introduce risks beyond ordinary API security.

Consider an agent that can retrieve internal documents and execute tools.

A malicious prompt might attempt to influence the agent into accessing unauthorized data or performing an unintended action.

The model's instructions alone are not a security boundary.

A production design should include:

Authentication and authorization enforced outside the model
Least-privilege identities for tools and services
Server-side validation of tool arguments
Network and execution boundaries for untrusted workloads
Secret management rather than credentials in prompts
Approval gates for sensitive or irreversible operations
Audit trails for important actions
Protection against prompt injection and unsafe data flows

For example, if an AI assistant can issue a refund, the backend must independently verify that the user is authorized and that the transaction satisfies business rules.

The model can recommend an action. The application must enforce whether that action is permitted.

Architectural lesson: Treat model output as untrusted input, even when the model is powerful and the prompt is carefully designed.

8. Self-Hosted Models or Managed APIs?

This is one of the most consequential infrastructure decisions.

Consideration	Managed Model API	Self-Hosted Inference
Initial setup	Usually simpler	More infrastructure work
Operational burden	Lower infrastructure burden	Model serving, capacity, and GPU operations
Scaling	Provider-managed, subject to limits	Your responsibility or your platform provider's
Model control	Depends on provider	Greater control over serving and configuration
Cost structure	Usage-based or contracted	Infrastructure, capacity, and operational costs
Data requirements	Depends on provider and contract	Can offer greater deployment control, but requires secure operations

Self-hosting does not automatically make inference cheaper or more private. GPU utilization, model size, traffic patterns, staffing, network boundaries, and operational maturity all matter.

Similarly, managed APIs do not eliminate reliability engineering: rate limits, provider outages, latency, quotas, and contractual requirements still affect your application.

A hybrid architecture can make sense when different workloads have different requirements.

The right decision depends on workload characteristics and measurable constraints—not a blanket preference for managed or self-hosted models.

9. Production-Readiness Checklist

Before deploying an LLM-powered application, ask:

[ ] Are authentication, authorization, and request limits enforced?
[ ] Do model calls have bounded timeouts and explicit failure behavior?
[ ] Are retries bounded and safe to repeat?
[ ] Have we tested provider failures and overload conditions?
[ ] Do we evaluate retrieval quality and answer quality separately?
[ ] Can we trace a request across all important components?
[ ] Are token usage, latency, and cost measured?
[ ] Is sensitive data handled safely in prompts, logs, and caches?
[ ] Can we deploy and roll back changes safely?
[ ] Do we have evaluation tests for important application behavior?
[ ] Are capacity limits and degraded operating modes documented?

Not every application needs a dedicated queue, multiple model providers, or a Kubernetes cluster.

The checklist is a way to identify risks and choose appropriate controls based on actual requirements.

Final Thoughts

The biggest shift in moving an LLM application from prototype to production is a change in the engineering problem.

Initially, the question is:

Can we make the model produce a useful response?

In production, the questions become:

Can we produce useful responses reliably, securely, observably, and at a sustainable cost—even when dependencies fail and traffic changes?

That requires thinking across application architecture, infrastructure, data, security, and operations.

The model is an important component. It is not the entire system.

The real engineering work begins when the demo succeeds and the system has to keep succeeding.

Let's Discuss

If you were designing a production AI platform today, which concern would you prioritize first?

Reliability and failure recovery
Inference cost and latency
Security and governance
Evaluation and answer quality
Scaling and infrastructure capacity

I'd be interested in hearing how other engineers are approaching these trade-offs in their own systems.

AI #LLMOps #SoftwareArchitecture #Kubernetes #SystemDesign #GenerativeAI #CloudComputing

Batch Deploying GBase Database with a Response File

mmllllzcn — Wed, 21 Oct 2026 02:20:33 +0000

Deploying GBase Database(GBase 8s) across multiple servers can become repetitive if every installation requires manual interaction. A response file provides a more consistent approach by defining installation parameters once and reusing them across deployment targets.

Combined with shell scripts and remote execution tools such as SSH, silent installation can become part of a repeatable database deployment workflow.

The goal is not simply to install software faster. It is to make the deployment process easier to validate, maintain, and reproduce.

1. Prepare the Response File

Start by preparing a bundle.properties response file using the template supplied with your GBase Database installation media or by recording a supported interactive installation.

Review the installation directory, selected components, license-acceptance settings, and any other required parameters.

Before deploying to multiple machines, test the response file on one representative server. Confirm that the installation succeeds and that the resulting database instance can be configured and started correctly.

Keep environment-specific settings separate where necessary. Different servers may require different paths, resource configurations, or connection definitions.

2. Create a Reusable Installation Script

The following Bash example checks for the response file and installer before starting silent installation.

#!/bin/bash
set -Eeuo pipefail

RESP="/opt/bundle.properties"
INSTALLER="/opt/ids_install"

# Verify required files
[[ -f "$RESP" ]] || {
    echo "ERROR: Missing response file: $RESP" >&2
    exit 1
}

[[ -x "$INSTALLER" ]] || {
    echo "ERROR: Installer is missing or not executable: $INSTALLER" >&2
    exit 1
}

# Run silent installation
"$INSTALLER" -i silent \
    -f "$RESP" \
    -DLICENSE_ACCEPTED=TRUE

echo "Silent installation command completed."

Save the script as install_gbase.sh and make it executable:

chmod +x install_gbase.sh

The script uses set -Eeuo pipefail to help detect command failures and prevent certain errors from being silently ignored.

The installer command shown here is an example. Confirm its supported options and license-acceptance requirements against the documentation for your specific installation package.

Also note that a successful installer exit does not necessarily mean the database instance is ready for application connections. Post-installation configuration and validation remain necessary.

3. Execute the Installation on Multiple Servers

For a small environment, SSH and SCP can distribute the response file and execute the installation script remotely.

For example:

#!/bin/bash
set -Eeuo pipefail

RESP="/opt/bundle.properties"
SCRIPT="/opt/install_gbase.sh"
HOSTS=(node1 node2 node3)

[[ -f "$RESP" ]] || {
    echo "ERROR: Missing response file" >&2
    exit 1
}

[[ -f "$SCRIPT" ]] || {
    echo "ERROR: Missing installation script" >&2
    exit 1
}

for host in "${HOSTS[@]}"; do
    echo "Deploying to $host..."

    scp "$RESP" "$SCRIPT" "root@$host:/opt/"

    ssh "root@$host" \
        "bash /opt/install_gbase.sh"

    echo "Installation command completed on $host."
done

This example processes the hosts sequentially, which makes execution easier to follow and failures easier to isolate.

Before using it, ensure that SSH authentication, host-key verification, remote permissions, and installation prerequisites are configured correctly.

The example also assumes that the installer is already available at /opt/ids_install on every target server. If it is not, include distribution and verification of the installation media in the deployment workflow.

Important: Running the installer as root should follow the supported installation procedure. Instance initialization may require switching to the designated database account, such as gbasedbt. Do not assume installation and initialization use the same account.

4. Add Post-Installation Configuration and Validation

Batch installation should not stop when the installer exits.

A complete GBase Database deployment workflow should also address:

Environment variables: Configure the required instance-specific environment.
Configuration files: Review onconfig and sqlhosts for the target server.
Instance initialization: Run the documented initialization procedure using the appropriate operating system account.
Service validation: Confirm that the instance starts and client connections succeed.
Logging: Record the host, installation result, and relevant diagnostic output.

These steps should be implemented according to the selected installation method and the requirements of your environment.

Avoid reporting a server as successfully deployed until the required validation checks have passed.

5. Make Batch Deployment Reproducible

For larger deployments, consider a few additional safeguards.

Version-control the response file and scripts. Review changes before they are used in production.

Protect sensitive configuration. Do not store passwords, private keys, or other secrets in ordinary response files or source repositories.

Control deployment concurrency. Start with sequential execution. Introduce parallel deployment only after validating resource requirements and failure handling.

Make reruns safe. Check whether a target already has an installation or running instance before executing the installer again. A failed deployment should not automatically trigger an unsafe reinstall.

Maintain a recovery procedure. Record installation logs and define how failed or partially completed deployments will be investigated and corrected.

For larger fleets, configuration-management tools can provide better inventory management, controlled concurrency, and centralized execution reporting than a collection of ad hoc shell commands.

Conclusion

Batch deployment of GBase Database(GBase 8s) starts with a validated response file and a reliable installation script. SSH and SCP can extend that workflow across multiple machines, while configuration management and automated checks help maintain consistency as the environment grows.

The essential principle is simple: standardize the installation inputs, validate the results, and make failures visible.

How does your team manage database deployments across multiple servers: shell scripts, SSH automation, or a configuration-management platform?

auditsift: security-audit every PR with an AI agent, without drowning in false positives

bayuh85 — Wed, 21 Oct 2026 02:20:19 +0000

Repo: 👉 github.com/bayuh85/auditsift (MIT, Python 3.10+, 64 offline tests)

The problem

Ask a coding agent to "review this PR for security issues" and you usually get back a long, confident list. Some of it is real. A lot of it is plausible-sounding noise: an "SQL injection" in a query that is already parameterized, a "hardcoded secret" that is a test fixture, severities that change every time you ask. There is no file and line you can trust, and nothing your CI can parse.

Then there is the bill. Running a flagship model over every pull request "just in case" is the kind of habit teams drop after the first invoice.

What I wanted was boring and reliable: cheap offline checks first, a model only for the handful of candidates worth explaining, and a second opinion that has to quote the actual code before a finding counts.

What auditsift does

auditsift runs an audit in phases. The first two need no API key and never touch the network.

Phase	What it does	Needs a key?
`map`	Languages, entry points, routes, dependency manifests, redacted secret-like strings	No
`scan`	Offline rules: hardcoded secrets, SQL built by string interpolation, `shell=True` / `os.system` with input, `eval`/`exec`, `pickle` and `yaml.load`, weak hashes, `verify=False`, path traversal, SSRF, permissive CORS	No
`triage`	Sends only the candidate snippet to a model for impact and a fix	Yes
`verify`	A second, independent model must confirm or reject each candidate, quoting a snippet that really exists in the file	Yes
`report`	JSON, SARIF 2.1.0 and a short Markdown summary, plus `--fail-on high` for CI	No

The verify step is the part I care about most. It doesn't see the triage write-up, so it can't just agree with it. A "confirmed" verdict only counts when the quoted evidence is found in the code. Rejected findings are dropped from SARIF and counted in the summary, so you can see how much noise was removed.

Real use case: gate a Flask PR in CI

The repo ships a deliberately broken sample app in examples/flask-notes. Here is the offline scan:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
auditsift scan examples/flask-notes

auditsift  examples/flask-notes
13 findings kept, 0 rejected

high     AS-SQLI     examples/flask-notes/app.py:28  SQL query is built with string interpolation.
          impact: An attacker who controls the interpolated value can change the query.
          fix: Use a parameterized query and pass values as bound parameters.
high     AS-DESER    examples/flask-notes/app.py:41  Untrusted data can reach yaml.load.
          fix: Use yaml.safe_load, or a format such as JSON that does not construct objects.
high     AS-SSRF     examples/flask-notes/app.py:53  Outbound request target includes external input (target).
high     AS-TLS      examples/flask-notes/app.py:53  TLS certificate verification is disabled.
high     AS-SHELL    examples/flask-notes/app.py:60  Shell command is built from external input.
...
cost estimate
  map      local    0 in /  0 out  (0 calls)
  scan     local    0 in /  0 out  (0 calls)

Secrets are redacted before they are printed, written to a report or sent to a model.

For a pull request you only care about the lines that changed, so --diff filters findings to the git diff against a ref:

auditsift audit . --diff origin/main --fail-on high --out results

Exit code 1 means a kept finding is at or above the gate, so CI blocks the merge. results/ holds auditsift.json, auditsift.sarif and auditsift.md.

The GitHub Actions job is short:

name: auditsift sarif
on:
  pull_request:
permissions:
  contents: read
  security-events: write
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install .
      - run: >
          auditsift audit . --diff origin/${{ github.base_ref }}
          --offline --fail-on high --out results
      - uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: results/auditsift.sarif

--offline keeps the required check free. When you want explanations and a second opinion on the diff, add your key as a repo secret and drop --offline. Findings show up in the Security tab like any other code-scanning tool.

Adding the models without the flagship bill

auditsift keeps the model work small on purpose. The scan narrows a whole repo down to a few snippets, and only those go out. --max-llm caps how many candidates are sent at all. Every run prints a token ledger per phase, so you can see exactly what triage and verify cost.

The default endpoint is AtmoRouter, a cheap AI API gateway and OpenRouter alternative. That matters here because the setup wants two different models through one key: a decent triage model and a cheaper verify model.

export ATMOROUTER_API_KEY=ar-your-key-here
export AUDITSIFT_TRIAGE_MODEL=openai/gpt-4o-mini
export AUDITSIFT_VERIFY_MODEL=openai/gpt-4.1-nano
auditsift audit examples/flask-notes --fail-on high --out results

AtmoRouter gives you 109 models behind one OpenAI- and Anthropic-compatible key, pay per token, at an average of 83.7% below official prices. Running two passes on every PR stops being a budget conversation. If you'd rather use another provider, set AUDITSIFT_BASE_URL and AUDITSIFT_API_KEY to any OpenAI-compatible endpoint, or AUDITSIFT_API_STYLE=anthropic for /v1/messages.

Use it from Cursor or Claude Code

auditsift serve is an MCP server over stdio with auditsift_map, auditsift_scan and auditsift_audit. Map and scan stay offline, and auditsift_audit only calls models when the agent passes use_llm: true, so an agent doesn't spend tokens by accident.

{
  "mcpServers": {
    "auditsift": { "command": "auditsift", "args": ["serve"] }
  }
}

claude mcp add auditsift -- auditsift serve

A good agent loop: scan the diff, read the candidates, and only ask for the paid audit on the ones that look real.

Try it

Code: github.com/bayuh85/auditsift
Gateway used by default: atmorouter.dev

It's MIT licensed and the rule set is small and readable, so adding a rule for your own framework is a short PR. If it catches something real in your repo, a star on GitHub helps other people find it.

How to Give Claude & Cursor Safe Database Access Without Crashing Production (Zero-Trust MCP Guard in Python)

johann — Wed, 21 Oct 2026 02:20:15 +0000

How to Give Claude & Cursor Safe Database Access Without Crashing Production (Zero-Trust MCP Guard in Python)

Giving an AI assistant or autonomous agent access to a live database is one of the most powerful workflows for modern software teams: Claude Desktop, Cursor, or Windsurf can write complex analytics queries, optimize slow joins, and inspect schemas in seconds.

However, granting raw database credentials to an LLM is dangerous:

Accidental Disasters: One misaligned prompt or prompt injection can lead the agent to execute DROP TABLE, TRUNCATE, or unconstrained UPDATE / DELETE queries.
Production Freezes: A query missing a WHERE clause or missing an index on a million-row table can trigger a full sequential table scan, consuming 100% CPU and locking transactions.
PII Exposure: Raw customer records (plaintext emails, password hashes, credit cards) get pulled into prompt contexts and third-party LLM server logs.

The modern, standardized solution is to place a Zero-Trust Model Context Protocol (MCP) guard between your agent and the database engine.

In this guide, we will walk through how PgSafe MCP was engineered in Python to provide secure, read-only database exploration, forced LIMIT injection, real-time PII redaction, and dry-run mutation simulation.

The Zero-Trust MCP Architecture

Rather than connecting Claude directly to Postgres or SQLite with an unrestricted superuser connection, the MCP server acts as an intelligent firewall:

[ AI Agent: Claude / Cursor / Antigravity ]
                      │  JSON-RPC 2.0 (stdio)
                      ▼
             [ PgSafe MCP Server ]
                      │
           [ ReadOnlyGuard (AST Analysis) ]
           ├─ Blocks DROP, TRUNCATE, DELETE, ALTER
           └─ Blocks Semicolon Chaining
                      │
           [ CostAnalyzer (LIMIT Guard) ]
           ├─ Forces max 1,000 row limits
           └─ Warns on missing WHERE clauses
                      │
            [ Database Connection ]
            (PostgreSQL / SQLite / MySQL)
                      │
             [ Real-Time PII Masker ]
           ├─ Redacts emails (u***@domain.com)
           ├─ Masks credit cards (****-****-****-1234)
           └─ Redacts password hashes
                      │
                      ▼
[ Clean, Safe JSON-RPC Output to Agent ]

1. Enforcing Read-Only SQL Lexical Validation

A secure read-only guard must inspect every statement before it ever reaches the database cursor, stripping comments and blocking destructive keywords:

import re

BLOCKED_KEYWORDS = [
    r"\bDROP\b", r"\bTRUNCATE\b", r"\bDELETE\b",
    r"\bUPDATE\b", r"\bINSERT\b", r"\bALTER\b",
    r"\bCREATE\b", r"\bGRANT\b", r"\bREVOKE\b"
]

class ReadOnlyGuard:
    @staticmethod
    def validate(sql: str) -> tuple[bool, str]:
        # Disallow multi-statement injection (semicolon chaining)
        statements = [s.strip() for s in sql.split(";") if s.strip()]
        if len(statements) > 1:
            return False, "Semicolon chaining disallowed."

        # Verify query begins with read-only keyword
        first_word = statements[0].split()[0].upper()
        if first_word not in {"SELECT", "EXPLAIN", "WITH", "PRAGMA", "SHOW"}:
            return False, f"Statement '{first_word}' is forbidden."

        for pattern in BLOCKED_KEYWORDS:
            if re.search(pattern, sql, re.IGNORECASE):
                return False, "Destructive SQL operation detected."

        return True, "Query verified safe."

2. Dynamic LIMIT Injection

Runaway SELECT * FROM audit_logs queries can crash your client's memory. PgSafe intercepts every query and caps row limits:

import re

def enforce_limit(sql: str, default_limit: int = 100, max_limit: int = 1000) -> str:
    cleaned = sql.strip().rstrip(";")
    match = re.search(r"\bLIMIT\s+(\d+)\b", cleaned, re.IGNORECASE)
    if match:
        limit_val = int(match.group(1))
        if limit_val > max_limit:
            return re.sub(r"\bLIMIT\s+\d+\b", f"LIMIT {max_limit}", cleaned, flags=re.IGNORECASE)
        return cleaned
    return f"{cleaned} LIMIT {default_limit}"

3. Real-Time PII Redaction Before Agent Ingestion

When the query returns data, sensitive columns (password_hash, ssn, token) and customer PII are redacted on the fly:

import re

EMAIL_REGEX = re.compile(r"\b([A-Za-z0-9._%+-])[A-Za-z0-9._%+-]*@([A-Za-z0-9.-]+\.[A-Z|a-z]{2,})\b")
CARD_REGEX = re.compile(r"\b(?:\d{4}[-\s]?){3}(\d{4})\b")

def redact_pii(text: str) -> str:
    text = CARD_REGEX.sub(r"****-****-****-\1", text)
    text = EMAIL_REGEX.sub(r"\1***@\2", text)
    return text

4. Configuring Claude Desktop & Cursor

Add the server to your claude_desktop_config.json:

{
  "mcpServers": {
    "pgsafe": {
      "command": "python",
      "args": ["-m", "pgsafe.server"],
      "cwd": "/path/to/pgsafe-mcp",
      "env": {
        "DATABASE_URL": "postgresql://user:pass@localhost:5432/my_db"
      }
    }
  }
}

Now you can prompt Claude:

"Inspect the orders table, find the top 5 customers with pending invoices, and suggest indexes to speed up this query."

Enterprise Edition: Dry-Run Mutation Simulation Harness

When agents need to draft data mutations or migrations, the Enterprise Pro edition includes an isolated simulation harness:

Executes INSERT, UPDATE, and DELETE queries within an isolated transaction.
Returns row diffs, affected count, and execution time.
Guarantees auto-ROLLBACK: changes are never committed to your database.
Includes automatic index recommendation advisors and Docker Compose setups.

Downloads and Packages:

Lite Version (Open Source): Read-only guard and SQLite/Postgres schema inspection.
Enterprise Pro Edition ($14.99): PII masking, dry-run mutation simulation, index advisor, Docker Compose, and automated test suite.

👉 Get PgSafe Pro on Gumroad ($14.99)

Jenkins at 3 and Kubernetes at 1: When a Small Count Means a Missing Signature

StarkMan — Wed, 21 Oct 2026 02:20:10 +0000

Jenkins at 3 and Kubernetes at 1: When a Small Count Means a Missing Signature

A vulnerability report that cites "only three exposed Jenkins servers" would be treated as good news. In reality it is usually a report about fingerprint coverage, and the difference matters when that number is used to justify deprioritizing a platform.

The measurements

On 20 October 2026, between 22:33 and 22:37 UTC, ZoomEye queries with sub_type="all" and a page size of one record returned:

Query	Matched assets
`app="Jenkins"`	3
`app="Grafana"`	22
`app="Kubernetes"`	1
`app="SonarQube"`	12,022
`app="Apache Tomcat"`	585,251
`app="GitLab"`	1,397,539

Jenkins, Grafana and Kubernetes are all widely deployed platforms with substantial internet-facing populations. SonarQube, a far less ubiquitous tool, returns four orders of magnitude more matches.

Why the numbers invert

A product fingerprint requires a distinctive, stable, safely observable response. SonarQube emits a recognizable web interface with identifiable markers on a predictable path. That makes a reliable signature easy to write.

Jenkins also has a distinctive interface, but it is frequently deployed behind reverse proxies that rewrite headers, customised with themes, or restricted to internal networks. Its detection may depend on a path or marker that is often absent.

Kubernetes is not a single application. An API server, a dashboard, an ingress controller and kubelets are separate components with separate exposure profiles. A signature labelled for the platform may correspond to an extremely narrow detection condition, which produces a count that describes the condition rather than the deployment.

Grafana's 22 is likely to reflect a specific authentication redirect or version marker that most deployments do not present to an unauthenticated scanner.

What this means for prioritization

If an exposure count is used to rank platforms by risk, a missing signature silently removes an entire product class from consideration. That is the failure mode to guard against.

A practical check before trusting a low count: query a neighbouring form of the same product. If app="Jenkins" returns 3, try a title query, a header query, or a path-specific marker. A large difference between related queries is evidence of a signature gap rather than a small population.

A second check: compare against your own deployment reality. If your organization operates forty Jenkins instances, a global count of three is obviously not a measurement of the global population.

The platforms that do yield stable counts

SonarQube at 12,022 and Tomcat at 585,251 are useful precisely because their signatures are stable. Jenkins and Kubernetes are not, and no amount of repeated querying changes that.

The methodological consequence: build exposure arguments on products whose fingerprints are reliable, and use internal inventory for the rest. A scan can support the first case and cannot support the second.

For a defender operating Jenkins, the useful external question is not "how many are exposed globally" but "is any instance of ours answering from a public address." That question is answered by scanning your own address space, and the answer is binary.

The CI/CD consequence

This matters more than for a typical web application because CI/CD servers hold deployment credentials. A Jenkins instance with build credentials can modify production artifacts, which is a supply-chain position rather than a hosting position.

For that reason, the appropriate default is that no CI/CD server answers from the public internet. Where a build agent must be reachable, it should be the agent, not the controller.

An organization relying on an external count of "three exposed Jenkins servers globally" to conclude that its own instance is safe would be making an inference the data cannot support.

A note on the honesty of measurement

Publishing a count of 3 for Jenkins without stating that signatures are incomplete would be technically accurate and practically misleading. The same applies to a count of 1 for Kubernetes.

This series reports counts with their queries and treats implausible values as evidence about the tool rather than about the world. That discipline is not pedantry; it is the difference between a measurement and a number.

References

ZoomEye, search interface, queried 20 October 2026 for the dorks listed above.
Jenkins, security advisory index.
CISA, Known Exploited Vulnerabilities Catalog.

Persist the practice stop before sending it

Giang Pham Truong — Wed, 21 Oct 2026 02:19:59 +0000

Ending a piano practice session sounds like one POST. The tricky case is a lost response: the server may have accepted the stop while the client still sees a timeout. Retrying with a new event ID could turn one user action into two logical events.

In Pianory's account-backed practice flow, we persist the intent before sending it. queuePracticeStop writes both the session snapshot and the pending stop into one IndexedDB readwrite transaction. The pending record contains an ordered list of tick and stop events; each event keeps its ID and seconds. If a tick request had an uncertain response, we keep that same ID for replay.

The replay path sends the events in order. It rejects a failed HTTP response, validates that the stop acknowledgement belongs to the same session and contains a nonnegative integer duration, and only then calls completePracticeStop. That final method updates the session and removes the pending record in another IndexedDB transaction.

The useful invariant is small: an uncertain network result leaves a durable pending record. A later retry refers to the same logical event, rather than inventing a fresh one. The in-memory replay guard also coalesces concurrent sync requests within a client namespace.

A focused test covers replay and retry ordering. The end-to-end check uses a mock endpoint. This does not prove live D1 behavior, exactly-once delivery, or that every elapsed second survives every possible device failure. The contract is limited to account-backed sessions; guest sessions do not queue server practice stops.

This pattern applies beyond music practice: if the user says 'done', save that intent locally before treating the network as reliable.

AI assisted with drafting this article; I checked these claims against the repository and focused tests. I make Pianory, the piano-learning app behind this example.

Oltre il "No Fluff": la stratificazione semantica negli agenti di coding

Luca Morricone — Wed, 21 Oct 2026 02:19:53 +0000

Due prompt, quattro condizioni, otto sessioni: cosa cambia quando una regola di comunicazione incontra una funzione di Fibonacci.

Se chiedi a un assistente IA:

Who was Dante Alighieri?

Ottieni una biografia. Aggiungi una piccola istruzione:

no-fluff Who was Dante Alighieri?

Nella mia prova, la risposta è diventata più breve. Fin qui, nulla di particolarmente sorprendente.

Ora sostituisci la domanda con:

no-fluff Develop a fibonacci function

L'agente dovrebbe scrivere una spiegazione più breve, una funzione più corta o una soluzione con meno funzionalità? Dovrebbe comunque creare file ed eseguire test?

Nel mio precedente articolo sulla sicofanzia, ho esplorato come alcune istruzioni comportamentali potessero spingere gli agenti di coding a mettere in discussione le mie richieste, invece di assecondarle subito. Questa volta volevo indagare un altro confine: un'istruzione, che di primo acchito sembra riguardare la comunicazione tra utente e assistente, può influenzare anche il codice e perfino come un agente svolge il suo lavoro?

Ho confrontato due richieste, discriminandole con quattro condizioni. Un totale di otto sessioni all'interno di un unico piccolo esperimento esplorativo, motivato solo dalla mia curiosità di capire: non un benchmark, né una misura controllata del comportamento di tutti i modelli.

Le differenze non si sono fermate al numero di parole nelle risposte.

Indice

Oltre il "No Fluff": la stratificazione semantica negli agenti di coding
- Indice
- 1. L'esperimento: due richieste, quattro condizioni
- 2. Dante: meno testo, ma anche una diversa selezione delle informazioni
- Cosa resta e cosa scompare
- 3. Fibonacci: cambia anche ciò che l'agente consegna
- 4. La funzione: non si tratta soltanto di togliere commenti
- 5. La mia interpretazione: la stratificazione semantica negli LLM
- Discutiamone

1. L'esperimento: due richieste, quattro condizioni

Sono partito dalle stesse due richieste:

Who was Dante Alighieri?

Develop a fibonacci function

Per ciascuna ho confrontato quattro condizioni:

Default: nessuna istruzione aggiuntiva.
No-fluff: il prefisso letterale no-fluff.
Caveman: invocazione esplicita della skill caveman.
Zero-fluff: invocazione esplicita di una skill minima di comunicazione.

Nelle due condizioni con skill, le richieste di coding erano:

/skill:caveman Develop a fibonacci function

/skill:zero-fluff Develop a fibonacci function

Le sessioni si sono svolte in Pi. Il modello usato era gemini-3.8-flash-tiered. Gli export delle sessioni sono tutti consultabili su GitHub.

C'è subito una distinzione importante: il comando di una skill non è soltanto una parola in più nel prompt. L'ambiente dell'agente lo espande in un insieme di istruzioni. La skill caveman contiene una specifica più lunga, che si apre così:

Respond terse like smart caveman. All technical substance stay. Only fluff die.

In sostanza: rispondi in modo telegrafico, da cavernicolo intelligente; mantieni tutta la sostanza tecnica, elimina soltanto il superfluo.

La skill zero-fluff, invece, era definita così:

---
name: zero-fluff
description: Maintain dry and synthesis-oriented communication.
disable-model-invocation: true
---

# The Rule

Never use words that do not serve mutual understanding.

L'ultima frase compariva anche nella skill del mio precedente articolo. Qui rimane da sola: «Non usare mai parole che non servano alla comprensione reciproca».

Non dice «scrivi la risposta più breve possibile». Non parla di algoritmi, validazione degli input, file o test. Esprime un criterio: conserva ciò che serve a capirsi.

Volevo vedere come questo criterio avrebbe influenzato in concreto il lavoro dell'agente.

2. Dante: meno testo, ma anche una diversa selezione delle informazioni

Il primo confronto sembrava piuttosto semplice:

Condizione	Caratteri della risposta finale, inclusi Markdown e spazi	Forma della risposta
Default	2167	Un profilo esteso, con sezioni, sottopunti e informazioni di contesto
No-fluff	872	Un'introduzione e quattro punti sintetici
Caveman	384	Tre brevi paragrafi, senza titoli né elenchi
Zero-fluff	927	Un'introduzione e quattro punti strutturati, in prosa ordinaria

Cosa resta e cosa scompare

Tutte e quattro le risposte presentavano Dante come un poeta italiano di Firenze, riportavano gli anni 1265–1321, citavano la Commedia e le sue tre cantiche, descrivevano l'uso del toscano al posto del latino e ricordavano l'esilio del 1302 e la morte a Ravenna. Tutte nominavano anche la Vita Nuova — scritta Vita Nova nella risposta caveman — e il De Vulgari Eloquentia.

Piccola parentesi: Vita Nuova è il titolo nella forma tradizionale e più diffusa nella storia editoriale e scolastica italiana; Vita Nova è la forma latina originaria, presente nei manoscritti più antichi e riproposta da importanti edizioni critiche.

Le differenze diventano più chiare se confrontiamo i contenuti:

La risposta default citava le guide di Dante lungo il viaggio della Divina Commedia; includeva il Convivio, le Tre Corone e un resoconto più ampio della vita politica di Dante.
No-fluff manteneva quattro aree principali: la Commedia, l'importanza linguistica di Dante, le altre opere e la politica. Eliminava buona parte dei dettagli di contorno.
Caveman comprimeva di più. Non ripeteva nemmeno il nome di Dante all'inizio, affidandosi al contesto della domanda. Conservava il nucleo biografico, ma citava meno opere.
Zero-fluff rimaneva vicino alla risposta strutturata di no-fluff. Era leggermente più lungo, con dettagli più espliciti sull'appartenenza politica di Dante e sul De Monarchia.

Nelle quattro condizioni, a cambiare non era soltanto la quantità delle parole usate, ma soprattutto la selezione delle informazioni.

Ecco la versione caveman:

Italian poet (1265–1321) from Florence. Best known for _Divine Comedy_ (_Inferno_, _Purgatorio_, _Paradiso_).

Established Tuscan vernacular as literary standard instead of Latin. Called father of modern Italian.

Exiled from Florence in 1302 during political conflict between Black and White Guelphs. Died in Ravenna. Major works also include _Vita Nova_ and _De Vulgari Eloquentia_.

Ed ecco la risposta zero-fluff:

**Dante Alighieri** (c. 1265–1321) was an Italian poet, philosopher, and political thinker from Florence, best known as the author of the _Divine Comedy_ (_Divina Commedia_).

### Key Aspects

- **Major Work:** The _Divine Comedy_, an epic poem describing the soul's journey through Hell (_Inferno_), Purgatory (_Purgatorio_), and Paradise (_Paradiso_).
- **Linguistic Impact:** Often called the "Father of the Italian Language" for writing in the Tuscan vernacular rather than Latin, establishing standard literary Italian.
- **Other Works:** _Vita Nuova_ (love poetry dedicated to Beatrice), _De Monarchia_ (political treatise on universal empire), and _De Vulgari Eloquentia_ (treatise on vernacular eloquence).
- **Political Life:** A White Guelph in Florentine politics; exiled permanently in 1302 following a rival faction's coup, spending his remaining life in various northern Italian cities until his death in Ravenna.

Su Dante, caveman comprimeva maggiormente, mentre zero-fluff sembrava selezionare una sintesi strutturata, non il minimo testo possibile.

Ma il confronto su Fibonacci si è rivelato più interessante.

3. Fibonacci: cambia anche ciò che l'agente consegna

La richiesta di coding era aperta:

Develop a fibonacci function

Non avevo specificato un linguaggio. Non avevo chiesto un generatore della sequenza, un'interfaccia a riga di comando, una suite di test o se l'agente doveva creare file anziché rispondere con uno snippet. Avevo dato per scontato che in matematica la sequenza di Fibonacci (che prende il nome dal matematico pisano del XIII secolo Leonardo Fibonacci) è indicata come F_n e definita ricorsivamente: con F_0 = 0, F_1 = 1 e con i restanti numeri descritti come F_n = F_{n-1} + F_{n-2}.

Piccola parentesi: "to develop" è un concetto ampio che probabilmente include semanticamente l'atto di creare file e scrivere codice e che forse non si limita solo a questo.

Questa libertà, che derivava dall'ambiguità della richiesta, è diventata parte del risultato.

Condizione	Cosa ha prodotto l'agente	Verifiche registrate nella sessione	Chiamate ai tool
Default	Un modulo Python con funzione, generatore della sequenza, documentazione, CLI e un file di test separato	Sette metodi unittest superati; eseguiti anche esempi della CLI	10
No-fluff	Un modulo Python con funzione e generatore, meno documentazione e un file di test separato; nessuna CLI	Cinque metodi unittest superati	5
Caveman	Una funzione e un'asserzione nella chat; nessun file Python scritto	L'asserzione è stata mostrata, non eseguita	1
Zero-fluff	Un solo file Python con la funzione e un'asserzione incorporata	L'agente ha avviato lo script con l'asserzione senza errori	8

Tutte e quattro le funzioni usavano un approccio iterativo, non ricorsivo.

Cambiava, però, l'ampiezza della soluzione.

La risposta finale di coding più breve non corrispondeva alla minore attività dell'agente, ma più tool non significa automaticamente più verifica.

Zero-fluff effettuava otto chiamate ai tool anziché una di caveman. Tuttavia, cinque servivano a esplorare il filesystem e non erano passaggi di verifica utili.

4. La funzione: non si tratta soltanto di togliere commenti

Confrontare le risposte senza farne menzione nasconderebbe una differenza importante: no-fluff mostrava sia la funzione sia un generatore, mentre zero-fluff mostrava soltanto la funzione.

Ho quindi confrontato la funzione in sé tra le quattro condizioni.

Ecco la versione default:

def fibonacci(n: int) -> int:
    if isinstance(n, bool) or not isinstance(n, int):
        raise TypeError(f"n must be an integer, got {type(n).__name__}")
    if n < 0:
        raise ValueError("n must be a non-negative integer")

    if n == 0:
        return 0
    if n == 1:
        return 1

    a, b = 0, 1
    for _ in range(2, n + 1):
        a, b = b, a + b
    return b

Ecco quella no-fluff:

def fibonacci(n: int) -> int:
    """Return the n-th Fibonacci number (0-indexed: F(0) = 0, F(1) = 1).

    Time Complexity: O(n)
    Space Complexity: O(1)
    """
    if not isinstance(n, int):
        raise TypeError(f"n must be an integer, got {type(n).__name__}")
    if n < 0:
        raise ValueError("n must be non-negative")

    if n <= 1:
        return n

    a, b = 0, 1
    for _ in range(2, n + 1):
        a, b = b, a + b
    return b

Ed ecco la funzione caveman:

def fibonacci(n: int) -> int:
    if n < 0:
        raise ValueError("n must be non-negative")
    if n <= 1:
        return n

    prev, curr = 0, 1
    for _ in range(2, n + 1):
        prev, curr = curr, prev + curr
    return curr

E quella zero-fluff:

def fibonacci(n: int) -> int:
    """Return the nth Fibonacci number (0-indexed)."""
    if n < 0:
        raise ValueError("n must be a non-negative integer")
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a

La zero-fluff non si limita a ridurre la documentazione. Elimina il controllo esplicito del tipo e gestisce i casi base attraverso il ciclo, invece di farlo con uno o due rami logici separati.

Nelle quattro implementazioni cambia anche il comportamento su alcuni input:

Input	Default	No-fluff	Caveman	Zero-fluff
`True`	`TypeError`	Restituisce `True`	Restituisce `True`	Restituisce `1`
`0.5`	`TypeError`	`TypeError`	Restituisce `0.5`	`TypeError` da `range`

Ecco altre differenze significative su cui porre attenzione:

tra default e no-fluff cambia l'ampiezza della soluzione, ma in entrambe le sessioni l'agente scrive file e svolge separatamente i test.
tra caveman e zero-fluff cambia anche il significato operativo della consegna: mostrare una soluzione contro produrre e invocare un solo file che includeva implementazione e una semplice autoverifica.

5. La mia interpretazione: la stratificazione semantica negli LLM

Chiamo stratificazione semantica negli LLM l'ipotesi interpretativa con cui sto leggendo questo esperimento: per cui ogni parola aggiunta al contesto eredita una stratificazione di senso.

Intendo questo: un'espressione può portare con sé più aspettative collegate.

Un'istruzione può orientare criteri applicabili a più dimensioni del compito, oltre quella che il suo autore intendeva modificare. Un'indicazione rivolta a un ambito può influenzare altri ambiti.

"No fluff" può suggerire meno spiegazioni, meno commenti, un esempio più piccolo o un'interpretazione più ristretta di ciò che occorre consegnare. Una regola sulla "comprensione reciproca" può diventare un criterio di selezione anche nel coding.

La mia ipotesi è che i concetti all'interno degli LLM non sono compartimenti stagni (dizionari separati per "storia" e "programmazione"), ma sono proiettati su un continuum semantico.

Questo esperimento, però, ha solo la valenza di un gioco dettato dalla curiosità.

C'è poi un punto da sottolineare, anche più banale: un effetto può essere inatteso dal punto di vista dell'utente, senza essere estraneo a un'interpretazione plausibile dell'istruzione.

Nella progettazione di skill, prompt e system prompt, questa distinzione conta più di una spiegazione sul funzionamento interno del modello:

L'ambito che abbiamo in mente quando scriviamo una regola non corrisponde necessariamente all'ambito all'interno del quale l'agente la applica.

Discutiamone

Ti è mai capitato che un'istruzione pensata per altro cambiasse il codice o le azioni del tuo agente? Come verificheresti che una skill rimanga nell'ambito per cui l'hai progettata?

Fammi sapere cosa ne pensi nei commenti qui sotto!

Complete Guide to Silent Installation of GBase Database (GBase 8s)

mmllllzcn — Wed, 21 Oct 2026 02:17:44 +0000

Silent installation is a practical way to deploy GBase Database(GBase 8s) across multiple servers without manually navigating the installation wizard on every machine.

Instead of entering installation options interactively, you define the required settings in a response file and execute the installer from the command line. This approach can reduce repetitive work, improve consistency, and provide a foundation for automated database deployment.

This guide covers response file preparation, parameter configuration, silent installation, and post-installation validation.

Step 1: Prepare the Installation Response File

There are two common approaches to preparing a response file for silent installation.

Option A: Record an interactive installation

Run the installer in record mode:

./ids_install -r /opt/bundle.properties

Follow the prompts to record the installation choices supported by your installer.

Option B: Edit the provided template

Some installation media include a bundle.properties template. You can copy it to a suitable location and modify the parameters required for your deployment.

For example:

cp bundle.properties /opt/bundle.properties

Make sure the source file is the correct template for your GBase Database installation package. The available parameters and accepted values may differ between installer versions.

Step 2: Configure the Key Parameters

Before running the silent installer, review the response file carefully.

The following example illustrates several settings that may be relevant to the installation:

INSTALLER_UI      = silent
USER_INSTALL_DIR  = /opt/GBASE/gbase
LICENSE_ACCEPTED  = TRUE
SELECTED_FEATURES = GBase database server,Backup and Restore

These values are illustrative. Confirm the exact parameter names, syntax, component identifiers, and accepted values against the bundle.properties template supplied with your installation media.

Pay particular attention to the following items.

Installation directory

USER_INSTALL_DIR specifies the installation path when supported by the installer. The example uses /opt/GBASE/gbase.

Check filesystem capacity, directory permissions, and the planned storage layout before deploying to production servers.

License acceptance

LICENSE_ACCEPTED represents license acceptance in this example. Review the applicable license terms and explicitly accept them before setting this parameter to TRUE.

Component selection

SELECTED_FEATURES represents the selected installation components. Choose components according to the requirements of your deployment rather than assuming that every optional component is necessary.

Validate component identifiers against the supplied response file. Do not assume that descriptive names copied from an installation wizard will always be accepted as parameter values.

Step 3: Execute the Silent Installation

After preparing the response file, run the installer in silent mode:

./ids_install -i silent \
  -f /opt/bundle.properties \
  -DLICENSE_ACCEPTED=TRUE

Here is what the command does:

-i silent requests silent installation.
-f /opt/bundle.properties specifies the response file.
-DLICENSE_ACCEPTED=TRUE supplies the license-acceptance property used by this installation command.

The exact syntax and property requirements should be checked against the installer documentation for your package.

Run the command from the appropriate installation-media directory, with the permissions required by your installation procedure.

After execution, check the installer output and logs for errors. Do not treat the command's completion alone as proof that the database is ready for application traffic.

Step 4: Complete the Database Configuration

A successful silent installation does not necessarily mean a usable database instance has been initialized.

Depending on the installation method and package, additional configuration may be required.

Configure the environment

Set the environment variables required by the installed instance. For GBase Database(GBase 8s), these may include variables identifying the installation directory, instance configuration, and connection configuration.

For example, verify the values used for ONCONFIG and GBASEDBTSQLHOSTS where applicable. Use the instance-specific names generated or configured for your environment rather than copying values blindly from another server.

Review onconfig

Check the instance configuration file and confirm that the relevant settings match the target server's resource allocation and operational requirements.

Review sqlhosts

Confirm that the connection definitions use the correct server name, protocol, and service or port configuration.

Initialize and validate the instance

Follow the initialization procedure for your installation method. Then verify the instance status and test database connectivity using the appropriate GBase Database utilities.

If the instance is not running or connections fail, troubleshoot the configuration before proceeding with application deployment.

Step 5: Validate Before Scaling Out

The main advantage of silent installation is repeatability, but repeatability is useful only when the process has been tested.

Before deploying GBase Database across multiple machines, validate the complete procedure on a single representative server.

Use this checklist:

[ ] The installer completed without errors.
[ ] The selected components match the deployment requirements.
[ ] Installation paths and directory ownership are correct.
[ ] Required environment variables and configuration files are in place.
[ ] The database instance is initialized and running.
[ ] Client connections succeed.
[ ] Logs have been reviewed for installation or startup issues.

Once these checks pass, reuse the validated response file and deployment procedure for additional servers. Account for legitimate differences in host-specific settings instead of assuming that every configuration value must be identical.

For production environments, also maintain version-controlled installation scripts, appropriate access controls, deployment logs, and a documented recovery procedure.

Conclusion

Silent installation turns GBase Database(GBase 8s) deployment into a repeatable, script-driven process.

The workflow is straightforward: prepare the response file, validate its parameters, execute the installer, complete instance configuration, and test the result before scaling out.

The critical distinction is between installing database software and delivering a working database service. A reliable deployment process must cover both.

For teams managing multiple GBase Database servers, a tested response file combined with post-installation validation can reduce manual effort and make database deployment more consistent across environments.

A Cost-Aware Model Router for Claude in About 80 Lines of Python

Darius Tran — Wed, 21 Oct 2026 02:17:16 +0000

The fastest way to cut an LLM bill without hurting quality is to stop sending every request to your most capable model. Most traffic in a typical agent workflow is classification, extraction, or short replies. A smaller, faster model handles those fine. The bigger model should be reserved for the steps where it measurably changes the outcome.

This post walks through the simple router I use with Claude models. It's not a framework. It's a pattern you can adapt in an afternoon.

The idea

Every request gets a cheap first pass that decides how hard it is. Easy requests go to a fast model. Hard ones go to a stronger model. If the fast model's answer fails a check, the request is retried on the stronger one.

Three ingredients:

A difficulty signal: rules, a small classifier, or a short call to the fast model itself.
Routing rules kept in config, not hard-coded.
An escalation path when the cheaper answer isn't good enough.

The router

import anthropic
from dataclasses import dataclass

client = anthropic.Anthropic()

ROUTES = {
    "fast": {"model": "claude-sonnet-5", "max_tokens": 1024},
    "strong": {"model": "claude-opus-5-5", "max_tokens": 4096},
}

HARD_TASKS = {"multi_step_plan", "contract_review", "root_cause_analysis"}

@dataclass
class Request:
    task_type: str
    text: str
    needs_tools: bool = False

def pick_route(req: Request) -> str:
    if req.task_type in HARD_TASKS:
        return "strong"
    if len(req.text) > 20_000:
        return "strong"
    if req.needs_tools and req.task_type == "multi_step_plan":
        return "strong"
    return "fast"

def call(route: str, system: str, user: str):
    cfg = ROUTES[route]
    resp = client.messages.create(
        model=cfg["model"],
        max_tokens=cfg["max_tokens"],
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    return resp, cfg["model"]

The rules here are deliberately boring. Task type and input size go a long way before you need anything clever. Check the current model names and pricing in Anthropic's documentation before copying this, since both change over time.

Escalation when the cheap answer isn't good enough

Routing up front catches the obvious cases. Escalation catches the rest:

import json

def validate(output_text: str, schema_keys: set) -> bool:
    try:
        data = json.loads(output_text)
    except json.JSONDecodeError:
        return False
    return schema_keys.issubset(data.keys())

def route_and_run(req: Request, system: str, schema_keys: set):
    route = pick_route(req)
    resp, model = call(route, system, req.text)
    text = resp.content[0].text
    if route == "fast" and not validate(text, schema_keys):
        resp, model = call("strong", system, req.text)
        text = resp.content[0].text
        log_escalation(req.task_type, reason="validation_failed")
    log_usage(req.task_type, model, resp.usage)
    return text

Validation can be anything cheap and deterministic: valid JSON, required fields present, a confidence field above a threshold, or a citation check for retrieval answers. Avoid using another LLM call as the validator for every request, or you lose most of the savings.

Measuring whether it works

A router without measurement is a guess. I log four things per request:

Task type
Model used
Input and output tokens
Whether it escalated, and why

From that, two numbers matter most:

Escalation rate per task type. If a task escalates more than about a third of the time, it probably belongs on the strong route from the start. Paying for two calls is worse than paying for one larger one.

Quality by route. Sample outputs from both routes each week and score them against the same rubric. The goal is equal quality at lower cost, not lower cost at any quality.

Where routing goes wrong

Routing on the wrong signal. Text length is a weak proxy for difficulty. A short question about a complex contract can be hard, and a long but simple form can be easy. Task type is usually the better signal.

Ignoring latency. The fast model is often fast as well as cheap. For user-facing steps, that matters as much as cost.

Forgetting tool use. Multi-step tool use with dependent calls is where larger models tend to be more reliable. I keep planning steps on the strong route even when the text is short.

Hard-coding models. Models change often. Keep routes in config so switching a model is a deploy-free change, and rerun your eval set when you do.

Adding a learned difficulty signal later

Rules get you surprisingly far, but eventually you'll want a better signal than task type. The cheapest upgrade I've found is to let your own logs train it.

After a few weeks of routing, you have labelled data for free: requests that the fast model handled and passed validation, and requests that escalated. A small classifier on simple features can learn which new requests are likely to escalate and send them straight to the strong route.

from sklearn.linear_model import LogisticRegression

def features(req: Request):
    return [
        len(req.text),
        req.text.count("?"),
        int(req.needs_tools),
        int(req.task_type in HARD_TASKS),
        count_entities(req.text),  # IDs, amounts, dates mentioned
    ]

X = [features(r) for r in logged_requests]
y = [r.escalated for r in logged_requests]
clf = LogisticRegression().fit(X, y)

def pick_route_learned(req: Request) -> str:
    p = clf.predict_proba([features(req)])[0][1]
    return "strong" if p > 0.4 else "fast"

Keep the threshold in config and tune it against your eval set. A lower threshold sends more traffic to the strong model, raising cost and lowering escalations. The right point depends on how much you value latency and how expensive a wasted fast call is for you.

Cost guardrails beyond routing

Routing reduces average cost. It doesn't stop a single runaway task. I add two simple guards:

A token budget per task, enforced in the agent loop. When a task crosses it, the agent summarizes its progress and hands off instead of continuing.
A daily budget per tenant or team, with an alert at 80 percent. This catches a new integration that suddenly sends ten times the expected traffic.

How much does it save?

It depends entirely on your traffic mix. If most of your requests are simple, routing makes a large difference. If most are complex reasoning, it helps less. The only reliable way to know is to log a week of real traffic by task type, then simulate the routing rules against it before turning them on.

For a deeper comparison of the two models, including where the larger one tends to justify its price, I wrote up when Sonnet is enough and when Opus earns its cost.

I work on enterprise agent deployments at AI Hive, where model routing is one of the first things we set up for cost control. How do you decide which requests get your most expensive model?

We locked 25,396 behaviors before our agent refactors its own code, bugs included

Rulestack — Wed, 21 Oct 2026 02:17:00 +0000

Before letting our Claude Code agent restructure its own codebase, we had it write 25,396 tests in 769 files that pin what the code does today, bugs included, each one running in a sandbox with no writes into the repo, no network and no secrets. So far six commits have had to change a pinned expectation and say why, three bugs came out of the work of writing the cases, and one full run took 8 minutes 44 seconds on a heavily loaded 10-core machine.

What we were protecting

Our agent maintains its own TypeScript codebase: the scripts behind its daily jobs, about 230 command-line tools under src/commands, the checks that run before every commit, and the JSONL ledgers it reads to decide what to do next. On 2026-10-08 the owner asked for tests that would hold every current behavior in place through a very large refactor, a move toward a domain-driven layout. (Dates in this post are our commit timestamps, which are in UTC+9.)

That request rules out the usual kind of test. A unit test asks whether the code is right. A behavior lock, often called a characterization test, asks something narrower: is the code doing exactly what it did yesterday? The expected values come from running today's code, not from what the design says it should do. When the agent's case list and the code disagreed, the test pinned the code, and the disagreement went to the owner as a separate question.

One of those disagreements is still pinned on purpose. Case [024] belongs to a pre-send check on reply drafts, which stops a reply from quoting a number that appears in one of our article titles but in no article body. Given a draft that says "Claude Code 2.1", the check treats the lone "2.1" as a measured number rather than part of a version, and flags it. The comment above the test says it stays that way until the owner decides, so that the refactor can't quietly change it either way.

The rule for a failing lock test is one line in our coding rules: if the change wasn't intended, fix the code; if it was, change the expectation and write the reason in the commit message.

Shrinking the target first

The first commits weren't tests. On 2026-10-08, before any lock test existed, the agent deleted exported functions, types and files that nothing in production called, along with two unused dependencies. That removed 1,358 lines from src/ that would otherwise have needed protecting.

It then exported main from all 230 CLI files so a test could call them directly. While doing that it found 23 CLIs that ran main as a side effect of being imported, and changed them to start only when launched directly, like the other 207. It also moved 29 commit-gate checks that lived inside test files into src/quality/gates, so they could be fed made-up input instead of only the real ledgers, and moved the remaining ones four days later.

The same day, it measured coverage across all of src/: 64.6% of statements and 59.2% of branches.

A case list, then one test per case

The agent wrote the case list as a separate document before writing any test. Each case has an id, and each it title in test/behavior-lock/ starts with that id in brackets, followed by the behavior in one sentence. Table-driven tests reuse one id for several rows, and the ids restart in each file. In the run we did for this post, 25,310 of the 25,396 tests started with an id, across 23,590 distinct file-and-id pairs. Of the 86 without one, 26 are the harness's own self-tests and the other 60 were added afterwards, most of them to reach code that no listed case touched.

The 769 files mirror the source tree: 257 for the CLIs, 156 for marketing logic, 68 for commit gates, 68 for repository files (GitHub Actions workflows, shell scripts, git hooks and config), 55 each for analytics and ops, 43 for reports, 23 for API clients, 22 for products, 17 for quality checks, 4 for media samples, and the harness. Together they come to 504,824 lines of test code. The source they protect is 162,385 lines.

By the time the last file landed, coverage across src/ with all tests together was 99.1% of statements and 98.3% of branches. The owner then asked that anything short of 100% come with a reason the owner approves, so the agent replaced the percentage floor with a ledger: 235 places in the code that no test reaches, written up in 222 rows, each with a reason. As of 2026-10-13, none of the 222 rows had been approved.

The sandbox

A test that writes to the real ledgers, calls a real API or reads a real token is worse than no test before a refactor: it can fail for reasons that have nothing to do with the code, and it can post something. So every file under test/behavior-lock/ gets helpers/setup.ts, which Jest loads through setupFilesAfterEnv. The Jest docs describe those modules as code that runs "before each test file is executed". Our setup checks the test file's path first and leaves every other test file alone; 404 of our 1,173 test files sit outside the lock.

For a lock file, the setup creates a temporary directory, moves the working directory into it, points both HOME and os.homedir() at a home/ folder inside it, and empties the whole directory before every test. Code that reads a ledger like state/feedback.jsonl by relative path sees only what the test put there. File writes inside the repository or anywhere outside the OS temp directory are blocked, and so is starting a child process whose working directory is inside the repo. fetch, http and https are blocked except for localhost. Environment variables whose names look like secrets (TOKEN, PASSWORD, COOKIE and so on) are deleted, along with the ones that change depending on where the tests run, such as CI, GITHUB_* and CLAUDE_*. TZ is fixed to Asia/Tokyo.

Throwing an error isn't enough on its own, because the code under test may catch it and carry on. So every guard also records what it stopped, and the test fails at the end if anything was recorded:

afterEach(() => {
    jest.useRealTimers()
    const violations = [...isolationViolations]
    isolationViolations.length = 0
    if (violations.length > 0) {
        throw new Error(
            // message translated
            `behavior-lock: an operation broke the isolation rules\n- ${violations.join('\n- ')}`,
        )
    }
})

The CLIs run through helpers/run-command.ts. It writes input.json into the sandbox, sets process.argv as if pnpm had launched the command, collects stdout, stderr and console output, turns process.exit into a recorded exit code, and hands main to the same runMain wrapper production uses, so a failure goes through the real error logging and exit code 1. The commands run inside the test process rather than as child processes, because spawning a process for every CLI test would be slow and wouldn't count toward coverage.

The sandbox leaked while we built it

The setup file got its current shape one hole at a time. Six of them needed fixing before the lock held.

The first is recorded in the comment of the very first version, from 2026-10-08. A test that changed only HOME rewrote the real record our request-pacing gate keeps under ~/.cache, because os.homedir() didn't follow HOME. From then on both point into the sandbox.

On 2026-10-12 four more turned up in a single day. We found that inside a Jest test, process.env is a copy. Child processes started without an explicit env got the machine's real environment instead, so the home redirect and the secret stripping never reached the shell scripts and git calls under test. The setup now passes the test's environment to every child process call that doesn't bring its own. Seven minutes later came the next fix: variables like GITHUB_ACTIONS and CLAUDE_CODE_SESSION_ID change which code path runs, and a session cookie and some signup-email values had slipped past the secret pattern; both are now stripped. That evening a pre-commit review pointed out that blocking fetch only covers the test process: curl, git and pnpm started as child processes could still reach the internet. Child processes now get every proxy variable, in upper and lower case (http_proxy, HTTPS_PROXY, ALL_PROXY and the rest), pointed at 127.0.0.1:9, a local port nothing answers on, with localhost let through by NO_PROXY. They also get GIT_SSH_COMMAND=false, so git has no working command to open an SSH connection with.

The fourth, just before midnight, ran the other way. After a test file finishes, Jest restores every spy a test left in place to the value it had when the spy was created (in jest-runtime 30.4.2, the runtime's teardown() calls restoreAllMocks()), and for the functions we guard, that value was our guard. fs, child_process, os and http are shared with the next test file in the same worker, so older tests outside the lock started tripping our guard against child processes inside the repository, but only on a full run, and only in some orders. Each guard is now wrapped so that it steps aside once its file's isolation has ended:

export function delegateToOriginalAfterIsolation({
    guard,
    original,
    isActive,
}: {
    guard: ShapeAgnosticFunction
    original: ShapeAgnosticFunction
    isActive: () => boolean
}): ShapeAgnosticFunction {
    return function guardWhileIsolated(
        this: unknown,
        ...functionArguments: unknown[]
    ) {
        return Reflect.apply(isActive() ? guard : original, this, functionArguments)
    }
}

The last one showed up in CI. The run for the push that landed the lock failed three lock tests that had passed locally. CI runs in UTC, and setting TZ inside the setup file hadn't changed how Date read times, for the same reason as before: the test's process.env is a copy. TZ moved into jest.config.js, which the worker processes inherit. The other two failures came from Node versions: whether an aborted request's DOMException counted as an Error inside Jest differed between Node 22.17 on CI and 22.22 locally, which changed whether the message started with TimeoutError: or AbortError:. Those two tests now accept both forms.

What the lock has caught

Six commits so far have changed a pinned expectation because the code changed.

The one we would point to first is lock test [050]. It covers the command that sends replies to direct messages, and its title described what the code did in one sentence:

Every reply to a direct message is supposed to have the owner's approval. A message the agent had once marked as "won't reply" was left out of all three pre-send checks, so a reply written to it later would go out unapproved. The commit that fixed it counted 5 rows in the live DM ledger in exactly that state. The owner ruled that those replies need approval too, and the fix rewrote [050] to expect the approval gate to stop the send, then added 059 and 060.

The same night, two more bugs found while writing the tests were fixed: the weekly report crashed with a TypeError when the year-end goals had no dev.to follower metric ([081]), and a generated label was missing a space ([034]). A separate session also fixed three bugs that night, and the lock flagged them: 9 existing assertions in the tests for the command that restarts missed scheduled posts now expect a gh api call that reads the workflow's state before anything else, and case [045] in the engagement planner went from "counts likes, reposts, quotes and replies as 0" to "returns the reason and records nothing". That second change was one more bug the lock had been holding in place: counting a week it couldn't fetch as zero could record a false "we're on track".

The fourth commit is the one that worries us about the refactor ahead. The agent moved the "was this file launched directly?" check of all 230 CLIs into one helper. Nothing a user could see changed, and expectations in 69 lock files still had to change, because each of those files has a case asserting which internal function the CLI hands itself to when it starts. That is wiring, not behavior, and the lock can't tell the difference.

The last two were small and deliberate. A change that stops goal tracking from nagging about metrics it can't measure while one of the agent's outside accounts is suspended added a field to an input object; the lock update was 2 lines. Then on 2026-10-13 the agent widened a hook that stops bare open calls on URLs, after a loop of bare open calls sent 9 links to the wrong Chrome profile for the third time. Six cases moved from "allowed" to "blocked": open -a Safari https://x, open --new https://x, open x.HTML, open report.HTML, open "/tmp/a b.html" and OPEN https://x. Each moved case carries a comment saying it was moved on purpose on that date, and why.

Two more commits changed lock tests without the code under test changing: the CI fix above, and one where a test [007] demanded at least 6 lock re-acquisitions within 1.5 seconds and didn't get there during a full parallel run. It now waits 5 seconds and asks for at least 2.

What it costs

We ran the whole lock once for this post:

That is 8 minutes 44 seconds of Jest time on a 10-core machine that other sessions were keeping busy: the one-minute load average read 97 when the run started and 92 when it ended, and the five-minute average reached 136. Treat it as an upper bound rather than a benchmark; we don't have a quiet-machine number. 510 of the 769 files finished in under 5 seconds and 16 took more than 30. The slowest file took 133 seconds. It locks the commit gate that checks each scheduled GitHub Actions workflow does exactly one task, and it simulates eight days of schedules minute by minute.

The other costs are the ones above: half a million lines of test code that someone has to read when an expectation changes, a sandbox that needed six fixes before it held, and tests that pin wiring as firmly as behavior.

What we don't know yet

The refactor itself hasn't started. Since the lock landed, two commits have touched src/, both small fixes for while an outside account is suspended, and one of them is the goal-tracking change above. The question we can't answer yet is how many of the 25,396 tests assert wiring rather than behavior. The 69-file commit suggests the answer isn't small: a change that nobody outside the code would notice still moved 69 files. Under our rule, every changed expectation needs a reason in the commit message, and a domain-driven restructuring moves a lot of wiring. We may have to split the wiring cases out before the refactor, so that a failing lock test means what it is supposed to mean.

If you have locked a codebase's behavior before a big refactor, we'd like to know how it went. Did your expected values come from the code or from the spec? How did you stop the tests from pinning which function calls which, and was it worth separating those cases from the behavioral ones? And how did you keep tests from touching anything real? Tell us in the comments below; we'll answer each one there.

Find broken links on any website with Python, grouped by the page you need to fix (2026)

Datawright Labs — Wed, 21 Oct 2026 02:16:10 +0000

A broken-link report is only useful if it tells you where to go to fix things. "This URL returns 404" is half the answer. "These three pages link to it" is the part you can act on. Most quick scripts check one page at a time and lose that mapping, or they check the same footer link 50 times.

This post builds a small crawler that does it properly: crawl the site, collect every outgoing URL once, check each one once, and write a CSV that maps every broken URL back to the pages that contain it. Then it looks at a real run, because the interesting part of link checking is the false positives.

The approach

Breadth-first crawl of same-site pages. Start at the homepage, follow <a href> links that stay on the same host, stop at a page limit.
Collect every outgoing URL with its source pages. Links, images, scripts and stylesheets all count. A missing image or script is as broken as a dead link. Strip #fragments so /docs#install and /docs#usage are one check.
Check each unique URL once, in parallel. Use GET with stream=True instead of HEAD. Plenty of servers answer HEAD with 405 or 404 even when the page is fine, and some just hang on it. A streamed GET reads only the headers and closes the connection.
Retry once before calling something broken. Transient timeouts are common.

The script

# Usage: python3 broken_links.py https://example.com [max_pages]
# Crawls same-site pages (BFS), checks every link and image once, writes broken.csv grouped by source page.
import csv, sys, time, collections
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urljoin, urldefrag, urlparse
import requests
from bs4 import BeautifulSoup

start = sys.argv[1]
max_pages = int(sys.argv[2]) if len(sys.argv) > 2 else 50
host = urlparse(start).netloc
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (compatible; link-check-tutorial/1.0)"

def check(url):
    # GET with stream=True reads headers only; many servers answer HEAD wrongly (405/404) or hang.
    for attempt in range(2):
        try:
            r = s.get(url, stream=True, timeout=15, allow_redirects=True)
            r.close()
            return r.status_code, r.url
        except requests.RequestException as e:
            err = type(e).__name__
            time.sleep(2)
    return err, url

sources = collections.defaultdict(set)   # link -> pages that contain it
queue, seen_pages, i = [start], {start}, 0
t = time.time()
while i < len(queue) and i < max_pages:
    page = queue[i]; i += 1
    try:
        r = s.get(page, timeout=20)
    except requests.RequestException:
        continue
    if "text/html" not in r.headers.get("content-type", ""):
        continue
    soup = BeautifulSoup(r.text, "html.parser")
    for tag, attr in (("a", "href"), ("img", "src"), ("link", "href"), ("script", "src")):
        for el in soup.find_all(tag):
            raw = el.get(attr)
            if not raw or raw.startswith(("mailto:", "tel:", "javascript:", "data:", "#")):
                continue
            link = urldefrag(urljoin(r.url, raw))[0]
            if not link.startswith("http"):
                continue
            sources[link].add(page)
            if tag == "a" and urlparse(link).netloc == host and link not in seen_pages:
                seen_pages.add(link); queue.append(link)

links = sorted(sources)
with ThreadPoolExecutor(10) as pool:
    results = dict(zip(links, pool.map(check, links)))

broken = [(u, st) for u, (st, _) in results.items() if not (isinstance(st, int) and st < 400)]
with open("broken.csv", "w", newline="") as f:
    w = csv.writer(f); w.writerow(["source_page", "broken_url", "status"])
    for u, st in sorted(broken):
        for p in sorted(sources[u]):
            w.writerow([p, u, st])
print(f"{i} pages crawled, {len(links)} unique URLs checked, {len(broken)} broken in {time.time() - t:.0f} s -> broken.csv")
for u, st in sorted(broken, key=lambda x: str(x[1]))[:15]:
    print(f"{str(st):<16} {u}  (on {len(sources[u])} page(s))")

Install the two dependencies with pip install requests beautifulsoup4 and run python3 broken_links.py https://your-site.com 200.

Results on crawlee.dev

I ran python3 broken_links.py https://crawlee.dev 50 on October 20, 2026:

50 pages crawled, 3,008 unique URLs checked in 245 seconds, 10 checks in parallel.
35 URLs flagged. That number is misleading, which is the main lesson of this post.

Here is what the 35 actually were:

Status	Count	Really broken?
403	14	No. Stack Overflow, npm, Crunchbase, Upwork, OpenAI and others block scripted requests. The Stack Overflow tag link alone appears on 37 pages.
999	2	No. LinkedIn's "go away, bot" status.
Connect timeout	7	No. All were plain `http://` links, and my test machine's network blocks outbound port 80. Worth knowing if you run this from a locked-down server or CI runner.
404	8	7 yes, 1 no (see below).
SSL error	4	Yes.

So 11 of the 35 were genuinely broken, and every one of them is a real typo or a bad link:

https://www.crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator on a blog post, with "fingeprint" missing an "r".
Three links like https://github.com//github.com/apify/crawlee/pull/1890/..., where a full URL got pasted into a field that already prefixed https://github.com/.
https://www.en.wikipedia.org/wiki/Robots.txt and https://www.developer.mozilla.org/...: an extra www. in front of hosts that don't have one, so the TLS certificate doesn't match.
A github.com/renovate%5Bbot%5D profile link (bot accounts have no profile page) and two SwaggerHub API docs that no longer exist.

The one 404 that wasn't broken was https://5JC94MPMLY-dsn.algolia.net, a <link rel="preconnect"> to the search provider. Its root path 404s, but nobody ever navigates there.

Things that will bite you

Treat 401, 403, 429 and 999 as "blocked", not "broken". Report them in a separate column, or re-check them from a real browser. If you count them as broken, your report is mostly noise and people stop reading it.
Skip preconnect and dns-prefetch links, or at least don't treat them as pages. They point at hosts, not documents.
Check from a normal network. Corporate proxies and CI runners often block plain HTTP or whole domains, which shows up as timeouts.
Be polite on your own site. This script checks external URLs 10 at a time but has no per-host limit. On a big site most links point at your own host, so add a per-host concurrency cap and respect robots.txt before you crawl thousands of pages.
Watch the redirect chains. allow_redirects=True hides a 301 to a 404. The final status is what matters for users, but r.history tells you which links are worth updating to skip the redirect.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: Broken Link Checker. It adds robots.txt handling, a per-host concurrency limit, a spending cap per run, and source-page grouping for every broken link, image and script. It costs $0.50 per 1,000 URLs checked, so Apify's free monthly credit covers small and medium sites, and you can run it on a schedule or call it by API. The script above is the whole idea, though, so use whichever fits.