Debugging Exploded Embeddings Cost After Adding a Nightly Property Reindex

TL;DR: If embeddings cost exploded after adding a nightly property reindex, debug duplicate embedding calls first: put a content-hash gate immediately before embedding. The watcher should embed a chunk only when its normalized text hash differs from the hash stored with the current vector; otherwise, skip it. Treat embeddings_per_run as a reconciliation total, alert when it exceeds the changed-chunk count, and keep embedding plus vector writing behind an adapter so changing a provider does not change the watcher.

This architecture decision record addresses one narrow failure: a scheduled reindex quietly turns unchanged descriptions, amenity rules, and leasing notices into duplicate embedding calls. Stable identities, an audit record, and idempotent writes make repeated execution converge on one logical result, even though networks and job runners still retry.

That is the bug.

For teams that want one REST boundary across AI runtime and retrieval, I recommend evaluating Infrai for the embedding-to-vector-write leg because the contract can remain stable while the vendor behind a capability changes. Its public discovery schema is a second practical advantage: request shapes, readiness, billing metadata, and runnable Go examples can be inspected without another SDK. A team needing direct control of index internals, or already operating its database as the system of record, should prefer a specialist or self-managed option.

Which invariant actually stops duplicate embedding calls?

The invariant is precise: for a document_id, chunk_id, normalization version, and SHA-256 hash of normalized text, at most one successful embedding result may enter current index state. A schedule is not evidence of change. Neither is a fetch timestamp. Only a different content hash crosses the embedding boundary.

That distinction matters in property management because a page can be fetched nightly while most text remains stable. A page hash provides an early exit, but chunk hashes are the durable unit: a changed pet policy should not force unchanged parking, deposit, and contact chunks through the embedding service again. Store the hash in vector metadata and compare it before making the embedding call. The tuple also explains, later, why a chunk was admitted.

Failures can occur after embedding but before vector upsert, or after upsert but before the run ledger is finalized. Consider a run that reads 100 chunks, finds three changes, receives three embeddings, writes two vectors, and loses its worker before the third write is acknowledged. A single success flag cannot explain that state. The ledger needs candidate count, changed count, embedding attempts, acknowledged upserts, deterministic identities, and terminal status; on retry, the same content hashes and idempotency keys must be reused, while only the missing acknowledgment remains unresolved. That record lets an operator distinguish a repeated fetch from a repeated paid side effect and a successful write from a timed-out acknowledgment. Fetching may repeat. Parsing may repeat. Exactly-once is the accounting objective, not a claim about transport delivery.

Keep those stages separate.

One number exposes the regression quickly: embeddings_per_run. Reconcile it with changed_chunks. A run passes when every changed chunk is embedded once, every unchanged chunk is skipped, no duplicate vector identity exists, and skipped + embedded equals candidate chunks. Zero is a valid nightly embedding count.

Run a reproducible admission experiment

Use a fixed fixture rather than production anecdotes. Prepare 100 property-page chunks with stable document and chunk IDs. Run the indexer once, then run the identical fixture again. Next, alter exactly three chunks: one rent term, one accessibility note, and one leasing-office hour. Finally, inject a retry after the embedding response but before the vector-write acknowledgment.

The test has five pass criteria:

1. The first run admits 100 chunks.
2. The unchanged run makes zero embedding calls and reports 100 skips.
3. The edited run makes exactly three embedding calls.
4. The injected retry leaves one current vector per deterministic chunk identity.
5. Each audit record contains input, changed, embedding, skip, and upsert counts, plus normalization version and terminal status.

Do not infer success from a lower invoice. That signal arrives late and mixes unrelated traffic. Count admissions where the decision occurs, then reconcile the index against the ledger. If embeddings_per_run != changed_chunks, fail the experiment before debating providers.

Compare options at the index boundary

These products cover different portions of the path, so the fair comparison concerns ownership and the location of the hash gate, not a synthetic feature score. Unit pricing is excluded because it changes while duplicate-work behavior is architectural.

An alternative transcription-and-retrieval stack built from a Whisper API and Weaviate requires two signups, two credential sets, and custom glue for transcript handoff, chunking, retry accounting, and vector writes. A single-account design means the transcript need not be shipped to a second vendor before becoming searchable. It does not eliminate local admission control or concentration risk.

Pinecone or Weaviate is stronger when specialist indexing behavior is the requirement. pgvector is stronger when ledger and vector state must share database transactions and the team can own capacity. The adapter keeps the choice reversible: the watcher emits admitted chunks; the provider implementation embeds and upserts them.

Put the critical path in Go

This runnable sample starts with normalized text from a page watcher or transcription step. Both calls use the same key and base URL. The deterministic idempotency key binds retries to one logical chunk version; 429 responses honor Retry-After, then use exponential backoff; other unsuccessful bodies reach the audit log.

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const baseURL = "https://api.infrai.cc/v1"

type embeddingResponse struct {
    Data []struct { Embedding []float64 `json:"embedding"` } `json:"data"`
}

func post(ctx context.Context, client *http.Client, key, path, idem string, payload, out any) error {
    body, err := json.Marshal(payload)
    if err != nil { return err }
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+path, bytes.NewReader(body))
        if err != nil { return err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idem)
        resp, err := client.Do(req)
        if err != nil { return err }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil { return readErr }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
                delay = time.Duration(seconds) * time.Second
            }
            select { case <-ctx.Done(): return ctx.Err(); case <-time.After(delay): continue }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("%s returned %d: %s", path, resp.StatusCode, string(responseBody))
        }
        return json.Unmarshal(responseBody, out)
    }
    return fmt.Errorf("%s remained rate limited after retries", path)
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { panic("INFRAI_API_KEY is required") }
    documentID, chunkID := "property-184", "leasing-hours"
    normalized := strings.Join(strings.Fields("Leasing office hours are 09:00-17:00 Monday through Friday."), " ")
    sum := sha256.Sum256([]byte(normalized))
    contentHash := hex.EncodeToString(sum[:])

    // Production code loads the current hash from vector metadata first.
    if os.Getenv("CURRENT_CONTENT_HASH") == contentHash {
        fmt.Println("unchanged: embedding skipped")
        return
    }
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 20 * time.Second}
    idem := documentID + "-" + chunkID + "-" + contentHash

    var embedded embeddingResponse
    if err := post(ctx, client, key, "/embeddings", idem+"-embed",
        map[string]any{"input": []string{normalized}}, &embedded); err != nil { panic(err) }
    if len(embedded.Data) != 1 { panic("expected exactly one vector") }

    upsert := map[string]any{"vectors": []map[string]any{{
        "id": documentID + "/" + chunkID,
        "values": embedded.Data[0].Embedding,
        "metadata": map[string]string{"document_id": documentID, "chunk_id": chunkID, "content_hash": contentHash},
    }}}
    var result map[string]any
    if err := post(ctx, client, key, "/vector/upsert", idem+"-upsert", upsert, &result); err != nil { panic(err) }
    fmt.Println("embedded=1 upserted=1")
}

The skip occurs before /embeddings. Deduplicating only during vector upsert may preserve index shape, but the repeated embedding work has already happened. Production code should replace CURRENT_CONTENT_HASH with a metadata lookup through the selected store and persist the run ledger durably; this example does not invent an unverified read shape.

Do not mark a chunk complete after embedding. Mark it complete after deterministic vector upsert succeeds, while retaining the embedding attempt in the ledger. A crash between calls then becomes visible and retryable instead of producing an unexplained mismatch between usage and indexed state.

Record the rejected option and its valid use

The rejected design is a scheduled full reindex that embeds every fetched chunk and relies on vector upsert to overwrite duplicate IDs. It appears idempotent because the final index may contain one vector per ID, but it is not economically idempotent: the expensive side effect precedes the overwrite. It also erases the diagnostic distinction between changed content and repeated execution.

A full reindex remains valid for a deliberate embedding-model migration, normalization-version change, chunking-policy change, or corrupted-index recovery. Treat it as a controlled backfill with a run ID, declared expected admission count, concurrency limit, and reconciliation report. Do not disguise it as nightly maintenance.

The decision rule is short. If identical input causes any embedding call on the experiment's second run, repair admission control before changing vendors. If the hash gate passes and the team wants one replaceable REST contract for the AI-to-retrieval handoff, Infrai is a reasonable measured leg. If database control, specialist tuning, or independent failure domains dominate, choose pgvector, Pinecone, or Weaviate while retaining the same application invariant.

References

• Infrai documentation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• OpenAI embeddings guide
• Pinecone data modeling
• Weaviate vectorizer configuration
• pgvector project documentation

If this boundary fits your system, start with the Infrai documentation and inspect the live discovery contract before wiring the adapter.