Object Store Watches

Prefixes instead of directories

A watch with "backend": "s3" or "backend": "azblob" polls a bucket instead of a directory. Everything else about the file-drop connector applies: the same watch fields, conversion options, sidecars, metering and single-replica rule.

The four lifecycle directories become key prefixes, relative to the block's prefix. With "in": "in" and "prefix": "acme/drop", the connector watches acme/drop/in/. Only objects directly under in/ are picked up; deeper "subfolders" are ignored.

S3-compatible stores

{
  "name": "s3-statements",
  "backend": "s3",
  "in": "in", "out": "out", "archive": "archive", "error": "error",
  "from": "mt940", "to": "camt.053",
  "s3": {
    "region": "eu-west-1",
    "bucket": "acme-drop",
    "prefix": "statements",
    "credentials": {
      "accessKeyIdEnv": "S3_ACCESS_KEY_ID",
      "secretAccessKeyEnv": "S3_SECRET_ACCESS_KEY"
    }
  }
}

s3 block fields: region and bucket (required), prefix, endpoint, pathStyle, allowInsecure and credentials.

Credentials

credentials takes one of two forms:

• Keys by environment variable name. accessKeyIdEnv and secretAccessKeyEnv (plus sessionTokenEnv for temporary credentials) name environment variables that hold the values. The connector reads them at startup.
• Web identity. "webIdentity": true exchanges the workload's projected OIDC token (AWS_WEB_IDENTITY_TOKEN_FILE) for temporary credentials with STS AssumeRoleWithWebIdentity. The role comes from AWS_ROLE_ARN or the block's roleArn; stsEndpoint overrides the STS endpoint. This is the EKS IRSA contract. Temporary credentials stay in memory and are renewed before they expire.

The role or key needs s3:ListBucket on the bucket (for the prefix), and s3:GetObject, s3:PutObject and s3:DeleteObject on the prefix's objects.

Azure Blob

{
  "name": "azure-statements",
  "backend": "azblob",
  "in": "in", "out": "out", "archive": "archive", "error": "error",
  "from": "mt940", "to": "camt.053",
  "azblob": {
    "account": "acmestorage",
    "container": "drop",
    "prefix": "statements",
    "sasTokenEnv": "AZ_SAS_TOKEN"
  }
}

Commit ordering and reprocessing

Object stores have no atomic rename, so the connector commits each file in this order:

1. Write out. Each output, and the .warnings.json sidecar (or .reason.json in error/ on failure), is written with a single PUT. A PUT is atomic per object, so there are no partial or temporary objects.
2. Notify (only if the watch has a sink).
3. Copy the input to archive/ (or error/). On Azure, the connector waits until the server-side copy reports success.
4. Delete the input from in/.

If the connector stops between the copy and the delete, the input is still in in/. The next poll processes it again, rewrites the same output bytes and the same archive copy, and completes the delete. Nothing is lost or half-written. The only trace is a second processing, which also means a second usage row and, with a sink, a second notification. Design downstream consumers to tolerate seeing the same file twice.

Limits

• Inputs are read in one request, fully in memory, and capped at 16 MiB. A larger object is moved to error/ with a message-too-large reason instead of being retried forever. It is never converted and never billed.
• Each output PUT is capped at 128 MiB.

Network rules

• Endpoints must be https with TLS 1.2 or newer, unless the block sets "allowInsecure": true.
• Every endpoint, including STS, is checked at startup and on every connection: loopback and private addresses are refused unless allowInsecure is set, and link-local addresses (including cloud instance metadata) are refused even then.
• Use allowInsecure only for a lab MinIO or a local Azurite.
• Redirects are never followed. Retries happen only on 5xx and 429 responses, with jitter, up to four attempts. There is no proxy support.
• Only the endpoints you configure are contacted (plus the regional STS endpoint for web identity).

Secrets

Credentials appear in the watches document only as environment variable names. A value that looks like a key itself (an AKIA… or ASIA… access key id, or anything containing / or =) is refused at startup without being echoed. A referenced variable that is unset or empty stops the connector at startup. Logs and errors carry operation names, HTTP statuses and provider error codes only; signed and SAS URLs are never logged.

On Helm, supply the variables through connector.extraEnv from a Secret, or use IRSA by setting connector.serviceAccountName to an annotated service account. Object-store watches need no data volume, so you can set connector.persistence.enabled: false:

connector:
  enabled: true
  serviceAccountName: transmute-connector   # annotated with an IAM role
  persistence:
    enabled: false
  extraEnv:
    - name: AZ_SAS_TOKEN
      valueFrom: {secretKeyRef: {name: transmute-azblob, key: sas-token}}
  config:
    watches:
      - name: s3-statements
        backend: s3
        in: in
        out: out
        archive: archive
        error: error
        from: mt940
        to: camt.053
        s3:
          region: eu-west-1
          bucket: acme-drop
          credentials: {webIdentity: true}