File-Drop Connector

How it works

The file-drop connector is the /connector binary in the self-hosted image. It watches directories you own, converts each file it finds with the same engine as POST /v1/convert (in-process, no HTTP and no API key), and moves the input along a fixed lifecycle:

in/  ──► read into memory ──► convert
     ├─ success ──► out/<base>.<ext> + out/<name>.warnings.json ──► move input to archive/
     └─ failure ──► error/<name>.reason.json ──► move input to error/

Files are processed in memory and moved between your own directories. Content never enters Postgres, Redis or logs. The connector never deletes or expires anything in archive/ or error/; retention there is yours to manage. The sidecar files are metadata only: warning codes, paths, sizes and durations, never an echo of the message.

This page covers local and SFTP directories. For buckets, see Object stores; for notifications, see Queues and notifications.

Configure watches

The connector reads one JSON document from the path in TRANSMUTE_CONNECTOR_CONFIG. The whole document is validated at startup; any invalid watch stops the connector.

{
  "watches": [
    {
      "name": "statements",
      "backend": "local",
      "in": "/data/in",
      "out": "/data/out",
      "archive": "/data/archive",
      "error": "/data/error",
      "from": "mt940",
      "to": "camt.053",
      "options": { "srVersion": "SR2025", "translitProfile": "base" },
      "pollInterval": "5s",
      "tenant": "11111111-1111-1111-1111-111111111111"
    }
  ]
}

SFTP

Set "backend": "sftp" and add an sftp block. The four directories are then paths on the remote server.

{
  "name": "remote-statements",
  "backend": "sftp",
  "in": "/upload/in", "out": "/upload/out",
  "archive": "/upload/archive", "error": "/upload/error",
  "from": "mt940", "to": "camt.053",
  "sftp": {
    "host": "files.example.test",
    "port": 22,
    "user": "transmute",
    "knownHosts": "/etc/transmute/known_hosts",
    "keyFile": "/etc/transmute/id_ed25519"
  }
}

The connector opens one SFTP connection per watch.

Pickup, ordering and crashes

• A file is picked up only once its size has stayed the same across consecutive polls, so a producer still writing is never read half-way. Expect a result within about two poll intervals.
• Files whose names start with . are never picked up. If your producer cannot write atomically, upload under a hidden name and rename into in/.
• Within a poll, files are processed in name order.
• The move out of in/ is the commit point and happens only after outputs and sidecars are fully written (outputs go to a temporary name first). If the connector stops at any moment, every file is either fully processed or still intact in in/, and the next run processes it again.
• On SIGINT/SIGTERM the connector finishes the file in progress, leaves the rest untouched and exits.

Outputs and sidecars

A successful conversion writes out/<base>.<ext>, where <base> is the input name without its extension, plus out/<input name>.warnings.json:

{
  "warnings": [],
  "meta": {
    "detectedType": "mt940", "targetType": "camt.053",
    "pages": 1, "sizeBytes": 278, "durationMs": 3
  }
}

A conversion that produces several outputs (paged MT statements, multi-statement files) writes <base>-1.<ext>, <base>-2.<ext> and so on. The extension comes from the target format: .xml for MX targets, .csv for csv-xero, .ofx, .qfx and .qbo for those targets, .json for accounting-transaction, and .txt for everything else (MT targets, and currently also bai2, csv-generic, csv-quickbooks and csv-sage).

A failure moves the input to error/ and writes error/<input name>.reason.json with the same problem object the API returns:

{
  "problem": {
    "type": "https://docs.transmute.403fin.io/errors/invalid-input",
    "title": "Invalid input",
    "status": 400,
    "detail": "Could not detect message format: …"
  },
  "sizeBytes": 20,
  "durationMs": 0
}

detail gives lengths, counts and positions, never your message content. To retry, fix the file and move it back into in/.

Metering

In metered mode the connector checks at startup that every configured tenant exists, so a mistyped UUID stops it rather than metering into nothing. A watch without tenant still runs but is not metered (warned once at startup). Connector usage is recorded for your own reporting; it is not counted against API keys.

Environment

Secrets are always passed by environment variable name or key-file path, never written in the watches document.

Running it

Docker Compose. Run the same image with the /connector entrypoint and mount your directories:

services:
  connector:
    image: ghcr.io/403ent/transmute:<release>
    entrypoint: ["/connector"]
    environment:
      TRANSMUTE_CONNECTOR_CONFIG: /etc/transmute/connector.json
    volumes:
      - /opt/transmute/connector.json:/etc/transmute/connector.json:ro
      - /srv/transmute-data:/data

Helm. Set connector.enabled: true and put the watches document under connector.config (or point connector.existingConfigMap at a ConfigMap with key config.json). The chart mounts it at /etc/transmute/connector/config.json and the data volume at connector.dataMountPath (default /data), backed by a PVC (connector.persistence.*), an existing claim or a hostPath. connector.metered (default true) passes DATABASE_URL when externalDatabase.existingSecret is set. Supply credential variables through connector.extraEnv with secretKeyRef.

connector.replicas must be 1: two connectors polling the same directories would race for the same files, and the chart refuses to render any other value.