Skip to main content
This page lists the issues integrators most often encounter and how to resolve each one.

​
Service won’t start

The service refuses to start without database credentials.Fix: set CUSTODY_DB_PASSWORD, or enable CUSTODY_DB_IAM_AUTH=true for AWS RDS IAM authentication.
The Quantum Chain RPC URL is missing.Fix: set CUSTODY_NODE_RPC_URL to a reachable HTTP endpoint (e.g. https://api.quantumapi.io/v1/rpc or http://localhost:8545).
You configured CUSTODY_SIGNER_TYPE=grpc but did not provide a target.Fix: set CUSTODY_SIGNER_GRPC_TARGET=host:port or switch to callback mode for development.
The encryption key is the wrong length.Fix: generate exactly 32 bytes and hex-encode them: openssl rand -hex 32.

​
Transactions stuck in PENDING_SIGNATURE

The most common cause: your external signer never delivered a signature.
  1. Inspect the transaction: 
    curl "$BASE_URL/v1/transactions/{id}" -H "Authorization: Bearer $API_KEY"
    Check signing_expires_at. If it has passed, the signing window expired.
  2. Verify your signer is reachable. For gRPC mode, the API logs signer.unreachable and triggers the circuit breaker.
  3. Confirm the wallet’s registered public key matches the key your signer is using. A mismatch produces signature verification failures, transitioning the transaction to FAILED.
  4. If the signer recovered, cancel the stale transaction and resubmit: 
    curl -X POST "$BASE_URL/v1/transactions/{id}/cancel" \
  -H "Authorization: Bearer $API_KEY"

​
Transactions stuck in PENDING_AML_SCREENING

The compliance provider has not returned a result.
  • For ShuftiPro, results are delivered async to your CUSTODY_SHUFTIPRO_CALLBACK_URL. Ensure the URL is reachable from the public internet.
  • Inspect the screening: 
    curl "$BASE_URL/v1/compliance/transactions/{txId}/screening" \
  -H "Authorization: Bearer $API_KEY"
  • If the provider is down, the circuit breaker opens after repeated failures. Set CUSTODY_COMPLIANCE_PROVIDER=noop to bypass screening temporarily (development only).

​
Webhook deliveries failing

HTTP 410 Gone is treated as a permanent removal signal. The endpoint will be marked DISABLED and no further deliveries are attempted.Fix: create a new endpoint with POST /v1/webhooks and update your subscriber.
Common causes:
  • Using the formatted body instead of the raw bytes.
  • Forgetting the timestamp + "." prefix in the HMAC input.
  • Using the wrong webhook secret (each endpoint has its own).
Fix: see Webhooks for the canonical signing formula and constant-time comparison snippets.
After 5 failed retries, the delivery moves to dead-letter status.Fix: inspect with GET /v1/webhooks/{id}/deliveries and replay successful ones with POST /v1/webhooks/{id}/replay/{deliveryId}.

​
Nonce conflicts

Symptoms: transactions transition to FAILED with nonce too low or replacement transaction underpriced.
  • Qustody assigns nonces sequentially per wallet. If you also send transactions from the same address through another path, you can collide.
  • Fix: route all outbound transfers through Qustody for a given wallet, or wait for outstanding transactions to confirm before issuing new ones.

​
Circuit breaker open

The signer or AML provider triggered the circuit breaker. New transactions queue while the breaker is half-open.
  • Check the signer_circuit_breaker_state Prometheus metric.
  • Investigate the provider; the breaker self-heals after the cool-down window.
  • For long outages, switch the affected component to a fallback (e.g. set a backup CUSTODY_NODE_RPC_URL via CUSTODY_NODE_BACKUP_RPC_URLS).

​
Database connection saturation

If you see pq: too many connections for role, raise the pool limits or scale Postgres: 
CUSTODY_DB_MAX_OPEN_CONNS=50
CUSTODY_DB_MAX_IDLE_CONNS=10
CUSTODY_DB_CONN_MAX_LIFETIME=10m
For Aurora, route reads to the reader endpoint with CUSTODY_DB_READ_REPLICA_ENABLED=true.

​
Authentication failures

HTTPSymptomCause
401 UnauthorizedMissing or wrong tokenBearer header missing, expired SSO access token
403 ForbiddenAuthenticated but disallowedRBAC permission missing for the resource+action
409 ConflictTransaction already in another stateE.g. trying to sign a BROADCASTING transaction
429 Too Many RequestsRate limitedLower request rate or raise CUSTODY_RATE_LIMIT_RPM
For SSO token issues, exchange the refresh token: POST /v1/auth/refresh.

​
Where to look

SourceWhat it tells you
GET /readyzWhether DB and signer are healthy
:9090/metricsPer-state queue depth, signer latency, breaker state
Audit log (GET /v1/audit)Who did what, when
Tracing (OTLP)Per-request waterfall when CUSTODY_OTEL_ENABLED=true
Server logsStructured JSON; filter by request_id
Webhook deliveries (GET /v1/webhooks/{id}/deliveries)Per-delivery status and response codes