> ## Documentation Index
> Fetch the complete documentation index at: https://docs.quantumapi.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common failure modes in Qustody and how to recover.

This page lists the issues integrators most often encounter and how to resolve each one.

## Service won't start

<AccordionGroup>
  <Accordion title="CUSTODY_DB_PASSWORD is required">
    The service refuses to start without database credentials.

    **Fix:** set `CUSTODY_DB_PASSWORD`, or enable `CUSTODY_DB_IAM_AUTH=true` for AWS RDS IAM authentication.
  </Accordion>

  <Accordion title="CUSTODY_NODE_RPC_URL is required">
    The Quantum Chain RPC URL is missing.

    **Fix:** set `CUSTODY_NODE_RPC_URL` to a reachable HTTP endpoint (e.g. `https://api.quantumapi.io/v1/rpc` or `http://localhost:8545`).
  </Accordion>

  <Accordion title="CUSTODY_SIGNER_GRPC_TARGET is required when signer type is grpc">
    You configured `CUSTODY_SIGNER_TYPE=grpc` but did not provide a target.

    **Fix:** set `CUSTODY_SIGNER_GRPC_TARGET=host:port` or switch to `callback` mode for development.
  </Accordion>

  <Accordion title="CUSTODY_WEBHOOK_ENCRYPTION_KEY must be 64 hex characters">
    The encryption key is the wrong length.

    **Fix:** generate exactly 32 bytes and hex-encode them: `openssl rand -hex 32`.
  </Accordion>
</AccordionGroup>

## Transactions stuck in PENDING\_SIGNATURE

The most common cause: your external signer never delivered a signature.

1. Inspect the transaction:
   ```bash theme={null}
   curl "$BASE_URL/v1/transactions/{id}" -H "Authorization: Bearer $API_KEY"
   ```
   Check `signing_expires_at`. If it has passed, the signing window expired.
2. Verify your signer is reachable. For gRPC mode, the API logs `signer.unreachable` and triggers the circuit breaker.
3. Confirm the wallet's registered public key matches the key your signer is using. A mismatch produces signature verification failures, transitioning the transaction to `FAILED`.
4. If the signer recovered, cancel the stale transaction and resubmit:
   ```bash theme={null}
   curl -X POST "$BASE_URL/v1/transactions/{id}/cancel" \
     -H "Authorization: Bearer $API_KEY"
   ```

## Transactions stuck in PENDING\_AML\_SCREENING

The compliance provider has not returned a result.

* For ShuftiPro, results are delivered async to your `CUSTODY_SHUFTIPRO_CALLBACK_URL`. Ensure the URL is reachable from the public internet.
* Inspect the screening:
  ```bash theme={null}
  curl "$BASE_URL/v1/compliance/transactions/{txId}/screening" \
    -H "Authorization: Bearer $API_KEY"
  ```
* If the provider is down, the circuit breaker opens after repeated failures. Set `CUSTODY_COMPLIANCE_PROVIDER=noop` to bypass screening temporarily (development only).

## Webhook deliveries failing

<AccordionGroup>
  <Accordion title="My endpoint returns 410 and deliveries stop">
    HTTP `410 Gone` is treated as a permanent removal signal. The endpoint will be marked `DISABLED` and no further deliveries are attempted.

    **Fix:** create a new endpoint with `POST /v1/webhooks` and update your subscriber.
  </Accordion>

  <Accordion title="Signatures don't verify">
    Common causes:

    * Using the formatted body instead of the raw bytes.
    * Forgetting the `timestamp + "."` prefix in the HMAC input.
    * Using the wrong webhook secret (each endpoint has its own).

    **Fix:** see [Webhooks](/concepts/webhooks) for the canonical signing formula and constant-time comparison snippets.
  </Accordion>

  <Accordion title="Deliveries land in dead-letter queue">
    After 5 failed retries, the delivery moves to dead-letter status.

    **Fix:** inspect with `GET /v1/webhooks/{id}/deliveries` and replay successful ones with `POST /v1/webhooks/{id}/replay/{deliveryId}`.
  </Accordion>
</AccordionGroup>

## Nonce conflicts

Symptoms: transactions transition to `FAILED` with `nonce too low` or `replacement transaction underpriced`.

* Qustody assigns nonces sequentially per wallet. If you also send transactions from the same address through another path, you can collide.
* **Fix:** route all outbound transfers through Qustody for a given wallet, or wait for outstanding transactions to confirm before issuing new ones.

## Circuit breaker open

The signer or AML provider triggered the circuit breaker. New transactions queue while the breaker is half-open.

* Check the `signer_circuit_breaker_state` Prometheus metric.
* Investigate the provider; the breaker self-heals after the cool-down window.
* For long outages, switch the affected component to a fallback (e.g. set a backup `CUSTODY_NODE_RPC_URL` via `CUSTODY_NODE_BACKUP_RPC_URLS`).

## Database connection saturation

If you see `pq: too many connections for role`, raise the pool limits or scale Postgres:

```bash theme={null}
CUSTODY_DB_MAX_OPEN_CONNS=50
CUSTODY_DB_MAX_IDLE_CONNS=10
CUSTODY_DB_CONN_MAX_LIFETIME=10m
```

For Aurora, route reads to the reader endpoint with `CUSTODY_DB_READ_REPLICA_ENABLED=true`.

## Authentication failures

| HTTP                    | Symptom                              | Cause                                                |
| ----------------------- | ------------------------------------ | ---------------------------------------------------- |
| `401 Unauthorized`      | Missing or wrong token               | Bearer header missing, expired SSO access token      |
| `403 Forbidden`         | Authenticated but disallowed         | RBAC permission missing for the resource+action      |
| `409 Conflict`          | Transaction already in another state | E.g. trying to sign a `BROADCASTING` transaction     |
| `429 Too Many Requests` | Rate limited                         | Lower request rate or raise `CUSTODY_RATE_LIMIT_RPM` |

For SSO token issues, exchange the refresh token: `POST /v1/auth/refresh`.

## Where to look

| Source                                                  | What it tells you                                      |
| ------------------------------------------------------- | ------------------------------------------------------ |
| `GET /readyz`                                           | Whether DB and signer are healthy                      |
| `:9090/metrics`                                         | Per-state queue depth, signer latency, breaker state   |
| Audit log (`GET /v1/audit`)                             | Who did what, when                                     |
| Tracing (OTLP)                                          | Per-request waterfall when `CUSTODY_OTEL_ENABLED=true` |
| Server logs                                             | Structured JSON; filter by `request_id`                |
| Webhook deliveries (`GET /v1/webhooks/{id}/deliveries`) | Per-delivery status and response codes                 |

## Related

* [Errors and codes](/qustody/errors)
* [Deployment](/qustody/deployment)
* [Security model](/qustody/security-model)