When Your AI API Keys Die at 2 AM: A Recovery Playbook

It is 2 AM. You send a message to your AI agent. Nothing happens. You send another. Still nothing. You check the logs and see the same line repeated thirty times: Invalid token.

I have been there more than once. The good news: the fix is almost always the same. The bad news: the fix is almost always a manual one, and the sooner you stop panicking and start running a checklist, the sooner your system is back online.

This is the checklist I run. It has not failed me yet.

The Error You Will See

The exact wording varies by provider, but the family of errors looks like this:

{
  "error": {
    "code": "",
    "message": "Invalid token",
    "type": "new_api_error"
  }
}

It might say "Invalid API key," "Authentication failed," or "401 Unauthorized." It is always the same underlying problem: the string the server received in the Authorization header does not match a key it has on file.

What Killed the Key (Probably)

When a key that worked yesterday fails today, the cause is one of four things, in order of likelihood:

Expiration. Most providers put a hard expiry on keys, even if they do not show it in the dashboard. If you generated a batch of keys during a setup binge three months ago, several of them have probably already expired without warning.

Revocation. Some providers will revoke keys if the associated account has a billing issue, a usage violation, or a suspicious activity flag. The dashboard will not necessarily tell you this; the next request just fails.

Account problems. If your account is suspended, in arrears, or under review, all keys tied to it stop working. The error looks identical to a key problem, but the fix is at the account level.

Format change. Rare, but it happens. A provider might switch from bearer tokens to signed headers, or change the prefix from sk- to something else. The fix is to read the provider's recent changelog.

The First Five Minutes: Diagnose

SSH into the server where the agent stack runs and source the env file so your shell has the same variables the daemon does.

ssh -i "<your-ssh-key>" root@<your-vps-ip>
source /root/.openfang/.env

Then make a single direct call to the provider's API. The curl tells you, in one second, whether the key itself is dead or whether the problem is downstream in the agent stack.

curl -X POST https://<your-provider-endpoint>/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/kimi-k2.6",
    "messages": [{"role": "user", "content": "Test"}],
    "max_tokens": 50
  }'

If you get a 200 with a real completion, the key is fine and the problem is somewhere in your agent code. If you get a 401 with the same error your agent saw, the key is dead. Stop here. Everything else is wasted time until the key is replaced.

The Next Ten Minutes: Replace or Switch

You have two paths. The choice depends on whether the provider itself is healthy.

Path A: regenerate the key with the same provider. Log into the provider's dashboard, navigate to API keys, and either regenerate the failing key or create a new one. Copy the new value, paste it into your env file, and restart the service.

# Back up the current env first
cp /root/.openfang/.env /root/.openfang/.env.backup.$(date +%Y%m%d)

# Edit the env file
nano /root/.openfang/.env

# Replace the old key with the new one
export OPENAI_API_KEY=<your-new-key>

# Restart
systemctl restart openfang.service

Path B: switch to a different provider entirely. If the primary is having a wider outage, or if you have been meaning to migrate off it, this is the moment. Update two things: the env file with the new key, and the config file with the new endpoint and model name.

For a direct switch to another OpenAI-compatible provider:

export OPENAI_API_KEY=<your-new-key>
export OPENAI_BASE_URL=https://<new-provider>/v1

And in your config:

[default_model]
api_key_env = "OPENAI_API_KEY"
base_url = "https://<new-provider>/v1"
model = "<new-model>"
provider = "openai"

Then restart and re-run the curl smoke test.

The Next Five Minutes: Verify End to End

A working curl is not the same as a working agent. The agent stack has more failure modes (MCP servers, channel adapters, memory loads) that a single API call does not exercise.

Check the daemon status:

/root/.openfang/bin/openfang-wrapper status

Check that all expected agents are listed and healthy.

Check the channel adapter (Telegram, in my case):

/root/.openfang/bin/openfang-wrapper channel test telegram

Then send a real message from the channel end (Telegram app, Slack DM, whatever you use) and confirm the agent responds within ten seconds.

The Next Ten Minutes: Document It

The temptation at 2:30 AM is to go to bed. The discipline is to write three lines in your incident log before you do.

1. What failed and when you noticed.
2. What you changed.
3. What the root cause was, if you know it.

A log entry that says "BluesMinds key expired 2026-05-24, switched to OpenAgentic, no data loss" is enough. When the next key expires, you will read that line and skip the panic phase entirely.

Local Models as the Always-On Fallback

If you are running on a box that has a few GB of RAM to spare, installing Ollama and pulling a small model gives you a fallback that cannot be revoked by an upstream provider. It is not as smart as a frontier model, but it is always available.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1

Then point your config at the local endpoint:

[default_model]
provider = "ollama"
base_url = "http://localhost:11434"
model = "llama3.1"

I do not run Ollama on my 1GB VPS; the memory budget is too tight. But I keep a laptop with Ollama ready as a manual failover target. If both my primary and fallback providers died simultaneously, I would ssh-tunnel the laptop into the VPS and repoint the agent stack in five minutes.

What I Have Learned From Three 2 AM Pages

The fix is never the cause. If a key expired, no amount of restarting the daemon will help. Diagnose first, then act.

The smoke test is non-negotiable. Every recovery, I run the curl. Every time, the curl tells me something I would have missed otherwise.

The fallback is only a fallback if you test it. I had a configured Sumopod fallback that I had never actually called. When I finally needed it, the base URL had changed, the model name was different, and the auth format was bearer-prefixed differently. I lost forty minutes rediscovering what a single test a month earlier would have caught.

The incident log is worth more than the fix. A bad night becomes a useful system only if you write down what happened before you forget.

If you are running any kind of agent stack on top of a paid API, you will hit this problem eventually. Build the checklist now, while you are calm. Run it once on a Sunday afternoon to confirm each step works. Then the next time the dashboard says Invalid token at 2 AM, you will be back in bed in fifteen minutes instead of staring at logs until sunrise.