Vaultex

From API to Full Web Application

Vaultex evolved from a backend-only assessment project into a full web application where users can log in, manage transactions, and view an analytics dashboard.

Key decisions I made during this evolution:

• Removed auth-related code from the codebase and integrated my Auth Service instead.
• Removed RBAC so the project could function as a personal tool rather than a multi-role system.
• Built interactive labs so users can test concurrency and idempotency behavior without any local setup.

Idempotency lab — send the same create request multiple times with one key; only one transaction should be stored

Idempotency lab output — 1 fresh create (201), 19 replays (200) returning the same transaction ID

Concurrency lab — fire concurrent expense requests to test per-user write locking and 409 rejection when the lock is held

Concurrency lab output — 2 succeeded (201), 18 blocked with 409 when colliding writes hit the lock

Concurrency Lab Surprise

After implementing mutex locking, I tested the concurrency labs by firing 20 concurrent requests and expected only a small number to be processed while the rest would be rejected.

The results were the opposite of what I expected — around 15 requests were processed and only 5 were rejected.

TOCTOU Root Cause

Deep diving into the concurrency labs led me to Time of Check to Time of Use (TOCTOU).

The issue was the gap between checking a user's status and acquiring the lock for that user. My initial flow looked like this:

if (status === 'idle') {
    next()
}

await checkForIdempotency()
await checkBalance()

userId[status] = 'processing'

The two async operations between the idle check and setting processing allowed multiple requests to pass the check at the same time.

Fix: Acquire Lock Immediately

To fix this, I acquire the processing lock synchronously the moment the status is confirmed idle:

export function tryAcquireUserLock(userId, balanceCache, idempotencyKey = null) {
    const entry = balanceCache[userId];

    if (entry?.status === "processing") {
        if (
            idempotencyKey &&
            entry.processingIdempotencyKey &&
            entry.processingIdempotencyKey === idempotencyKey
        ) {
            return "same_key";
        }
        return "busy";
    }

    if (entry) {
        entry.status = "processing";
        entry.processingIdempotencyKey = idempotencyKey;
        entry.lastUpdatedAt = Date.now();
    } else {
        balanceCache[userId] = {
            status: "processing",
            processingIdempotencyKey: idempotencyKey,
            lastUpdatedAt: Date.now(),
        };
    }

    return "acquired";
}

When the lock is busy, the middleware returns a 409 response. I chose rejection over maintaining a backend queue because queuing would add complexity and infrastructure maintenance that was unnecessary for a personal tool at my current scale.

Lock Lifecycle via Express Hooks

I release the lock using hooks on the Express response object so a user's status does not stay stuck on processing indefinitely.

• res.on('finish', release) — runs after the response has been fully sent to the client. This is the normal path: the request finished processing, so the lock can be released and the status returns to idle.
• res.on('close', release) — runs when the connection is closed before the response completes — for example, if the client cancels the request or the connection drops. Without this, an interrupted request would leave the lock held and block all future requests for that user.

Both hooks call the same release function, guarded by a released flag so the lock is only cleared once:

let released = false;
const release = () => {
    if (released) return;
    released = true;
    releaseUserLock(userId, balanceCache);
};

res.on("finish", release);
res.on("close", release);

The mutex still lives in local process memory. For horizontal scaling, I would need to move to Redis or another distributed locking mechanism.

Idempotency-Aware Mutex

Once concurrency handling worked, idempotency retries started behaving like concurrency failures.

The mutex was not idempotency-key aware — it treated every in-flight request the same regardless of whether the idempotency key matched. A retry with the same key would get rejected even though it should be allowed through.

Each user entry in the in-memory balanceCache object tracks both lock state and the key currently being processed:

if (entry) {
    entry.status = "processing";
    entry.processingIdempotencyKey = idempotencyKey;
    entry.lastUpdatedAt = Date.now();
} else {
    balanceCache[userId] = {
        status: "processing",
        processingIdempotencyKey: idempotencyKey,
        lastUpdatedAt: Date.now(),
    };
}

When a request arrives while status is already processing, the lock check now compares idempotency keys before rejecting:

if (entry?.status === "processing") {
    if (
        idempotencyKey &&
        entry.processingIdempotencyKey &&
        entry.processingIdempotencyKey === idempotencyKey
    ) {
        return "same_key";  // allow retry with matching key
    }
    return "busy";  // different key — reject with 409
}

This way, concurrent requests with different keys are blocked, but a legitimate retry with the same idempotency key can proceed.

Additional Improvements

• Built an endpoint to clear lab-related transactions so they do not clutter the user's transaction history.
• Added a setInterval cleanup mechanism to remove in-memory cache entries older than 24 hours when their status is idle, so the cache does not grow indefinitely.
• Moved balance cache rebuilding to POST/PATCH/DELETE instead of rebuilding lazily the first time a user ID appears in the cache.