fix: Resolve database encryption atomicity issues and enhance debugging

This commit addresses critical data corruption issues caused by non-atomic
file writes during database encryption, and adds comprehensive diagnostic
logging to help debug encryption-related failures.

**Problem:**
Users reported "Unsupported state or unable to authenticate data" errors
when starting the application after system crashes or Docker container
restarts. The root cause was non-atomic writes of encrypted database files:

1. Encrypted data file written (step 1)
2. Metadata file written (step 2)
→ If process crashes between steps 1 and 2, files become inconsistent
→ New IV/tag in data file, old IV/tag in metadata
→ GCM authentication fails on next startup
→ User data permanently inaccessible

**Solution - Atomic Writes:**

1. Write-to-temp + atomic-rename pattern:
   - Write to temporary files (*.tmp-timestamp-pid)
   - Perform atomic rename operations
   - Clean up temp files on failure

2. Data integrity validation:
   - Add dataSize field to metadata
   - Verify file size before decryption
   - Early detection of corrupted writes

3. Enhanced error diagnostics:
   - Key fingerprints (SHA256 prefix) for verification
   - File modification timestamps
   - Detailed GCM auth failure messages
   - Automatic diagnostic info generation

**Changes:**

database-file-encryption.ts:
- Implement atomic write pattern in encryptDatabaseFromBuffer
- Implement atomic write pattern in encryptDatabaseFile
- Add dataSize field to EncryptedFileMetadata interface
- Validate file size before decryption in decryptDatabaseToBuffer
- Enhanced error messages for GCM auth failures
- Add getDiagnosticInfo() function for comprehensive debugging
- Add debug logging for all encryption/decryption operations

system-crypto.ts:
- Add detailed logging for DATABASE_KEY initialization
- Log key source (env var vs .env file)
- Add key fingerprints to all log messages
- Better error messages when key loading fails

db/index.ts:
- Automatically generate diagnostic info on decryption failure
- Log detailed debugging information to help users troubleshoot

**Debugging Info Added:**

- Key initialization: source, fingerprint, length, path
- Encryption: original size, encrypted size, IV/tag prefixes, temp paths
- Decryption: file timestamps, metadata content, key fingerprint matching
- Auth failures: .env file status, key availability, file consistency
- File diagnostics: existence, readability, size validation, mtime comparison

**Backward Compatibility:**
- dataSize field is optional (metadata.dataSize?: number)
- Old encrypted files without dataSize continue to work
- No migration required

**Testing:**
- Compiled successfully
- No breaking changes to existing APIs
- Graceful handling of legacy v1 encrypted files

Fixes data loss issues reported by users experiencing container restarts
and system crashes during database saves.
This commit is contained in:
ZacharyZcR
2025-11-06 08:18:18 +08:00
parent 9ca7df6542
commit 6c3fa14064
3 changed files with 469 additions and 11 deletions

View File

@@ -80,14 +80,34 @@ class SystemCrypto {
async initializeDatabaseKey(): Promise<void> {
try {
const dataDir = process.env.DATA_DIR || "./db/data";
const envPath = path.join(dataDir, ".env");
const envKey = process.env.DATABASE_KEY;
if (envKey && envKey.length >= 64) {
this.databaseKey = Buffer.from(envKey, "hex");
const keyFingerprint = crypto
.createHash("sha256")
.update(this.databaseKey)
.digest("hex")
.substring(0, 16);
databaseLogger.info("DATABASE_KEY loaded from environment variable", {
operation: "db_key_loaded_from_env",
keyFingerprint,
keyLength: envKey.length,
dataDir,
});
return;
}
const dataDir = process.env.DATA_DIR || "./db/data";
const envPath = path.join(dataDir, ".env");
databaseLogger.debug(
"DATABASE_KEY not found in environment, checking .env file",
{
operation: "db_key_checking_file",
envPath,
},
);
try {
const envContent = await fs.readFile(envPath, "utf8");
@@ -95,14 +115,47 @@ class SystemCrypto {
if (dbKeyMatch && dbKeyMatch[1] && dbKeyMatch[1].length >= 64) {
this.databaseKey = Buffer.from(dbKeyMatch[1], "hex");
process.env.DATABASE_KEY = dbKeyMatch[1];
const keyFingerprint = crypto
.createHash("sha256")
.update(this.databaseKey)
.digest("hex")
.substring(0, 16);
databaseLogger.info("DATABASE_KEY loaded from .env file", {
operation: "db_key_loaded_from_file",
keyFingerprint,
keyLength: dbKeyMatch[1].length,
envPath,
});
return;
} else {
databaseLogger.warn(
"DATABASE_KEY found in .env but invalid or too short",
{
operation: "db_key_invalid_in_file",
envPath,
hasMatch: !!dbKeyMatch,
keyLength: dbKeyMatch?.[1]?.length || 0,
requiredLength: 64,
},
);
}
} catch {}
} catch (fileError) {
databaseLogger.warn("Failed to read .env file for DATABASE_KEY", {
operation: "db_key_file_read_failed",
envPath,
error:
fileError instanceof Error ? fileError.message : "Unknown error",
willGenerateNew: true,
});
}
await this.generateAndGuideDatabaseKey();
} catch (error) {
databaseLogger.error("Failed to initialize database key", error, {
operation: "db_key_init_failed",
dataDir: process.env.DATA_DIR || "./db/data",
});
throw new Error("Database key initialization failed");
}