fix: Resolve database encryption atomicity issues and enhance debugging
This commit addresses critical data corruption issues caused by non-atomic file writes during database encryption, and adds comprehensive diagnostic logging to help debug encryption-related failures. **Problem:** Users reported "Unsupported state or unable to authenticate data" errors when starting the application after system crashes or Docker container restarts. The root cause was non-atomic writes of encrypted database files: 1. Encrypted data file written (step 1) 2. Metadata file written (step 2) → If process crashes between steps 1 and 2, files become inconsistent → New IV/tag in data file, old IV/tag in metadata → GCM authentication fails on next startup → User data permanently inaccessible **Solution - Atomic Writes:** 1. Write-to-temp + atomic-rename pattern: - Write to temporary files (*.tmp-timestamp-pid) - Perform atomic rename operations - Clean up temp files on failure 2. Data integrity validation: - Add dataSize field to metadata - Verify file size before decryption - Early detection of corrupted writes 3. Enhanced error diagnostics: - Key fingerprints (SHA256 prefix) for verification - File modification timestamps - Detailed GCM auth failure messages - Automatic diagnostic info generation **Changes:** database-file-encryption.ts: - Implement atomic write pattern in encryptDatabaseFromBuffer - Implement atomic write pattern in encryptDatabaseFile - Add dataSize field to EncryptedFileMetadata interface - Validate file size before decryption in decryptDatabaseToBuffer - Enhanced error messages for GCM auth failures - Add getDiagnosticInfo() function for comprehensive debugging - Add debug logging for all encryption/decryption operations system-crypto.ts: - Add detailed logging for DATABASE_KEY initialization - Log key source (env var vs .env file) - Add key fingerprints to all log messages - Better error messages when key loading fails db/index.ts: - Automatically generate diagnostic info on decryption failure - Log detailed debugging information to help users troubleshoot **Debugging Info Added:** - Key initialization: source, fingerprint, length, path - Encryption: original size, encrypted size, IV/tag prefixes, temp paths - Decryption: file timestamps, metadata content, key fingerprint matching - Auth failures: .env file status, key availability, file consistency - File diagnostics: existence, readability, size validation, mtime comparison **Backward Compatibility:** - dataSize field is optional (metadata.dataSize?: number) - Old encrypted files without dataSize continue to work - No migration required **Testing:** - Compiled successfully - No breaking changes to existing APIs - Graceful handling of legacy v1 encrypted files Fixes data loss issues reported by users experiencing container restarts and system crashes during database saves.
This commit is contained in:
@@ -80,14 +80,34 @@ class SystemCrypto {
|
||||
|
||||
async initializeDatabaseKey(): Promise<void> {
|
||||
try {
|
||||
const dataDir = process.env.DATA_DIR || "./db/data";
|
||||
const envPath = path.join(dataDir, ".env");
|
||||
|
||||
const envKey = process.env.DATABASE_KEY;
|
||||
if (envKey && envKey.length >= 64) {
|
||||
this.databaseKey = Buffer.from(envKey, "hex");
|
||||
const keyFingerprint = crypto
|
||||
.createHash("sha256")
|
||||
.update(this.databaseKey)
|
||||
.digest("hex")
|
||||
.substring(0, 16);
|
||||
|
||||
databaseLogger.info("DATABASE_KEY loaded from environment variable", {
|
||||
operation: "db_key_loaded_from_env",
|
||||
keyFingerprint,
|
||||
keyLength: envKey.length,
|
||||
dataDir,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const dataDir = process.env.DATA_DIR || "./db/data";
|
||||
const envPath = path.join(dataDir, ".env");
|
||||
databaseLogger.debug(
|
||||
"DATABASE_KEY not found in environment, checking .env file",
|
||||
{
|
||||
operation: "db_key_checking_file",
|
||||
envPath,
|
||||
},
|
||||
);
|
||||
|
||||
try {
|
||||
const envContent = await fs.readFile(envPath, "utf8");
|
||||
@@ -95,14 +115,47 @@ class SystemCrypto {
|
||||
if (dbKeyMatch && dbKeyMatch[1] && dbKeyMatch[1].length >= 64) {
|
||||
this.databaseKey = Buffer.from(dbKeyMatch[1], "hex");
|
||||
process.env.DATABASE_KEY = dbKeyMatch[1];
|
||||
|
||||
const keyFingerprint = crypto
|
||||
.createHash("sha256")
|
||||
.update(this.databaseKey)
|
||||
.digest("hex")
|
||||
.substring(0, 16);
|
||||
|
||||
databaseLogger.info("DATABASE_KEY loaded from .env file", {
|
||||
operation: "db_key_loaded_from_file",
|
||||
keyFingerprint,
|
||||
keyLength: dbKeyMatch[1].length,
|
||||
envPath,
|
||||
});
|
||||
return;
|
||||
} else {
|
||||
databaseLogger.warn(
|
||||
"DATABASE_KEY found in .env but invalid or too short",
|
||||
{
|
||||
operation: "db_key_invalid_in_file",
|
||||
envPath,
|
||||
hasMatch: !!dbKeyMatch,
|
||||
keyLength: dbKeyMatch?.[1]?.length || 0,
|
||||
requiredLength: 64,
|
||||
},
|
||||
);
|
||||
}
|
||||
} catch {}
|
||||
} catch (fileError) {
|
||||
databaseLogger.warn("Failed to read .env file for DATABASE_KEY", {
|
||||
operation: "db_key_file_read_failed",
|
||||
envPath,
|
||||
error:
|
||||
fileError instanceof Error ? fileError.message : "Unknown error",
|
||||
willGenerateNew: true,
|
||||
});
|
||||
}
|
||||
|
||||
await this.generateAndGuideDatabaseKey();
|
||||
} catch (error) {
|
||||
databaseLogger.error("Failed to initialize database key", error, {
|
||||
operation: "db_key_init_failed",
|
||||
dataDir: process.env.DATA_DIR || "./db/data",
|
||||
});
|
||||
throw new Error("Database key initialization failed");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user