v1.9.0 (#437)
* fix: Resolve database encryption atomicity issues and enhance debugging (#430) * fix: Resolve database encryption atomicity issues and enhance debugging This commit addresses critical data corruption issues caused by non-atomic file writes during database encryption, and adds comprehensive diagnostic logging to help debug encryption-related failures. **Problem:** Users reported "Unsupported state or unable to authenticate data" errors when starting the application after system crashes or Docker container restarts. The root cause was non-atomic writes of encrypted database files: 1. Encrypted data file written (step 1) 2. Metadata file written (step 2) → If process crashes between steps 1 and 2, files become inconsistent → New IV/tag in data file, old IV/tag in metadata → GCM authentication fails on next startup → User data permanently inaccessible **Solution - Atomic Writes:** 1. Write-to-temp + atomic-rename pattern: - Write to temporary files (*.tmp-timestamp-pid) - Perform atomic rename operations - Clean up temp files on failure 2. Data integrity validation: - Add dataSize field to metadata - Verify file size before decryption - Early detection of corrupted writes 3. Enhanced error diagnostics: - Key fingerprints (SHA256 prefix) for verification - File modification timestamps - Detailed GCM auth failure messages - Automatic diagnostic info generation **Changes:** database-file-encryption.ts: - Implement atomic write pattern in encryptDatabaseFromBuffer - Implement atomic write pattern in encryptDatabaseFile - Add dataSize field to EncryptedFileMetadata interface - Validate file size before decryption in decryptDatabaseToBuffer - Enhanced error messages for GCM auth failures - Add getDiagnosticInfo() function for comprehensive debugging - Add debug logging for all encryption/decryption operations system-crypto.ts: - Add detailed logging for DATABASE_KEY initialization - Log key source (env var vs .env file) - Add key fingerprints to all log messages - Better error messages when key loading fails db/index.ts: - Automatically generate diagnostic info on decryption failure - Log detailed debugging information to help users troubleshoot **Debugging Info Added:** - Key initialization: source, fingerprint, length, path - Encryption: original size, encrypted size, IV/tag prefixes, temp paths - Decryption: file timestamps, metadata content, key fingerprint matching - Auth failures: .env file status, key availability, file consistency - File diagnostics: existence, readability, size validation, mtime comparison **Backward Compatibility:** - dataSize field is optional (metadata.dataSize?: number) - Old encrypted files without dataSize continue to work - No migration required **Testing:** - Compiled successfully - No breaking changes to existing APIs - Graceful handling of legacy v1 encrypted files Fixes data loss issues reported by users experiencing container restarts and system crashes during database saves. * fix: Cleanup PR * Update src/backend/utils/database-file-encryption.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/backend/utils/database-file-encryption.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/backend/utils/database-file-encryption.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/backend/utils/database-file-encryption.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update src/backend/utils/database-file-encryption.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: LukeGus <bugattiguy527@gmail.com> Co-authored-by: Luke Gustafson <88517757+LukeGus@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Merge metadata and DB into 1 file * fix: Add initial command palette * Feature/german language support (#431) * Update translation.json Fixed some translation issues for German, made it more user friendly and common. * Update translation.json added updated block for serverStats * Update translation.json Added translations * Update translation.json Removed duplicate of "free":"Free" * feat: Finalize command palette * fix: Several bug fixes for terminals, server stats, and general feature improvements * feat: Enhanced security, UI improvements, and animations (#432) * fix: Remove empty catch blocks and add error logging * refactor: Modularize server stats widget collectors * feat: Add i18n support for terminal customization and login stats - Add comprehensive terminal customization translations (60+ keys) for appearance, behavior, and advanced settings across all 4 languages - Add SSH login statistics translations - Update HostManagerEditor to use i18n for all terminal customization UI elements - Update LoginStatsWidget to use i18n for all UI text - Add missing logger imports in backend files for improved debugging * feat: Add keyboard shortcut enhancements with Kbd component - Add shadcn kbd component for displaying keyboard shortcuts - Enhance file manager context menu to display shortcuts with Kbd component - Add 5 new keyboard shortcuts to file manager: - Ctrl+D: Download selected files - Ctrl+N: Create new file - Ctrl+Shift+N: Create new folder - Ctrl+U: Upload files - Enter: Open/run selected file - Add keyboard shortcut hints to command palette footer - Create helper function to parse and render keyboard shortcuts * feat: Add i18n support for command palette - Add commandPalette translation section with 22 keys to all 4 languages - Update CommandPalette component to use i18n for all UI text - Translate search placeholder, group headings, menu items, and shortcut hints - Support multilingual command palette interface * feat: Add smooth transitions and animations to UI - Add fade-in/fade-out transition to command palette (200ms) - Add scale animation to command palette on open/close - Add smooth popup animation to context menu (150ms) - Add visual feedback for file selection with ring effect - Add hover scale effect to file grid items - Add transition-all to list view items for consistent behavior - Zero JavaScript overhead, pure CSS transitions - All animations under 200ms for instant feel * feat: Add button active state and dashboard card animations - Add active:scale-95 to all buttons for tactile click feedback - Add hover border effect to dashboard cards (150ms transition) - Add pulse animation to dashboard loading states - Pure CSS transitions with zero JavaScript overhead - Improves enterprise-level feel of UI * feat: Add smooth macOS-style page transitions - Add fullscreen crossfade transition for login/logout (300ms fade-out + 400ms fade-in) - Add slide-in-from-right animation for all page switches (Dashboard, Terminal, SSH Manager, Admin, Profile) - Fix TypeScript compilation by adding esModuleInterop to tsconfig.node.json - Pass handleLogout from DesktopApp to LeftSidebar for consistent transition behavior All page transitions now use Tailwind animate-in utilities with 300ms duration for smooth, native-feeling UX * fix: Add key prop to force animation re-trigger on tab switch Each page container now has key={currentTab} to ensure React unmounts and remounts the element on every tab switch, properly triggering the slide-in animation * revert: Remove page transition animations Page switching animations were not noticeable enough and felt unnecessary. Keep only the login/logout fullscreen crossfade transitions which provide clear visual feedback for authentication state changes * feat: Add ripple effect to login/logout transitions Add three-layer expanding ripple animation during fadeOut phase: - Ripples expand from screen center using primary theme color - Each layer has staggered delay (0ms, 150ms, 300ms) for wave effect - Ripples fade out as they expand to create elegant visual feedback - Uses pure CSS keyframe animation, no external libraries Total animation: 800ms ripple + 300ms screen fade * feat: Add smooth TERMIX logo animation to transitions Changes: - Extend transition duration from 300ms/400ms to 800ms/600ms for more elegant feel - Reduce ripple intensity from /20,/15,/10 to /8,/5 for subtlety - Slow down ripple animation from 0.8s to 2s with cubic-bezier easing - Add centered TERMIX logo with monospace font and subtitle - Logo fades in from 80% scale, holds, then fades out at 110% scale - Total effect: 1.2s logo animation synced with 2s ripple waves Creates a premium, branded transition experience * feat: Enhance transition animation with premium details Timing adjustments: - Extend fadeOut from 800ms to 1200ms - Extend fadeIn from 600ms to 800ms - Slow background fade to 700ms for elegance Visual enhancements: - Add 4-layer ripple waves (10%, 7%, 5%, 3% opacity) with staggered delays - Ripple animation extended to 2.5s with refined opacity curve - Logo blur effect: starts at 8px, sharpens to 0px, exits at 4px - Logo glow effect: triple-layer text-shadow using primary theme color - Increase logo size from text-6xl to text-7xl - Subtitle delayed fade-in from bottom with smooth slide animation Creates a cinematic, polished brand experience * feat: Redesign login page with split-screen cinematic layout Major redesign of authentication page: Left Side (40% width): - Full-height gradient background using primary theme color - Large TERMIX logo with glow effect - Subtitle and tagline - Infinite animated ripple waves (3 layers) - Hidden on mobile, shows brand identity Right Side (60% width): - Centered glassmorphism card with backdrop blur - Refined tab switcher with pill-style active state - Enlarged title with gradient text effect - Added welcome subtitles for better UX - Card slides in from bottom on load - All existing functionality preserved Visual enhancements: - Tab navigation: segmented control style in muted container - Active tab: white background with subtle shadow - Smooth 200ms transitions on all interactions - Card: rounded-2xl, shadow-xl, semi-transparent border Creates premium, modern login experience matching transition animations * feat: Update login page theme colors and add i18n support - Changed login page gradient from blue to match dark theme colors - Updated ripple effects to use theme primary color - Added i18n translation keys for login page (auth.tagline, auth.description, auth.welcomeBack, auth.createAccount, auth.continueExternal) - Updated all language files (en, zh, de, ru, pt-BR) with new translations - Fixed TypeScript compilation issues by clearing build cache * refactor: Use shadcn Tabs component and fix modal styling - Replace custom tab navigation with shadcn Tabs component - Restore border-2 border-dark-border for modal consistency - Remove circular icon from login success message - Simplify authentication success display * refactor: Remove ripple effects and gradient from login page - Remove animated ripple background effects - Remove gradient background, use solid color (bg-dark-bg-darker) - Remove text-shadow glow effect from logo - Simplify brand showcase to clean, minimal design * feat: Add decorative slash and remove subtitle from login page - Add decorative slash divider with gradient lines below TERMIX logo - Remove subtitle text (welcomeBack and createAccount) - Simplify page title to show only the main heading * feat: Add diagonal line pattern background to login page - Replace decorative slash with subtle diagonal line pattern background - Use repeating-linear-gradient at 45deg angle - Set very low opacity (0.03) for subtle effect - Pattern uses theme primary color * fix: Display diagonal line pattern on login background - Combine background color and pattern in single style attribute - Use white semi-transparent lines (rgba 0.03 opacity) - 45deg angle, 35px spacing, 2px width - Remove separate overlay div to ensure pattern visibility * security: Fix user enumeration vulnerability in login - Unify error messages for invalid username and incorrect password - Both return 401 status with 'Invalid username or password' - Prevent attackers from enumerating valid usernames - Maintain detailed logging for debugging purposes - Changed from 404 'User not found' to generic auth failure message * security: Add login rate limiting to prevent brute force attacks - Implement LoginRateLimiter with IP and username-based tracking - Block after 5 failed attempts within 15 minutes - Lock account/IP for 15 minutes after threshold - Automatic cleanup of expired entries every 5 minutes - Track remaining attempts in logs for monitoring - Return 429 status with remaining time on rate limit - Reset counters on successful login - Dual protection: both IP-based and username-based limits * French translation (#434) * Adding French Language * Enhancements * feat: Replace the old ssh tools system with a new dedicated sidebar * fix: Merge zac/luke * fix: Finalize new sidebar, improve and loading animations * Added ability to close non-primary tabs involved in a split view (#435) * fix: General bug fixes/small feature improvements * feat: General UI improvements and translation updates * fix: Command history and file manager styling issues * feat: General bug fixes, added server stat commands, improved split screen, link accounts, etc * fix: add Accept header for OIDC callback request (#436) * Delete DOWNLOADS.md * fix: add Accept header for OIDC callback request --------- Co-authored-by: Luke Gustafson <88517757+LukeGus@users.noreply.github.com> * fix: More bug fixes and QOL fixes * fix: Server stats not respecting interval and fixed SSH toool type issues * fix: Remove github links * fix: Delete account spacing * fix: Increment version * fix: Unable to delete hosts and add nginx for terminal * fix: Unable to delete hosts * fix: Unable to delete hosts * fix: Unable to delete hosts * fix: OIDC/local account linking breaking both logins * chore: File cleanup * feat: Max terminal tab size and save current file manager sorting type * fix: Terminal display issue, migrate host editor to use combobox * feat: Add snippet folder/customization system * fix: Fix OIDC linking and prep release * fix: Increment version --------- Co-authored-by: ZacharyZcR <zacharyzcr1984@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Max <herzmaximilian@gmail.com> Co-authored-by: SlimGary <trash.slim@gmail.com> Co-authored-by: jarrah31 <jarrah31@gmail.com> Co-authored-by: Kf637 <mail@kf637.tech>
This commit was merged in pull request #437.
This commit is contained in:
Luke Gustafson
GitHub
@@ -6,10 +6,185 @@ import { Client, type ConnectConfig } from "ssh2";
|
||||
import { getDb } from "../database/db/index.js";
|
||||
import { sshData, sshCredentials } from "../database/db/schema.js";
|
||||
import { eq, and } from "drizzle-orm";
|
||||
import { statsLogger } from "../utils/logger.js";
|
||||
import { statsLogger, sshLogger } from "../utils/logger.js";
|
||||
import { SimpleDBOps } from "../utils/simple-db-ops.js";
|
||||
import { AuthManager } from "../utils/auth-manager.js";
|
||||
import type { AuthenticatedRequest } from "../../types/index.js";
|
||||
import { collectCpuMetrics } from "./widgets/cpu-collector.js";
|
||||
import { collectMemoryMetrics } from "./widgets/memory-collector.js";
|
||||
import { collectDiskMetrics } from "./widgets/disk-collector.js";
|
||||
import { collectNetworkMetrics } from "./widgets/network-collector.js";
|
||||
import { collectUptimeMetrics } from "./widgets/uptime-collector.js";
|
||||
import { collectProcessesMetrics } from "./widgets/processes-collector.js";
|
||||
import { collectSystemMetrics } from "./widgets/system-collector.js";
|
||||
import { collectLoginStats } from "./widgets/login-stats-collector.js";
|
||||
|
||||
async function resolveJumpHost(
|
||||
hostId: number,
|
||||
userId: string,
|
||||
): Promise<any | null> {
|
||||
try {
|
||||
const hosts = await SimpleDBOps.select(
|
||||
getDb()
|
||||
.select()
|
||||
.from(sshData)
|
||||
.where(and(eq(sshData.id, hostId), eq(sshData.userId, userId))),
|
||||
"ssh_data",
|
||||
userId,
|
||||
);
|
||||
|
||||
if (hosts.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const host = hosts[0];
|
||||
|
||||
if (host.credentialId) {
|
||||
const credentials = await SimpleDBOps.select(
|
||||
getDb()
|
||||
.select()
|
||||
.from(sshCredentials)
|
||||
.where(
|
||||
and(
|
||||
eq(sshCredentials.id, host.credentialId as number),
|
||||
eq(sshCredentials.userId, userId),
|
||||
),
|
||||
),
|
||||
"ssh_credentials",
|
||||
userId,
|
||||
);
|
||||
|
||||
if (credentials.length > 0) {
|
||||
const credential = credentials[0];
|
||||
return {
|
||||
...host,
|
||||
password: credential.password,
|
||||
key:
|
||||
credential.private_key || credential.privateKey || credential.key,
|
||||
keyPassword: credential.key_password || credential.keyPassword,
|
||||
keyType: credential.key_type || credential.keyType,
|
||||
authType: credential.auth_type || credential.authType,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return host;
|
||||
} catch (error) {
|
||||
statsLogger.error("Failed to resolve jump host", error, {
|
||||
operation: "resolve_jump_host",
|
||||
hostId,
|
||||
userId,
|
||||
});
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function createJumpHostChain(
|
||||
jumpHosts: Array<{ hostId: number }>,
|
||||
userId: string,
|
||||
): Promise<Client | null> {
|
||||
if (!jumpHosts || jumpHosts.length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
let currentClient: Client | null = null;
|
||||
const clients: Client[] = [];
|
||||
|
||||
try {
|
||||
for (let i = 0; i < jumpHosts.length; i++) {
|
||||
const jumpHostConfig = await resolveJumpHost(jumpHosts[i].hostId, userId);
|
||||
|
||||
if (!jumpHostConfig) {
|
||||
statsLogger.error(`Jump host ${i + 1} not found`, undefined, {
|
||||
operation: "jump_host_chain",
|
||||
hostId: jumpHosts[i].hostId,
|
||||
});
|
||||
clients.forEach((c) => c.end());
|
||||
return null;
|
||||
}
|
||||
|
||||
const jumpClient = new Client();
|
||||
clients.push(jumpClient);
|
||||
|
||||
const connected = await new Promise<boolean>((resolve) => {
|
||||
const timeout = setTimeout(() => {
|
||||
resolve(false);
|
||||
}, 30000);
|
||||
|
||||
jumpClient.on("ready", () => {
|
||||
clearTimeout(timeout);
|
||||
resolve(true);
|
||||
});
|
||||
|
||||
jumpClient.on("error", (err) => {
|
||||
clearTimeout(timeout);
|
||||
statsLogger.error(`Jump host ${i + 1} connection failed`, err, {
|
||||
operation: "jump_host_connect",
|
||||
hostId: jumpHostConfig.id,
|
||||
ip: jumpHostConfig.ip,
|
||||
});
|
||||
resolve(false);
|
||||
});
|
||||
|
||||
const connectConfig: any = {
|
||||
host: jumpHostConfig.ip,
|
||||
port: jumpHostConfig.port || 22,
|
||||
username: jumpHostConfig.username,
|
||||
tryKeyboard: true,
|
||||
readyTimeout: 30000,
|
||||
};
|
||||
|
||||
if (jumpHostConfig.authType === "password" && jumpHostConfig.password) {
|
||||
connectConfig.password = jumpHostConfig.password;
|
||||
} else if (jumpHostConfig.authType === "key" && jumpHostConfig.key) {
|
||||
const cleanKey = jumpHostConfig.key
|
||||
.trim()
|
||||
.replace(/\r\n/g, "\n")
|
||||
.replace(/\r/g, "\n");
|
||||
connectConfig.privateKey = Buffer.from(cleanKey, "utf8");
|
||||
if (jumpHostConfig.keyPassword) {
|
||||
connectConfig.passphrase = jumpHostConfig.keyPassword;
|
||||
}
|
||||
}
|
||||
|
||||
if (currentClient) {
|
||||
currentClient.forwardOut(
|
||||
"127.0.0.1",
|
||||
0,
|
||||
jumpHostConfig.ip,
|
||||
jumpHostConfig.port || 22,
|
||||
(err, stream) => {
|
||||
if (err) {
|
||||
clearTimeout(timeout);
|
||||
resolve(false);
|
||||
return;
|
||||
}
|
||||
connectConfig.sock = stream;
|
||||
jumpClient.connect(connectConfig);
|
||||
},
|
||||
);
|
||||
} else {
|
||||
jumpClient.connect(connectConfig);
|
||||
}
|
||||
});
|
||||
|
||||
if (!connected) {
|
||||
clients.forEach((c) => c.end());
|
||||
return null;
|
||||
}
|
||||
|
||||
currentClient = jumpClient;
|
||||
}
|
||||
|
||||
return currentClient;
|
||||
} catch (error) {
|
||||
statsLogger.error("Failed to create jump host chain", error, {
|
||||
operation: "jump_host_chain",
|
||||
});
|
||||
clients.forEach((c) => c.end());
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
interface PooledConnection {
|
||||
client: Client;
|
||||
@@ -79,7 +254,7 @@ class SSHConnectionPool {
|
||||
private async createConnection(
|
||||
host: SSHHostWithCredentials,
|
||||
): Promise<Client> {
|
||||
return new Promise((resolve, reject) => {
|
||||
return new Promise(async (resolve, reject) => {
|
||||
const client = new Client();
|
||||
const timeout = setTimeout(() => {
|
||||
client.end();
|
||||
@@ -120,7 +295,12 @@ class SSHConnectionPool {
|
||||
),
|
||||
);
|
||||
} else if (host.password) {
|
||||
const responses = prompts.map(() => host.password || "");
|
||||
const responses = prompts.map((p) => {
|
||||
if (/password/i.test(p.prompt)) {
|
||||
return host.password || "";
|
||||
}
|
||||
return "";
|
||||
});
|
||||
finish(responses);
|
||||
} else {
|
||||
finish(prompts.map(() => ""));
|
||||
@@ -129,7 +309,44 @@ class SSHConnectionPool {
|
||||
);
|
||||
|
||||
try {
|
||||
client.connect(buildSshConfig(host));
|
||||
const config = buildSshConfig(host);
|
||||
|
||||
if (host.jumpHosts && host.jumpHosts.length > 0 && host.userId) {
|
||||
const jumpClient = await createJumpHostChain(
|
||||
host.jumpHosts,
|
||||
host.userId,
|
||||
);
|
||||
|
||||
if (!jumpClient) {
|
||||
clearTimeout(timeout);
|
||||
reject(new Error("Failed to establish jump host chain"));
|
||||
return;
|
||||
}
|
||||
|
||||
jumpClient.forwardOut(
|
||||
"127.0.0.1",
|
||||
0,
|
||||
host.ip,
|
||||
host.port,
|
||||
(err, stream) => {
|
||||
if (err) {
|
||||
clearTimeout(timeout);
|
||||
jumpClient.end();
|
||||
reject(
|
||||
new Error(
|
||||
"Failed to forward through jump host: " + err.message,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
config.sock = stream;
|
||||
client.connect(config);
|
||||
},
|
||||
);
|
||||
} else {
|
||||
client.connect(config);
|
||||
}
|
||||
} catch (err) {
|
||||
clearTimeout(timeout);
|
||||
reject(err);
|
||||
@@ -156,7 +373,7 @@ class SSHConnectionPool {
|
||||
if (!conn.inUse && now - conn.lastUsed > maxAge) {
|
||||
try {
|
||||
conn.client.end();
|
||||
} catch {}
|
||||
} catch (error) {}
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
@@ -176,7 +393,7 @@ class SSHConnectionPool {
|
||||
for (const conn of connections) {
|
||||
try {
|
||||
conn.client.end();
|
||||
} catch {}
|
||||
} catch (error) {}
|
||||
}
|
||||
}
|
||||
this.connections.clear();
|
||||
@@ -214,7 +431,7 @@ class RequestQueue {
|
||||
if (request) {
|
||||
try {
|
||||
await request();
|
||||
} catch {}
|
||||
} catch (error) {}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -382,7 +599,8 @@ interface SSHHostWithCredentials {
|
||||
enableFileManager: boolean;
|
||||
defaultPath: string;
|
||||
tunnelConnections: unknown[];
|
||||
statsConfig?: string;
|
||||
jumpHosts?: Array<{ hostId: number }>;
|
||||
statsConfig?: string | StatsConfig;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
userId: string;
|
||||
@@ -427,34 +645,64 @@ class PollingManager {
|
||||
}
|
||||
>();
|
||||
|
||||
parseStatsConfig(statsConfigStr?: string): StatsConfig {
|
||||
parseStatsConfig(statsConfigStr?: string | StatsConfig): StatsConfig {
|
||||
if (!statsConfigStr) {
|
||||
return DEFAULT_STATS_CONFIG;
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(statsConfigStr);
|
||||
return { ...DEFAULT_STATS_CONFIG, ...parsed };
|
||||
} catch (error) {
|
||||
statsLogger.warn(
|
||||
`Failed to parse statsConfig: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
);
|
||||
return DEFAULT_STATS_CONFIG;
|
||||
|
||||
let parsed: StatsConfig;
|
||||
|
||||
if (typeof statsConfigStr === "object") {
|
||||
parsed = statsConfigStr;
|
||||
} else {
|
||||
try {
|
||||
let temp: any = JSON.parse(statsConfigStr);
|
||||
|
||||
if (typeof temp === "string") {
|
||||
temp = JSON.parse(temp);
|
||||
}
|
||||
|
||||
parsed = temp;
|
||||
} catch (error) {
|
||||
statsLogger.warn(
|
||||
`Failed to parse statsConfig: ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
{
|
||||
operation: "parse_stats_config_error",
|
||||
statsConfigStr,
|
||||
},
|
||||
);
|
||||
return DEFAULT_STATS_CONFIG;
|
||||
}
|
||||
}
|
||||
|
||||
const result = { ...DEFAULT_STATS_CONFIG, ...parsed };
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
async startPollingForHost(host: SSHHostWithCredentials): Promise<void> {
|
||||
const statsConfig = this.parseStatsConfig(host.statsConfig);
|
||||
|
||||
const existingConfig = this.pollingConfigs.get(host.id);
|
||||
|
||||
if (existingConfig) {
|
||||
if (existingConfig.statusTimer) {
|
||||
clearInterval(existingConfig.statusTimer);
|
||||
existingConfig.statusTimer = undefined;
|
||||
}
|
||||
if (existingConfig.metricsTimer) {
|
||||
clearInterval(existingConfig.metricsTimer);
|
||||
existingConfig.metricsTimer = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
if (!statsConfig.statusCheckEnabled && !statsConfig.metricsEnabled) {
|
||||
this.pollingConfigs.delete(host.id);
|
||||
this.statusStore.delete(host.id);
|
||||
this.metricsStore.delete(host.id);
|
||||
return;
|
||||
}
|
||||
|
||||
const config: HostPollingConfig = {
|
||||
host,
|
||||
statsConfig,
|
||||
@@ -466,7 +714,10 @@ class PollingManager {
|
||||
this.pollHostStatus(host);
|
||||
|
||||
config.statusTimer = setInterval(() => {
|
||||
this.pollHostStatus(host);
|
||||
const latestConfig = this.pollingConfigs.get(host.id);
|
||||
if (latestConfig && latestConfig.statsConfig.statusCheckEnabled) {
|
||||
this.pollHostStatus(latestConfig.host);
|
||||
}
|
||||
}, intervalMs);
|
||||
} else {
|
||||
this.statusStore.delete(host.id);
|
||||
@@ -478,7 +729,10 @@ class PollingManager {
|
||||
this.pollHostMetrics(host);
|
||||
|
||||
config.metricsTimer = setInterval(() => {
|
||||
this.pollHostMetrics(host);
|
||||
const latestConfig = this.pollingConfigs.get(host.id);
|
||||
if (latestConfig && latestConfig.statsConfig.metricsEnabled) {
|
||||
this.pollHostMetrics(latestConfig.host);
|
||||
}
|
||||
}, intervalMs);
|
||||
} else {
|
||||
this.metricsStore.delete(host.id);
|
||||
@@ -505,27 +759,51 @@ class PollingManager {
|
||||
}
|
||||
|
||||
private async pollHostMetrics(host: SSHHostWithCredentials): Promise<void> {
|
||||
const config = this.pollingConfigs.get(host.id);
|
||||
if (!config || !config.statsConfig.metricsEnabled) {
|
||||
return;
|
||||
}
|
||||
|
||||
const currentHost = config.host;
|
||||
|
||||
try {
|
||||
const metrics = await collectMetrics(host);
|
||||
this.metricsStore.set(host.id, {
|
||||
const metrics = await collectMetrics(currentHost);
|
||||
this.metricsStore.set(currentHost.id, {
|
||||
data: metrics,
|
||||
timestamp: Date.now(),
|
||||
});
|
||||
} catch (error) {}
|
||||
} catch (error) {
|
||||
const errorMessage =
|
||||
error instanceof Error ? error.message : String(error);
|
||||
|
||||
const latestConfig = this.pollingConfigs.get(currentHost.id);
|
||||
if (latestConfig && latestConfig.statsConfig.metricsEnabled) {
|
||||
statsLogger.warn("Failed to collect metrics for host", {
|
||||
operation: "metrics_poll_failed",
|
||||
hostId: currentHost.id,
|
||||
hostName: currentHost.name,
|
||||
error: errorMessage,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
stopPollingForHost(hostId: number): void {
|
||||
stopPollingForHost(hostId: number, clearData = true): void {
|
||||
const config = this.pollingConfigs.get(hostId);
|
||||
if (config) {
|
||||
if (config.statusTimer) {
|
||||
clearInterval(config.statusTimer);
|
||||
config.statusTimer = undefined;
|
||||
}
|
||||
if (config.metricsTimer) {
|
||||
clearInterval(config.metricsTimer);
|
||||
config.metricsTimer = undefined;
|
||||
}
|
||||
this.pollingConfigs.delete(hostId);
|
||||
this.statusStore.delete(hostId);
|
||||
this.metricsStore.delete(hostId);
|
||||
if (clearData) {
|
||||
this.statusStore.delete(hostId);
|
||||
this.metricsStore.delete(hostId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -554,11 +832,23 @@ class PollingManager {
|
||||
}
|
||||
|
||||
async refreshHostPolling(userId: string): Promise<void> {
|
||||
const hosts = await fetchAllHosts(userId);
|
||||
const currentHostIds = new Set(hosts.map((h) => h.id));
|
||||
|
||||
for (const hostId of this.pollingConfigs.keys()) {
|
||||
this.stopPollingForHost(hostId);
|
||||
this.stopPollingForHost(hostId, false);
|
||||
}
|
||||
|
||||
await this.initializePolling(userId);
|
||||
for (const hostId of this.statusStore.keys()) {
|
||||
if (!currentHostIds.has(hostId)) {
|
||||
this.statusStore.delete(hostId);
|
||||
this.metricsStore.delete(hostId);
|
||||
}
|
||||
}
|
||||
|
||||
for (const host of hosts) {
|
||||
await this.startPollingForHost(host);
|
||||
}
|
||||
}
|
||||
|
||||
destroy(): void {
|
||||
@@ -712,6 +1002,7 @@ async function resolveHostCredentials(
|
||||
tunnelConnections: host.tunnelConnections
|
||||
? JSON.parse(host.tunnelConnections as string)
|
||||
: [],
|
||||
jumpHosts: host.jumpHosts ? JSON.parse(host.jumpHosts as string) : [],
|
||||
statsConfig: host.statsConfig || undefined,
|
||||
createdAt: host.createdAt,
|
||||
updatedAt: host.updatedAt,
|
||||
@@ -911,59 +1202,6 @@ async function withSshConnection<T>(
|
||||
}
|
||||
}
|
||||
|
||||
function execCommand(
|
||||
client: Client,
|
||||
command: string,
|
||||
): Promise<{
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
code: number | null;
|
||||
}> {
|
||||
return new Promise((resolve, reject) => {
|
||||
client.exec(command, { pty: false }, (err, stream) => {
|
||||
if (err) return reject(err);
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
let exitCode: number | null = null;
|
||||
stream
|
||||
.on("close", (code: number | undefined) => {
|
||||
exitCode = typeof code === "number" ? code : null;
|
||||
resolve({ stdout, stderr, code: exitCode });
|
||||
})
|
||||
.on("data", (data: Buffer) => {
|
||||
stdout += data.toString("utf8");
|
||||
})
|
||||
.stderr.on("data", (data: Buffer) => {
|
||||
stderr += data.toString("utf8");
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function parseCpuLine(
|
||||
cpuLine: string,
|
||||
): { total: number; idle: number } | undefined {
|
||||
const parts = cpuLine.trim().split(/\s+/);
|
||||
if (parts[0] !== "cpu") return undefined;
|
||||
const nums = parts
|
||||
.slice(1)
|
||||
.map((n) => Number(n))
|
||||
.filter((n) => Number.isFinite(n));
|
||||
if (nums.length < 4) return undefined;
|
||||
const idle = (nums[3] ?? 0) + (nums[4] ?? 0);
|
||||
const total = nums.reduce((a, b) => a + b, 0);
|
||||
return { total, idle };
|
||||
}
|
||||
|
||||
function toFixedNum(n: number | null | undefined, digits = 2): number | null {
|
||||
if (typeof n !== "number" || !Number.isFinite(n)) return null;
|
||||
return Number(n.toFixed(digits));
|
||||
}
|
||||
|
||||
function kibToGiB(kib: number): number {
|
||||
return kib / (1024 * 1024);
|
||||
}
|
||||
|
||||
async function collectMetrics(host: SSHHostWithCredentials): Promise<{
|
||||
cpu: {
|
||||
percent: number | null;
|
||||
@@ -1026,298 +1264,38 @@ async function collectMetrics(host: SSHHostWithCredentials): Promise<{
|
||||
return requestQueue.queueRequest(host.id, async () => {
|
||||
try {
|
||||
return await withSshConnection(host, async (client) => {
|
||||
let cpuPercent: number | null = null;
|
||||
let cores: number | null = null;
|
||||
let loadTriplet: [number, number, number] | null = null;
|
||||
const cpu = await collectCpuMetrics(client);
|
||||
const memory = await collectMemoryMetrics(client);
|
||||
const disk = await collectDiskMetrics(client);
|
||||
const network = await collectNetworkMetrics(client);
|
||||
const uptime = await collectUptimeMetrics(client);
|
||||
const processes = await collectProcessesMetrics(client);
|
||||
const system = await collectSystemMetrics(client);
|
||||
|
||||
let login_stats = {
|
||||
recentLogins: [],
|
||||
failedLogins: [],
|
||||
totalLogins: 0,
|
||||
uniqueIPs: 0,
|
||||
};
|
||||
try {
|
||||
const [stat1, loadAvgOut, coresOut] = await Promise.all([
|
||||
execCommand(client, "cat /proc/stat"),
|
||||
execCommand(client, "cat /proc/loadavg"),
|
||||
execCommand(
|
||||
client,
|
||||
"nproc 2>/dev/null || grep -c ^processor /proc/cpuinfo",
|
||||
),
|
||||
]);
|
||||
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
const stat2 = await execCommand(client, "cat /proc/stat");
|
||||
|
||||
const cpuLine1 = (
|
||||
stat1.stdout.split("\n").find((l) => l.startsWith("cpu ")) || ""
|
||||
).trim();
|
||||
const cpuLine2 = (
|
||||
stat2.stdout.split("\n").find((l) => l.startsWith("cpu ")) || ""
|
||||
).trim();
|
||||
const a = parseCpuLine(cpuLine1);
|
||||
const b = parseCpuLine(cpuLine2);
|
||||
if (a && b) {
|
||||
const totalDiff = b.total - a.total;
|
||||
const idleDiff = b.idle - a.idle;
|
||||
const used = totalDiff - idleDiff;
|
||||
if (totalDiff > 0)
|
||||
cpuPercent = Math.max(0, Math.min(100, (used / totalDiff) * 100));
|
||||
}
|
||||
|
||||
const laParts = loadAvgOut.stdout.trim().split(/\s+/);
|
||||
if (laParts.length >= 3) {
|
||||
loadTriplet = [
|
||||
Number(laParts[0]),
|
||||
Number(laParts[1]),
|
||||
Number(laParts[2]),
|
||||
].map((v) => (Number.isFinite(v) ? Number(v) : 0)) as [
|
||||
number,
|
||||
number,
|
||||
number,
|
||||
];
|
||||
}
|
||||
|
||||
const coresNum = Number((coresOut.stdout || "").trim());
|
||||
cores = Number.isFinite(coresNum) && coresNum > 0 ? coresNum : null;
|
||||
login_stats = await collectLoginStats(client);
|
||||
} catch (e) {
|
||||
cpuPercent = null;
|
||||
cores = null;
|
||||
loadTriplet = null;
|
||||
statsLogger.debug("Failed to collect login stats", {
|
||||
operation: "login_stats_failed",
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
}
|
||||
|
||||
let memPercent: number | null = null;
|
||||
let usedGiB: number | null = null;
|
||||
let totalGiB: number | null = null;
|
||||
try {
|
||||
const memInfo = await execCommand(client, "cat /proc/meminfo");
|
||||
const lines = memInfo.stdout.split("\n");
|
||||
const getVal = (key: string) => {
|
||||
const line = lines.find((l) => l.startsWith(key));
|
||||
if (!line) return null;
|
||||
const m = line.match(/\d+/);
|
||||
return m ? Number(m[0]) : null;
|
||||
};
|
||||
const totalKb = getVal("MemTotal:");
|
||||
const availKb = getVal("MemAvailable:");
|
||||
if (totalKb && availKb && totalKb > 0) {
|
||||
const usedKb = totalKb - availKb;
|
||||
memPercent = Math.max(0, Math.min(100, (usedKb / totalKb) * 100));
|
||||
usedGiB = kibToGiB(usedKb);
|
||||
totalGiB = kibToGiB(totalKb);
|
||||
}
|
||||
} catch (e) {
|
||||
memPercent = null;
|
||||
usedGiB = null;
|
||||
totalGiB = null;
|
||||
}
|
||||
|
||||
let diskPercent: number | null = null;
|
||||
let usedHuman: string | null = null;
|
||||
let totalHuman: string | null = null;
|
||||
let availableHuman: string | null = null;
|
||||
try {
|
||||
const [diskOutHuman, diskOutBytes] = await Promise.all([
|
||||
execCommand(client, "df -h -P / | tail -n +2"),
|
||||
execCommand(client, "df -B1 -P / | tail -n +2"),
|
||||
]);
|
||||
|
||||
const humanLine =
|
||||
diskOutHuman.stdout
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean)[0] || "";
|
||||
const bytesLine =
|
||||
diskOutBytes.stdout
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean)[0] || "";
|
||||
|
||||
const humanParts = humanLine.split(/\s+/);
|
||||
const bytesParts = bytesLine.split(/\s+/);
|
||||
|
||||
if (humanParts.length >= 6 && bytesParts.length >= 6) {
|
||||
totalHuman = humanParts[1] || null;
|
||||
usedHuman = humanParts[2] || null;
|
||||
availableHuman = humanParts[3] || null;
|
||||
|
||||
const totalBytes = Number(bytesParts[1]);
|
||||
const usedBytes = Number(bytesParts[2]);
|
||||
|
||||
if (
|
||||
Number.isFinite(totalBytes) &&
|
||||
Number.isFinite(usedBytes) &&
|
||||
totalBytes > 0
|
||||
) {
|
||||
diskPercent = Math.max(
|
||||
0,
|
||||
Math.min(100, (usedBytes / totalBytes) * 100),
|
||||
);
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
diskPercent = null;
|
||||
usedHuman = null;
|
||||
totalHuman = null;
|
||||
availableHuman = null;
|
||||
}
|
||||
|
||||
const interfaces: Array<{
|
||||
name: string;
|
||||
ip: string;
|
||||
state: string;
|
||||
rxBytes: string | null;
|
||||
txBytes: string | null;
|
||||
}> = [];
|
||||
try {
|
||||
const ifconfigOut = await execCommand(
|
||||
client,
|
||||
"ip -o addr show | awk '{print $2,$4}' | grep -v '^lo'",
|
||||
);
|
||||
const netStatOut = await execCommand(
|
||||
client,
|
||||
"ip -o link show | awk '{gsub(/:/, \"\", $2); print $2,$9}'",
|
||||
);
|
||||
|
||||
const addrs = ifconfigOut.stdout
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean);
|
||||
const states = netStatOut.stdout
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean);
|
||||
|
||||
const ifMap = new Map<string, { ip: string; state: string }>();
|
||||
for (const line of addrs) {
|
||||
const parts = line.split(/\s+/);
|
||||
if (parts.length >= 2) {
|
||||
const name = parts[0];
|
||||
const ip = parts[1].split("/")[0];
|
||||
if (!ifMap.has(name)) ifMap.set(name, { ip, state: "UNKNOWN" });
|
||||
}
|
||||
}
|
||||
for (const line of states) {
|
||||
const parts = line.split(/\s+/);
|
||||
if (parts.length >= 2) {
|
||||
const name = parts[0];
|
||||
const state = parts[1];
|
||||
const existing = ifMap.get(name);
|
||||
if (existing) {
|
||||
existing.state = state;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const [name, data] of ifMap.entries()) {
|
||||
interfaces.push({
|
||||
name,
|
||||
ip: data.ip,
|
||||
state: data.state,
|
||||
rxBytes: null,
|
||||
txBytes: null,
|
||||
});
|
||||
}
|
||||
} catch (e) {}
|
||||
|
||||
let uptimeSeconds: number | null = null;
|
||||
let uptimeFormatted: string | null = null;
|
||||
try {
|
||||
const uptimeOut = await execCommand(client, "cat /proc/uptime");
|
||||
const uptimeParts = uptimeOut.stdout.trim().split(/\s+/);
|
||||
if (uptimeParts.length >= 1) {
|
||||
uptimeSeconds = Number(uptimeParts[0]);
|
||||
if (Number.isFinite(uptimeSeconds)) {
|
||||
const days = Math.floor(uptimeSeconds / 86400);
|
||||
const hours = Math.floor((uptimeSeconds % 86400) / 3600);
|
||||
const minutes = Math.floor((uptimeSeconds % 3600) / 60);
|
||||
uptimeFormatted = `${days}d ${hours}h ${minutes}m`;
|
||||
}
|
||||
}
|
||||
} catch (e) {}
|
||||
|
||||
let totalProcesses: number | null = null;
|
||||
let runningProcesses: number | null = null;
|
||||
const topProcesses: Array<{
|
||||
pid: string;
|
||||
user: string;
|
||||
cpu: string;
|
||||
mem: string;
|
||||
command: string;
|
||||
}> = [];
|
||||
try {
|
||||
const psOut = await execCommand(
|
||||
client,
|
||||
"ps aux --sort=-%cpu | head -n 11",
|
||||
);
|
||||
const psLines = psOut.stdout
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter(Boolean);
|
||||
if (psLines.length > 1) {
|
||||
for (let i = 1; i < Math.min(psLines.length, 11); i++) {
|
||||
const parts = psLines[i].split(/\s+/);
|
||||
if (parts.length >= 11) {
|
||||
topProcesses.push({
|
||||
pid: parts[1],
|
||||
user: parts[0],
|
||||
cpu: parts[2],
|
||||
mem: parts[3],
|
||||
command: parts.slice(10).join(" ").substring(0, 50),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const procCount = await execCommand(client, "ps aux | wc -l");
|
||||
const runningCount = await execCommand(
|
||||
client,
|
||||
"ps aux | grep -c ' R '",
|
||||
);
|
||||
totalProcesses = Number(procCount.stdout.trim()) - 1;
|
||||
runningProcesses = Number(runningCount.stdout.trim());
|
||||
} catch (e) {}
|
||||
|
||||
let hostname: string | null = null;
|
||||
let kernel: string | null = null;
|
||||
let os: string | null = null;
|
||||
try {
|
||||
const hostnameOut = await execCommand(client, "hostname");
|
||||
const kernelOut = await execCommand(client, "uname -r");
|
||||
const osOut = await execCommand(
|
||||
client,
|
||||
"cat /etc/os-release | grep '^PRETTY_NAME=' | cut -d'\"' -f2",
|
||||
);
|
||||
|
||||
hostname = hostnameOut.stdout.trim() || null;
|
||||
kernel = kernelOut.stdout.trim() || null;
|
||||
os = osOut.stdout.trim() || null;
|
||||
} catch (e) {}
|
||||
|
||||
const result = {
|
||||
cpu: { percent: toFixedNum(cpuPercent, 0), cores, load: loadTriplet },
|
||||
memory: {
|
||||
percent: toFixedNum(memPercent, 0),
|
||||
usedGiB: usedGiB ? toFixedNum(usedGiB, 2) : null,
|
||||
totalGiB: totalGiB ? toFixedNum(totalGiB, 2) : null,
|
||||
},
|
||||
disk: {
|
||||
percent: toFixedNum(diskPercent, 0),
|
||||
usedHuman,
|
||||
totalHuman,
|
||||
availableHuman,
|
||||
},
|
||||
network: {
|
||||
interfaces,
|
||||
},
|
||||
uptime: {
|
||||
seconds: uptimeSeconds,
|
||||
formatted: uptimeFormatted,
|
||||
},
|
||||
processes: {
|
||||
total: totalProcesses,
|
||||
running: runningProcesses,
|
||||
top: topProcesses,
|
||||
},
|
||||
system: {
|
||||
hostname,
|
||||
kernel,
|
||||
os,
|
||||
},
|
||||
cpu,
|
||||
memory,
|
||||
disk,
|
||||
network,
|
||||
uptime,
|
||||
processes,
|
||||
system,
|
||||
login_stats,
|
||||
};
|
||||
|
||||
metricsCache.set(host.id, result);
|
||||
@@ -1365,7 +1343,7 @@ function tcpPing(
|
||||
settled = true;
|
||||
try {
|
||||
socket.destroy();
|
||||
} catch {}
|
||||
} catch (error) {}
|
||||
resolve(result);
|
||||
};
|
||||
|
||||
@@ -1438,6 +1416,67 @@ app.post("/refresh", async (req, res) => {
|
||||
res.json({ message: "Polling refreshed" });
|
||||
});
|
||||
|
||||
app.post("/host-updated", async (req, res) => {
|
||||
const userId = (req as AuthenticatedRequest).userId;
|
||||
const { hostId } = req.body;
|
||||
|
||||
if (!SimpleDBOps.isUserDataUnlocked(userId)) {
|
||||
return res.status(401).json({
|
||||
error: "Session expired - please log in again",
|
||||
code: "SESSION_EXPIRED",
|
||||
});
|
||||
}
|
||||
|
||||
if (!hostId || typeof hostId !== "number") {
|
||||
return res.status(400).json({ error: "Invalid hostId" });
|
||||
}
|
||||
|
||||
try {
|
||||
const host = await fetchHostById(hostId, userId);
|
||||
if (host) {
|
||||
await pollingManager.startPollingForHost(host);
|
||||
res.json({ message: "Host polling started" });
|
||||
} else {
|
||||
res.status(404).json({ error: "Host not found" });
|
||||
}
|
||||
} catch (error) {
|
||||
statsLogger.error("Failed to start polling for host", error, {
|
||||
operation: "host_updated",
|
||||
hostId,
|
||||
userId,
|
||||
});
|
||||
res.status(500).json({ error: "Failed to start polling" });
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/host-deleted", async (req, res) => {
|
||||
const userId = (req as AuthenticatedRequest).userId;
|
||||
const { hostId } = req.body;
|
||||
|
||||
if (!SimpleDBOps.isUserDataUnlocked(userId)) {
|
||||
return res.status(401).json({
|
||||
error: "Session expired - please log in again",
|
||||
code: "SESSION_EXPIRED",
|
||||
});
|
||||
}
|
||||
|
||||
if (!hostId || typeof hostId !== "number") {
|
||||
return res.status(400).json({ error: "Invalid hostId" });
|
||||
}
|
||||
|
||||
try {
|
||||
pollingManager.stopPollingForHost(hostId, true);
|
||||
res.json({ message: "Host polling stopped" });
|
||||
} catch (error) {
|
||||
statsLogger.error("Failed to stop polling for host", error, {
|
||||
operation: "host_deleted",
|
||||
hostId,
|
||||
userId,
|
||||
});
|
||||
res.status(500).json({ error: "Failed to stop polling" });
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/metrics/:id", validateHostId, async (req, res) => {
|
||||
const id = Number(req.params.id);
|
||||
const userId = (req as AuthenticatedRequest).userId;
|
||||
|
||||
Reference in New Issue
Block a user