Log Reference
Every structured log entry across all modules. Levels: ERROR > WARN > INFO > DEBUG > TRACE. Entries marked AUDIT are security-auditable events for SIEM integration.
Reverse Proxy
Load Balancer
Pool Management: loadbalancer INFO pool created (strategy, backends count, health_check/circuit_breaker/outlier_detection enabled) loadbalancer INFO pool deleted loadbalancer INFO pool updated loadbalancer INFO backend added (pool_id, backend_id, address) loadbalancer INFO backend removed (pool_id, backend_id) loadbalancer INFO backend draining (pool_id, backend_id) loadbalancer WARN failed to initialize circuit state (pool_id, backend_id, error) loadbalancer WARN failed to initialize outlier state (pool_id, backend_id, error) loadbalancer ERROR failed to check pool existence (pool_id, error) loadbalancer ERROR failed to store pool config (pool_id, error) loadbalancer ERROR failed to get pool for deletion (pool_id, error) loadbalancer ERROR failed to delete pool (pool_id, error) loadbalancer ERROR failed to get pool (pool_id, error) loadbalancer ERROR failed to list pools (error) loadbalancer ERROR failed to get pool for update (pool_id, error) loadbalancer ERROR failed to update pool (pool_id, error) loadbalancer ERROR failed to get pool for add backend (pool_id, error) loadbalancer ERROR failed to add backend (pool_id, backend_id, error) loadbalancer ERROR failed to get pool for remove backend (pool_id, error) loadbalancer ERROR failed to remove backend (pool_id, backend_id, error) loadbalancer ERROR failed to get health state for drain (pool_id, backend_id, error) loadbalancer ERROR failed to update health state for drain (pool_id, backend_id, error)
Backend Selection: loadbalancer DEBUG backends excluded from selection (pool_id, total_backends, healthy_backends, excluded) loadbalancer DEBUG backend selected (pool_id, backend_id, strategy, healthy_backends, latency)
Health Checks: loadbalancer INFO backend health state changed — unhealthy to healthy (pool_id, backend_id, consecutive_ok, latency) loadbalancer WARN backend health state changed — healthy to unhealthy (pool_id, backend_id, consecutive_fails, error) loadbalancer DEBUG health check passed (pool_id, backend_id, consecutive_ok, latency) loadbalancer DEBUG health check failed (pool_id, backend_id, consecutive_fails, error) loadbalancer ERROR failed to store health state (pool_id, backend_id, error)
Circuit Breaker: loadbalancer INFO circuit breaker state changed (pool_id, backend_id, from_state, to_state, error_ratio) loadbalancer INFO per-protocol circuit breaker state changed (pool_id, backend_id, protocol, from_state, to_state, error_ratio) loadbalancer INFO circuit breaker reset (pool_id, backend_id, reset_by) loadbalancer WARN circuit breaker expression compilation failed (expression, error) loadbalancer WARN circuit breaker expression evaluation failed (expression, error) loadbalancer DEBUG circuit breaker threshold evaluation (combine_mode, conditions_met, error_ratio, error_threshold, p95_latency_ms, latency_threshold_ms, network_error_ratio, network_threshold) loadbalancer ERROR failed to store circuit state (pool_id, backend_id, error) loadbalancer ERROR failed to reset circuit (pool_id, backend_id, error)
Connection Tracking: loadbalancer ERROR failed to update connection count (pool_id, backend_id, error)
Rate Limiting: loadbalancer DEBUG rate limit exceeded (pool_id, user_id, limit, current_count, cost, retry_after) loadbalancer ERROR failed to update rate limit state (pool_id, key, error)
Outlier Detection: loadbalancer INFO backend ejected due to outlier detection (pool_id, backend_id, reason, ejection_count, duration, re_admit_at) loadbalancer INFO backend re-admitted after ejection period (pool_id, backend_id, total_ejections) loadbalancer INFO backend manually un-ejected (pool_id, backend_id) loadbalancer DEBUG outlier success rate analysis (pool_id, eligible_backends, avg_success_rate, stdev, threshold, stdev_factor) loadbalancer DEBUG outlier failure percentage analysis (pool_id, eligible_backends, threshold, ejected_count, max_ejectable) loadbalancer ERROR failed to save outlier state (pool_id, backend_id, error) loadbalancer ERROR failed to save outlier state on re-admission (pool_id, backend_id, error) loadbalancer ERROR failed to reset outlier interval stats (pool_id, backend_id, error)
DNS Discovery: loadbalancer INFO DNS discovery enabled (pool_id, hostname, refresh) loadbalancer INFO DNS discovery disabled (pool_id) loadbalancer INFO DNS discovery updated backends (pool_id, hostname, total_ips, added, removed) loadbalancer WARN DNS discovery resolution failed (pool_id, hostname, error) loadbalancer WARN DNS discovery returned no IPs (pool_id, hostname) loadbalancer WARN failed to add discovered backend (pool_id, ip, error) loadbalancer WARN failed to remove discovered backend (pool_id, ip, error)Request Shadow/Mirror
Dispatch Lifecycle: shadow.dispatch DEBUG Shadow request succeeded (shadow_name, status_code, latency_ms) shadow.dispatch WARN Shadow request failed with status (shadow_name, status_code, latency_ms) shadow.dispatch WARN Shadow request error (shadow_name, error_type, latency_ms, error)Reverse Proxy
Routing & Dispatch: proxy.dispatcher DEBUG Matched proxy route / no path match / no routes for hostname proxy.request DEBUG Proxying request to backend proxy.error INFO Request canceled (client disconnected — expected) proxy.error ERROR Proxy request failed (timeout, connection refused, etc.) proxy.redirect DEBUG Rewriting redirect Location header proxy.assets WARN Path traversal attempt / invalid path detected proxy.debug_timing DEBUG Full proxy roundtrip timing summary
Authentication & Authorization: proxy.reauth INFO AUDIT Re-authentication required (reauth rule matched) proxy.oidc DEBUG Redirecting to internal/external OIDC provider proxy.oidc ERROR Failed to generate PKCE verifier, CSRF token, or state encryption proxy.oidc.callback INFO AUDIT OIDC proxy authentication completed proxy.oidc.callback WARN AUDIT State expired, CSRF validation failed proxy.oidc.callback WARN OAuth error from IdP, state decryption failed, host mismatch proxy.oidc.callback ERROR AUDIT Token exchange failed proxy.oidc.callback ERROR Session creation failed
Header Signing: proxy.signing WARN Rotation interval too short / cluster key too short proxy.signing ERROR Key derivation failed (initial or rotation)
Bearer Token Injection: proxy.bearer_inject WARN No username/session, decryption failed (will re-mint) proxy.bearer_inject ERROR MintBearerToken failed, wrong response type, encryption error proxy.bearer_inject DEBUG Bearer token minted for backend proxy.bearer_refresh WARN Background refresh failed proxy.bearer_refresh DEBUG Background refresh completed
HTML Rewriting: proxy.rewrite WARN Response too large to buffer, streaming without rewrite proxy.rewrite DEBUG Chunked/binary response streaming, Brotli/Zstd disabled
WebSocket E2OE: proxy.ws_e2oe.* INFO Relay started / ended proxy.ws_e2oe.* ERROR Accept failed / backend dial failed
Session Monitoring: proxy.group_monitor INFO Monitor started, check completed proxy.group_monitor INFO AUDIT User groups changed, updating session proxy.group_monitor WARN Session update wait failed proxy.group_monitor ERROR Group fetch failed, session update failed
Lifecycle: proxy.init INFO Proxy service initialized proxy.init ERROR Initialization failed proxy.reload ERROR Reload failed proxy.ca_rotation INFO Transport caches invalidated (CA rotation) proxy.director WARN PROXY protocol: invalid source IP
Transport & DNS: proxy.transport DEBUG Transport configured for route proxy.dns DEBUG/INFO Backend DNS resolution, DNSSEC validation proxy.dns WARN/ERROR DNS resolution failed, fallback to system DNS proxy.dns.quic DEBUG/WARN QUIC-specific DNS resolution and connection
Circuit Breaker & Load Balancing: proxy.circuit_breaker WARN Circuit breaker open / fallback activated proxy.outlier_detection WARN Outlier detection config warnings proxy.dns_discovery WARN DNS discovery config warnings proxy.health_check WARN Health check config warnings proxy.fallback ERROR Invalid fallback URL / fallback service error
JIT-2FA: proxy.jit2fa ERROR JIT-2FA middleware creation failure
Request Signing: proxy.request_signing WARN Body hash failure proxy.request_signing DEBUG Request signed successfully proxy.signing_key DEBUG/WARN Public key endpoint access and validation proxy.signature_verify DEBUG/WARN Signature verification endpoint handler proxy.request_signature_verify DEBUG/WARN Request signature verification handler
Shadow/Mirror: proxy.shadow DEBUG Shadow request dispatched
Co-browsing: proxy.cobrowse.started INFO Co-browse session started proxy.cobrowse.stopped INFO Session stopped proxy.cobrowse.recorder_connected INFO Recorder WebSocket connected proxy.cobrowse.recorder_disconnected INFO Recorder disconnected proxy.cobrowse.reconnected INFO Recorder reconnected proxy.cobrowse.grace_expired INFO Cleanup grace period expired proxy.cobrowse.ws_upgrade_failed WARN WebSocket upgrade failed proxy.cobrowse.publish_failed WARN Event publish failed proxy.cobrowse.input_write_failed WARN Input forwarding failed proxy.cobrowse.recorder_ws_not_found WARN Recorder WS not found in cluster store proxy.cobrowse.input_received DEBUG Forwarding interaction event to recorder proxy.cobrowse.input_subscribe_failed WARN Input channel subscription failed proxy.cobrowse.recorder_stats INFO Recorder WebSocket session ended
Configuration: proxy.route INFO Route configured (full route details) proxy.config INFO Global config summary, cert loading proxy.config WARN Duplicate route detection, config validation
Access Control: proxy.access DEBUG Route access check (app, host, groups, reason)
Canary: proxy.canary DEBUG Routing to stable/canary backend (app, version, backend)
Retry: proxy.retry INFO Retrying request (app, attempt, backend) proxy.retry DEBUG Retry succeeded (app, attempt) proxy.retry WARN Retry budget exceeded (app, pool_id) proxy.retry WARN All retry attempts exhausted (app, max_attempts)
Hedge: proxy.hedge DEBUG Hedge fired (app, hedges, delay, primary_backend) proxy.hedge DEBUG Hedge skipped (app, reason) proxy.hedge WARN All hedge attempts failed (app, total_attempts)Authentication
Device Code Authorization
Init (module startup): devicecode.init INFO Device Code authorization disabled in config devicecode.init INFO Device Code authorization (RFC 8628) initialized
Authorize (code generation, RFC 8628 Section 3.1-3.2): devicecode.authorize ERROR Failed to generate device code devicecode.authorize ERROR Failed to generate user code devicecode.authorize ERROR Failed to store device code devicecode.authorize ERROR Failed to achieve quorum for device code storage devicecode.authorize WARN Failed to store user code reverse lookup devicecode.authorize INFO Device authorization codes generated
Verify (user code validation): devicecode.verify INFO Invalid user code format (not BASE20)
Complete (user authorization/denial): devicecode.complete INFO Device code already handled devicecode.complete ERROR Failed to generate tokens for device authorization devicecode.complete ERROR Failed to get token response devicecode.complete ERROR Invalid token response type devicecode.complete INFO Generated tokens for device authorization devicecode.complete ERROR Failed to broadcast authorization update devicecode.complete ERROR Failed to achieve quorum for authorization devicecode.complete WARN Concurrent modification detected (version mismatch) devicecode.complete INFO Device authorization completed
Poll (device code polling, RFC 8628 Section 3.4-3.5): devicecode.poll WARN Failed to lookup device code devicecode.poll WARN Client ID mismatch devicecode.poll DEBUG Client polling too fast devicecode.poll WARN Failed to replicate LastPoll update across cluster devicecode.poll WARN Failed to initiate LastPoll broadcast devicecode.poll INFO Device authorization denied by user devicecode.poll INFO Device authorization grantedJust-In-Time Two-Factor Authentication
Login Interception: jit2fa.intercept INFO AUDIT Login POST intercepted jit2fa.parse_error WARN Failed to extract credentials from login form jit2fa.credentials INFO AUDIT Credentials extracted from login form
Webhook Validation: jit2fa.validate_webhook DEBUG Validating credentials via webhook jit2fa.webhook INFO AUDIT Webhook validation successful / invalid credentials jit2fa.webhook ERROR AUDIT Webhook validation failed (HTTP error)
OTP: jit2fa.otp INFO AUDIT OTP sent successfully jit2fa.otp ERROR AUDIT Failed to send OTP jit2fa.otp.verify INFO AUDIT OTP verification successful / failed jit2fa.resend WARN AUDIT Failed to extend session expiry on resend
Session: jit2fa.session INFO AUDIT Authenticated session created (replay/header/two-phase/token_handoff) jit2fa.redirect INFO AUDIT No valid session, redirecting to login jit2fa.logout INFO AUDIT Logout intercepted, clearing session
Rate Limiting: jit2fa.ratelimit.status DEBUG Rate limit check passed jit2fa.ratelimit WARN Rate limit check failed (fail-open)
Token Handoff — Entry Path: jit2fa.handoff.entry INFO AUDIT Rejected: missing return_url query parameter jit2fa.handoff.entry WARN AUDIT Rejected: return_url not in allowed_return_urls jit2fa.handoff.entry INFO AUDIT Rejected: dpop_jkt malformed (charset or length) jit2fa.handoff.entry INFO AUDIT Rejected: require_dpop=true but caller did not supply dpop_jkt jit2fa.handoff.entry INFO AUDIT Valid URL, no session — redirecting to login (dpop_bound=true|false) jit2fa.handoff.entry INFO AUDIT Valid session — minting directly (fast path, dpop_bound=true|false)
Token Handoff — JKT Cookie: jit2fa.handoff.jkt_cookie WARN AUDIT Handoff JKT cookie failed revalidation (tampered or truncated)
Token Handoff — Mint Step: jit2fa.handoff.mint ERROR AUDIT Revalidation failed before mint (cookie tamper suspected) jit2fa.handoff.mint ERROR AUDIT Refusing to mint without username jit2fa.handoff.mint ERROR AUDIT require_dpop=true but no dpop_jkt reached finalize (caller bypassed entry) jit2fa.handoff.mint ERROR AUDIT return_url malformed after fragment strip (operator wildcard too permissive) jit2fa.handoff.mint ERROR AUDIT oidc.MintBearerToken call failed jit2fa.handoff.mint ERROR AUDIT oidc.MintBearerToken returned error jit2fa.handoff.mint INFO AUDIT Minted access token and redirecting caller (fields: username, audience, expires_in, dpop_bound, dpop_jkt?)
Token Handoff — Bearer Top-of-Tree Check: jit2fa.handoff.bearer INFO AUDIT Authorization header present but token is empty jit2fa.handoff.bearer ERROR AUDIT Validator call failed (oidc.ValidateIDToken hexdcall error) jit2fa.handoff.bearer WARN AUDIT Token rejected by validator (bad sig / expired / wrong issuer) jit2fa.handoff.bearer WARN AUDIT Audience mismatch (cross-mapping token replay attempt — alert signal) jit2fa.handoff.bearer INFO AUDIT require_dpop=true but token has no cnf.jkt (legacy client post-rollout) jit2fa.handoff.bearer INFO AUDIT DPoP-bound token but no DPoP header on request (client bug) jit2fa.handoff.bearer ERROR AUDIT oidc.ValidateDPoP hexdcall call failed jit2fa.handoff.bearer INFO AUDIT DPoP proof rejected by validator (stale iat / wrong htu / replayed jti) jit2fa.handoff.bearer WARN AUDIT DPoP proof thumbprint does not match token cnf.jkt — possible token theft jit2fa.handoff.bearer INFO AUDIT Accepted, forwarding to backend (fields: username, audience, dpop_bound, dpop_jkt?)
Token Handoff — DPoP Proof Validation: jit2fa.handoff.bearer.dpop INFO AUDIT DPoP proof validated, thumbprint matches token cnf.jkt (fields: username, dpop_jkt, htm, htu — one line per bearer-authenticated API call on a DPoP-bound mapping)
Token Handoff — Refresh: jit2fa.handoff.refresh INFO AUDIT Missing refresh_token parameter jit2fa.handoff.refresh INFO AUDIT Token rejected by validator (expired or invalid) jit2fa.handoff.refresh INFO AUDIT Audience mismatch (not a refresh token for this mapping) jit2fa.handoff.refresh INFO AUDIT Token not DPoP-bound jit2fa.handoff.refresh INFO AUDIT Missing DPoP proof header jit2fa.handoff.refresh INFO AUDIT DPoP proof rejected by validator jit2fa.handoff.refresh WARN AUDIT DPoP thumbprint mismatch — different key (abuse signal) jit2fa.handoff.refresh INFO AUDIT Token has no valid auth_time (cannot enforce session lifetime) jit2fa.handoff.refresh INFO AUDIT Absolute session lifetime exceeded (auth_time + max > now) jit2fa.handoff.refresh ERROR ValidateIDToken call failed (hexdcall error) jit2fa.handoff.refresh ERROR DPoP proof validation call failed (hexdcall error) jit2fa.handoff.refresh ERROR Failed to mint new access token jit2fa.handoff.refresh WARN Failed to mint rotated refresh token (returning access only) jit2fa.handoff.refresh INFO AUDIT Minted new token pair (success) (fields: username, audience, access_expires_in, session_remaining_hours, dpop_jkt)
Log level policy: - INFO+AUDIT for routine rejections caused by malformed client input (missing params, stale proofs, client-side bugs, rollout friction). These land in the audit stream for trace reconstruction but do not trigger operator alerts. - WARN+AUDIT only for events that indicate abuse or attack: open-redirect whitelist probing, signature forgery, cross-mapping replay attempts, DPoP thumbprint mismatches. Alert on these. - ERROR+AUDIT for internal system errors (hexdcall failures, signing key missing, cookie tamper on revalidation) that need operator investigation regardless of attack status.
The bearer "accepted" path fires per request on DPoP-boundmappings. On high-throughput SPAs hitting the backend at 50 rps,this can generate 50 audit lines per second per user per mapping.Filter at the log sink by event name + result if volume is aproblem — losing the accepted-path record at the emit site is asecurity regression, so the event is always emitted.
Full per-user audit trace pattern (grep): mapping_id=<ID> AND username=<user> AND event in {jit2fa.handoff.entry, jit2fa.handoff.mint, jit2fa.handoff.bearer, jit2fa.handoff.bearer.dpop}Kerberos Ticket Management & SPNEGO Browser SSO
SPNEGO (Browser SSO): kerberos.security WARN AUDIT SPNEGO token exceeds size limit kerberos.security INFO AUDIT SPNEGO auth successful / failed / decode failed / unmarshal failed kerberos.security ERROR AUDIT SPNEGO validated but no credentials in context kerberos.security WARN AUDIT SPNEGO auth for disabled user kerberos.spnego ERROR Failed to load keytab kerberos.spnego WARN User not found in directory / unexpected type / lookup failed kerberos.spnego WARN Keytab permissive permissions / missing service principal kerberos.spnego INFO Keytab loaded (from base64 or file)
Ticket Acquisition: kerberos.security INFO AUDIT Kerberos authentication successful kerberos.security INFO Kerberos authentication failed kerberos.acquire ERROR Failed to load krb5.conf
Ticket Access: kerberos.security INFO AUDIT Ticket access denied — invalid or expired session kerberos.write_file INFO AUDIT Created temporary ticket file
Ticket Lifecycle: kerberos.refresh INFO Ticket refreshed kerberos.refresh ERROR Failed to refresh ticket kerberos.revoke INFO Ticket revoked kerberos.revoke_user INFO User tickets revoked
Password Change: kerberos.security INFO Password change failed / successful / tickets revoked after change kerberos.password_change ERROR kpasswd pipe/start/write failures
Initialization: kerberos.init INFO Memory locking enabled kerberos.init WARN Memory locking failed — passwords may be swappedLDAP Authentication
ldapauth.authenticate DEBUG Empty username / empty password provided ldapauth.authenticate DEBUG Attempting LDAP bind ldapauth.authenticate INFO Bind successful / bind failed (invalid credentials) ldapauth.authenticate ERROR LDAP bind call failed (service error)Magic Link Authentication
Rate Limiting: magiclink.ratelimit.ip.status DEBUG Per-IP rate limit check passed magiclink.ratelimit.email.status DEBUG Per-email rate limit check passed
Initiate (magic link request): magiclink.initiate INFO Per-email rate limit exceeded magiclink.initiate ERROR Failed to create device code magiclink.initiate ERROR Failed to create magiclink session magiclink.initiate WARN Failed to dispatch magic link email magiclink.initiate INFO Magic link email queued
Poll (device code polling): magiclink.poll ERROR PollDeviceCode failed magiclink.poll ERROR Directory lookup failed during poll magiclink.poll INFO User invalid at poll time
PreVerify (read-only token validation): magiclink.preverify INFO Pre-verification successful, showing confirmation page
Verify (token consumption + action): magiclink.verify INFO Magic link denied by user magiclink.verify ERROR Directory lookup failed during verify magiclink.verify INFO Magic link signin_here — session on verifying device only magiclink.verify ERROR Failed to update device code authorization magiclink.verify INFO Magic link authorizedOIDC Provider
Authorization Code: oidc.authcode.generate INFO AUDIT Generating authorization code oidc.authcode.generate WARN AUDIT Rate limited / unknown client / invalid redirect URI oidc.authcode.generate WARN PKCE missing, unauthorized scope, IP not allowed oidc.auth ERROR RNG failure during code generation (critical)
Token Generation & Exchange: oidc.token.exchange INFO AUDIT Authorization code exchanged for tokens oidc.token.exchange WARN Invalid/expired code, PKCE failed, client/redirect mismatch oidc.tokens.generate INFO AUDIT Tokens issued successfully oidc.tokens.generate ERROR Token generation failed (signing key, RNG) oidc.tokens.saga ERROR Saga step failed during token storage oidc.token.refresh INFO AUDIT Token refresh requested oidc.token.refresh WARN Token not found, client mismatch, invalid scope oidc.tokens.refresh INFO AUDIT Tokens refreshed (internal) oidc.tokens.refresh WARN Refresh generation failed oidc.token.signing WARN Signing retry (threshold signer unavailable) oidc.token.signing ERROR All signing attempts failed oidc.ratelimit.status DEBUG Rate limit check result
ID Token: oidc.idtoken ERROR Signing key not loaded, signing failed oidc.idtoken DEBUG DPoP/cert binding applied, signer type
Crypto: oidc.crypto ERROR RNG failure in secure token generation (critical)
Introspection & Revocation: oidc.introspect DEBUG Token introspected (active true/false, type) oidc.revoke INFO AUDIT Token revoked oidc.revoke_user_tokens INFO Bulk user token revocation (account disable/delete)
Client Authentication & Validation: oidc.client_auth WARN Secret mismatch, JWT assertion failed, unknown method oidc.validation WARN Redirect URI invalid, wildcard rejected, entropy check oidc.pkce WARN Invalid verifier length/chars, plain method rejected oidc.pkce TRACE PKCE validation result
DPoP (RFC 9449): oidc.dpop WARN JTI replay detected oidc.dpop DEBUG Proof validation (htm/htu mismatch, expired, future) oidc.dpop.nonce WARN Nonce validation failed, storage error oidc.dpop.nonce DEBUG Nonce generated, validated, stored
PAR (RFC 9126): oidc.par INFO PAR request created oidc.par WARN Auth failed, request too large, replay attempt oidc.par ERROR Failed to generate request_uri
mTLS (RFC 8705): oidc.mtls WARN No certificate, CA mismatch, no identity fields oidc.mtls DEBUG SAN mismatch (URI/DNS/email/subject DN) oidc.mtls TRACE Client authenticated via matched method
M2M: oidc.client_credentials INFO AUDIT Access token generated oidc.jwt_bearer WARN Invalid JWT assertion
Keys & Init: oidc.init INFO OIDC provider initializing/disabled oidc.init ERROR Signing key validation failed (critical) oidc.keys INFO Key generated, threshold signing active oidc.keys WARN Threshold signer unhealthy/algorithm mismatch oidc.keys ERROR Key not configured, too short, low entropy oidc.key_history INFO Key history loaded/rotated oidc.key_history WARN Key history storage failed oidc.jwks DEBUG JWKS requested oidc.jwks WARN Unknown client requesting JWKS
UserInfo: oidc.userinfo INFO AUDIT UserInfo served oidc.userinfo WARN Token invalid, user not found, scope insufficient
Bearer Token Minting: oidc.mint_bearer INFO AUDIT Bearer token minted for proxy oidc.mint_bearer ERROR Minting failed (signing key, invalid request)
DCR (Dynamic Client Registration): oidc.dcr INFO AUDIT Dynamic client registered
PAT (Personal Access Tokens): oidc.pat.issue INFO AUDIT PAT issued oidc.pat.issue ERROR Signing key not loaded, signing/session failed
Token Validation: oidc.validate_id_token INFO ID token validated
Device Code: oidc.device_code INFO Generating tokens for device authorization oidc.device_code INFO AUDIT Device code grant successful oidc.device_code ERROR Token generation failed
Logout: oidc.logout INFO AUDIT Logout completed, tokens revoked
Health: oidc.healthcheck DEBUG Health check performedEmail OTP
Generate (OTP creation and delivery): otp.generate INFO AUDIT Email domain not allowed otp.generate INFO Device ID missing otp.generate INFO AUDIT Device already has OTP for different email otp.generate INFO AUDIT OTP resend blocked - max retries exceeded otp.generate DEBUG OTP resend denied - too soon otp.generate DEBUG Generating BASE20 OTP (consonants only) otp.generate DEBUG Generating numeric OTP otp.generate WARN Invalid UserpassOTPType configuration, defaulting to numeric otp.generate ERROR Failed to generate OTP code otp.generate ERROR Invalid OTP TTL configuration otp.generate ERROR Failed to broadcast OTP to cluster otp.generate ERROR Failed to achieve quorum for OTP storage otp.generate DEBUG OTP stored with cluster quorum otp.generate INFO AUDIT OTP code generated otp.generate WARN Failed to send OTP email
Validate (OTP code verification): otp.validate ERROR Failed to query OTP from storage otp.validate ERROR Failed to retrieve OTP otp.validate DEBUG No OTP found otp.validate ERROR Invalid OTP type in storage otp.validate DEBUG OTP validation attempt otp.validate INFO AUDIT OTP validation rejected - OTP is locked otp.validate ERROR Failed to delete expired OTP otp.validate INFO AUDIT OTP code expired otp.validate ERROR Failed to lock OTP after max retries exceeded otp.validate WARN AUDIT SECURITY: OTP locked due to max retry attempts exceeded otp.validate ERROR Failed to update OTP retry count otp.validate ERROR Failed to achieve quorum for OTP retry update otp.validate INFO AUDIT Invalid OTP code submitted otp.validate ERROR Failed to delete OTP after validation otp.validate DEBUG OTP deleted after successful validation otp.validate INFO AUDIT OTP validated and removed (replay prevention) otp.validate INFO AUDIT OTP validated successfully
Domain Check: otp.domain TRACE Invalid email format otp.domain TRACE Domain allowed otp.domain TRACE Domain not in allowed listTOTP Authenticator
Enroll (secret + QR generation): totp.enroll ERROR Failed to generate TOTP secret totp.enroll ERROR Failed to generate QR code totp.enroll INFO TOTP enrollment initiated
ConfirmEnroll (first-code verification and secret persistence): totp.enroll.confirm INFO TOTP enrollment verification failed - invalid code totp.enroll.confirm ERROR Failed to generate recovery codes totp.enroll.confirm ERROR Failed to store TOTP secret totp.enroll.confirm INFO TOTP enrollment confirmed and persisted
Validate (TOTP code verification): totp.validate INFO AUDIT TOTP validation failed - no enrollment found totp.validate ERROR AUDIT Failed to decode stored TOTP secret totp.validate INFO AUDIT TOTP validation failed - invalid code totp.validate WARN AUDIT Clock backward detected during TOTP validation - allowing code totp.validate WARN AUDIT TOTP replay detected - code already used totp.validate ERROR AUDIT Failed to update last used step - rejecting for safety totp.validate INFO AUDIT TOTP validation successful
Recovery (one-time recovery code validation): totp.recovery INFO Recovery code validation failed - no enrollment found totp.recovery INFO Recovery code validation failed - no codes remaining totp.recovery INFO Recovery code validation failed - invalid code totp.recovery ERROR Failed to consume recovery code - rejecting for safety totp.recovery INFO Recovery code validated and consumed
Delete (enrollment removal): totp.delete INFO No TOTP enrollment found to delete totp.delete INFO TOTP enrollment deletedWebAuthn Passkeys
Registration: webauthn.registration INFO AUDIT Begin/finish registration request webauthn.registration INFO Passkey registered / attestation validated webauthn.registration WARN Challenge mismatch / origin mismatch / attestation failed webauthn.registration ERROR Challenge generation / session storage / marshal failures
Authentication: webauthn.authentication INFO AUDIT New challenge issued webauthn.authentication ERROR AUDIT E2OE commitment mismatch — Tier 1 binding rejected webauthn.authentication INFO Auth successful / passkey not found / expired / invalid session webauthn.authentication WARN Origin mismatch / RP ID hash mismatch / signature verification failed webauthn.authentication ERROR ECDH keygen / challenge generation / session storage / cloned device / COSE key failures webauthn.authentication DEBUG Begin/finish request trace / counter validation / auth successful
Enrollment: webauthn.enroll INFO AUDIT Passkey enrolled (hash, device, active count) webauthn.enroll ERROR Failed to load existing passkeys / failed to store webauthn.enroll DEBUG Enroll request
Revocation: webauthn.revoke INFO AUDIT Passkey revoked (hash, device, reason, revoked_by) webauthn.revoke WARN No passkeys found / passkey not found in active list webauthn.revoke ERROR Failed to store revoked passkey webauthn.revoke DEBUG Revoke request
Storage: webauthn.storage DEBUG Loading/storing passkeys (active/revoked counts) webauthn.storage INFO Passkeys stored to moduledata
Expiration: webauthn.expiration INFO Check started / completed / reminder sent / disabled / skipping webauthn.expiration WARN Lock acquisition failed webauthn.expiration ERROR Scheduler registration / LoadAll / GetAllUsers failures
Initialization: webauthn.init INFO Provider initialized (RPID, origin, type, validity) / disabled webauthn.init ERROR Initialization failed
Lookup: webauthn.get DEBUG Passkey lookup webauthn.list DEBUG Passkey listingX.509 Client Certificate Authentication
Init & Lifecycle: x509.init WARN JetStream temporarily unavailable, retrying serial index rebuild x509.init ERROR Failed to rebuild serial index after retries x509.init ERROR Failed to initialize CRL x509.init INFO X.509 authentication enabled (CRL disabled) x509.cleanup INFO AUDIT X.509 module cleanup complete
Validate (certificate authentication pipeline): x509.validate ERROR Failed to parse DER certificate x509.validate WARN Certificate not yet valid / Certificate expired x509.validate ERROR No CA certificates available (config + ACME bundle empty) x509.validate WARN Certificate chain validation failed x509.validate WARN Failed to extract identity from certificate x509.validate ERROR Directory lookup failed x509.validate WARN User not found in directory x509.validate WARN Failed to check serial index, falling back to moduledata x509.validate ERROR Failed to check moduledata revocation x509.validate WARN Internal certificate revoked / not in registry - rejecting x509.validate WARN OCSP check failed x509.validate INFO Certificate validated successfully x509.validate DEBUG Validation stage progress (expiration, chain, CRL, identity, OCSP)
Enroll (internal CA certificate issuance): x509.enroll INFO Starting certificate enrollment x509.enroll WARN Invalid username format / Failed to load existing certificate x509.enroll ERROR Failed to enforce certificate limit / generate keypair x509.enroll ERROR Failed to sign certificate with CA / get CA bundle x509.enroll ERROR Failed to generate PKCS#12 password / build PKCS#12 bundle x509.enroll ERROR Failed to store certificate record x509.enroll WARN Failed to store serial index x509.enroll INFO AUDIT Certificate enrolled successfully
Revoke: x509.revoke INFO Revoking certificate x509.revoke WARN Failed to update serial index x509.revoke INFO AUDIT Certificate revoked successfully
Revoke By Serial (self-service): x509.revokeBySerial INFO Revoking certificate by serial x509.revokeBySerial WARN Failed to update serial index x509.revokeBySerial INFO AUDIT Certificate revoked by serial
Revoke All & Enforce Max: x509.revokeAll WARN Failed to update serial index x509.revokeAll INFO AUDIT Revoked certificates for user x509.enforceMax WARN Failed to update serial index x509.enforceMax INFO AUDIT Revoked oldest cert for user (max reached)
CRL: x509.crl.init ERROR Failed to download CRL from any server x509.crl.init INFO CRL loaded successfully x509.crl WARN CRL download failed, trying next URL x509.crl.refresh ERROR Failed to refresh CRL from any server x509.crl.refresh INFO CRL refreshed successfully x509.crl.refresh DEBUG Refreshing CRL x509.crl.rebuild WARN Failed to trigger CRL rebuild
OCSP: x509.ocsp DEBUG OCSP cache hit / cache miss - querying responder(s) x509.ocsp WARN No OCSP URLs configured and certificate has no AIA OCSP extension x509.ocsp WARN OCSP responder failed, trying next x509.ocsp.check WARN All OCSP responders unreachable (soft-fail enabled, allowing authentication) x509.ocsp.check ERROR All OCSP responders unreachable (hard-fail enabled, blocking authentication) x509.ocsp.check DEBUG OCSP query successful x509.ocsp.serial WARN OCSP cache lookup failed / cache wait failed x509.ocsp.serial DEBUG OCSP cache miss for session extension check / OCSP cache hit
Auto-Renewal: x509.renewal INFO Auto-renewal is disabled by configuration x509.renewal ERROR Failed to schedule auto-renewal x509.renewal INFO Auto-renewal scheduler registered x509.renewal WARN Failed to acquire renewal lock / wait for lock acquisition x509.renewal INFO Renewal check already in progress on another node, skipping x509.renewal INFO Starting certificate renewal check x509.renewal ERROR Failed to get all users / GetAllUsers failed / Invalid response x509.renewal ERROR Failed to renew certificate x509.renewal INFO Certificate renewal check completed x509.renewal WARN Skipping renewal - user has no email / no CertificateDER stored x509.renewal WARN Failed to enforce max certs limit x509.renewal WARN Failed to update serial index / get CA bundle / send renewal email x509.renewal INFO Certificate renewed successfully
Session Extension Validator: x509.session_validator DEBUG Checking certificate revocation for session extension x509.session_validator WARN AUDIT X.509 session missing required metadata - allowing extension x509.session_validator WARN Failed to check serial index, falling back to moduledata x509.session_validator WARN AUDIT Session extension rejected: internal certificate revoked x509.session_validator WARN Session extension rejected: internal certificate not in registry x509.session_validator WARN Session extension rejected: external certificate revoked (OCSP/cache) x509.session_validator WARN Soft-fail warnings (revocation check, OCSP, cert parse failures) x509.session_validator WARN OCSP check failed, rejecting extension (hard-fail) x509.session_validator WARN Unknown CA type in session metadata - allowing extension
Revocation Check (hexdcall operation): x509.check_revoked DEBUG Checking certificate revocation status / valid / OCSP passed x509.check_revoked WARN Failed to check serial index / not in registry / no cert DER x509.check_revoked INFO Internal certificate is revoked / External revoked (OCSP) x509.check_revoked ERROR Failed to parse certificate DER x509.check_revoked WARN OCSP check failed for external cert
Recovery (serial index rebuild at startup): x509.recovery INFO Starting serial index recovery from moduledata x509.recovery WARN Invalid x509 data format for user x509.recovery WARN Failed to store serial index for legacy/active/revoked cert x509.recovery INFO Serial index recovery completed / cancelled during shutdown
Storage: x509.storage INFO X509 certificate stored to moduledata x509.storage DEBUG Load/store operations, format parsing
Auto-Renew Opt-Out: x509.auto_renew INFO Auto-renewal opt-out updated
Revoked Certificates Query: x509.revoked ERROR Failed to retrieve serial index x509.revoked INFO Retrieved revoked certificates x509.revoked DEBUG Retrieving all revoked certificatesRADIUS Authentication (RADSEC + UDP)
Initialization: radius.init INFO RADIUS service disabled in configuration radius.init INFO RADIUS initialization starting (RADSEC TCP+TLS)... radius.init INFO RADIUS initialization starting (dual-mode: UDP + RADSEC TCP+TLS)... radius.init INFO Waiting for LDAP service to initialize radius.init INFO Shutdown requested during LDAP wait, aborting initialization radius.init INFO LDAP service ready, creating RADIUS server radius.init INFO Shutdown requested before server creation, aborting initialization radius.init ERROR Failed to create RADIUS server radius.init INFO Shutdown requested before listener creation radius.init ERROR Failed to resolve network interface IP radius.init INFO Resolved network interface for RADIUS radius.init ERROR Failed to create RADSEC listener radius.init ERROR Failed to start RADSEC listener radius.init INFO RADSEC listener started radius.init ERROR Failed to create UDP RADIUS listener radius.init ERROR Failed to start UDP RADIUS listener radius.init INFO UDP RADIUS listener started radius.init INFO RADIUS server started successfully radius.init WARN RADIUS auth_methods includes x509 but [authentication.x509] is not enabled — x509 auth will fail at runtime
Connection handling: radius.handler ERROR No TLS configuration available radius.handler WARN TLS handshake failed radius.handler INFO HXEP resolved real NAS IP radius.handler ERROR Rejecting HXEP connection — NAS has per-client mTLS (client_ca_pem) which cannot be enforced through edge proxy radius.handler WARN AUDIT Unknown NAS — connection from unregistered IP radius.handler DEBUG RADSEC connection established
UDP listener: radius.handler WARN UDP temporary read error, continuing radius.handler ERROR UDP fatal read error, stopping listener
RADSEC framing: radius.handler WARN Failed to read RADSEC frame header radius.handler WARN Invalid RADIUS packet length radius.handler WARN Incomplete RADSEC frame
Packet processing: radius.handler WARN AUDIT NAS rate limit exceeded radius.handler WARN AUDIT Concurrent authentication limit reached radius.handler WARN Failed to parse RADIUS packet radius.handler WARN Unexpected RADIUS packet code radius.handler INFO Missing User-Name attribute in Access-Request radius.handler WARN AUDIT User locked out
Authentication: radius.auth DEBUG Skipping x509 auth — no client certificate radius.auth ERROR x509auth bridge call failed radius.auth ERROR x509auth validation timed out or failed radius.auth INFO AUDIT Certificate validation rejected radius.auth INFO AUDIT Authentication failed radius.auth ERROR Authorization failed radius.auth INFO No matching mapping radius.auth INFO Authentication and authorization successful
MFA: radius.mfa WARN TOTP status check failed radius.mfa ERROR Failed to generate challenge token radius.mfa INFO MFA validated via recovery code radius.mfa ERROR Failed to encode Access-Challenge radius.mfa WARN Failed to send Access-Challenge radius.mfa ERROR Failed to get user info for MFA check radius.mfa ERROR AUDIT MFA method resolution failed radius.mfa INFO MFA skipped — no method available, skip_if_unavailable=true radius.mfa ERROR Failed to send email OTP radius.mfa INFO AUDIT Sending MFA challenge radius.mfa WARN Invalid or expired MFA challenge state radius.mfa WARN MFA challenge response from different NAS radius.mfa INFO MFA challenge response missing verification code radius.mfa INFO MFA validation failed radius.mfa ERROR Authorization failed after MFA radius.mfa INFO MFA authentication and authorization successful
Response encoding: radius.handler ERROR Failed to encode Access-Reject radius.handler WARN Failed to send Access-Reject radius.handler WARN Failed to set RADIUS attribute radius.handler ERROR Failed to encode Access-Accept radius.handler WARN Failed to send Access-Accept
Session recording: radius.session WARN Failed to create RADIUS session
Restrictions: radius.restrictions.geo ERROR Geo check failed - denying access (fail-closed) radius.restrictions.geo ERROR Geo check wait failed - denying access (fail-closed) radius.restrictions.geo ERROR Invalid geo check response type - denying access (fail-closed) radius.restrictions.geo INFO Access blocked by geo restriction radius.restrictions.time ERROR Time check failed - denying access (fail-closed) radius.restrictions.time ERROR Time check wait failed - denying access (fail-closed) radius.restrictions.time ERROR Invalid time check response type - denying access (fail-closed) radius.restrictions.time INFO Access blocked by time restrictionOnboarding Service
Init (route registration): onboarding.init INFO Onboarding disabled (console not enabled) onboarding.init INFO Onboarding service route registered at /onboarding
MFA Session (passkey enrollment session lifecycle): onboarding.mfa_session ERROR Failed to create mfa_pending session for passkey enrollment onboarding.mfa_session ERROR Invalid session response type
Passkey (enrollment flow): onboarding.passkey INFO Onboarding: authenticated user entering passkey enrollment AUDITSign-In Service
Authentication completion: signin.complete INFO Authentication completed
Finalize (session creation after successful auth): signin.finalize ERROR AUDIT Failed to create session signin.success INFO AUDIT User signed in successfully
Reauth (re-authentication session for protected proxy paths): signin.reauth ERROR Failed to create reauth session signin.reauth ERROR Unexpected reauth session response type signin.reauth INFO AUDIT Reauth session created during signin
LDAP password authentication: signin.ldap INFO AUDIT Attempting LDAP authentication signin.ldap ERROR LDAP bind call failed signin.ldap DEBUG LDAP bind successful, syncing user from directory signin.ldap WARN Failed to sync user from directory signin.ldap WARN User sync returned failure signin.ldap ERROR Failed to get user from directory signin.ldap INFO User not found in directory after sync signin.ldap INFO AUDIT Account is disabled signin.ldap INFO Password expired - creating temporary session for password change signin.ldap ERROR Failed to create password_expired session
MFA (multi-factor authentication flow): signin.mfa INFO AUDIT MFA required for user signin.mfa DEBUG Validating MFA session signin.mfa ERROR Session validation wait failed signin.mfa INFO MFA session not valid signin.mfa DEBUG MFA session validated successfully
MFA post-verification: signin.mfa DEBUG MFA verified - retrieving pending session signin.mfa.session ERROR Failed to wait for MFA session validation signin.mfa.session DEBUG MFA session retrieved - creating authenticated session signin.mfa.signup INFO MFA verified for signup - redirecting to passkey registration signin.mfa.groups WARN Directory lookup failed after MFA - using cached groups from pending session signin.mfa.complete DEBUG Returning success response to client
MFA OTP resend: signin.mfa.resend ERROR Failed to generate OTP signin.mfa.resend INFO OTP code resent
MFA email OTP verification: signin.mfa.otp ERROR OTP validation call failed signin.mfa.otp INFO AUDIT OTP validation failed signin.mfa.otp WARN OTP generation failed — user can resend from MFA page
MFA TOTP verification: signin.mfa.totp ERROR TOTP validation call failed signin.mfa.totp INFO AUDIT TOTP and recovery code validation both failed signin.mfa.totp INFO AUDIT TOTP validation failed - invalid code signin.mfa.totp INFO AUDIT User authenticated via recovery code signin.mfa.totp ERROR Failed to check TOTP enrollment status
WebAuthn passkey authentication: signin.passkey.begin DEBUG Beginning passkey authentication signin.passkey.begin ERROR BeginAuthentication failed signin.passkey.begin DEBUG WebAuthn challenge created signin.passkey.finish DEBUG Finishing passkey authentication signin.passkey.finish INFO FinishAuthentication failed signin.passkey.finish ERROR Failed to get user from directory signin.passkey.finish INFO User not found in directory after passkey auth signin.passkey.finish INFO Account is disabled signin.passkey.finish ERROR AUDIT E2OE: failed to persist Tier 1 ECDH state — channel will degrade to baseline
Kerberos SPNEGO authentication: signin.kerberos DEBUG Sending Negotiate challenge signin.kerberos ERROR AUDIT SPNEGO validation call failed signin.kerberos INFO AUDIT SPNEGO authentication failed signin.kerberos ERROR AUDIT Failed to create session for SPNEGO user signin.kerberos ERROR Invalid session create response signin.kerberos INFO AUDIT Kerberos SPNEGO authentication successful
Magic link passwordless authentication: signin.magiclink ERROR AUDIT Initiate failed signin.magiclink.verify INFO AUDIT Magic link verified signin.magiclink.verify ERROR Failed to finalize authentication
X.509 certificate authentication: signin.x509 DEBUG X.509 signin handler started signin.x509 INFO No client certificate provided signin.x509 ERROR Failed to validate certificate signin.x509 INFO AUDIT Certificate revoked signin.x509 INFO AUDIT Certificate expired signin.x509 INFO Certificate not yet valid signin.x509 INFO Certificate chain validation failed signin.x509 ERROR Certificate validation failed signin.x509 INFO Certificate validation failed signin.x509 DEBUG Capping session TTL to certificate validity signin.x509 ERROR Failed to create session signin.x509 ERROR Session creation timeout signin.x509 ERROR Invalid session response signin.x509 INFO AUDIT X.509 authentication successfulIdentity & Directory
Directory Cache
Init (module startup): directory.init INFO Directory service disabled - no LDAP configured directory.init INFO Waiting for LDAP connection pool to initialize directory.init INFO Initializing directory service directory.init INFO Cluster and memory storage ready, starting initial sync directory.init ERROR Initial sync failed directory.init INFO Directory service initialized
Callback registration: directory.callback INFO Registered user updated callback directory.callback INFO Registered user disabled callback
Full sync (periodic and on-demand): directory.sync.full INFO Full sync loop started directory.sync.full ERROR Full sync failed directory.sync.full INFO Starting full sync from LDAP directory.sync.full ERROR Failed to call LDAP GetAllUsers directory.sync.full ERROR Failed to get users from LDAP directory.sync.full WARN (dynamic license enforcement message) directory.sync.full INFO Retrieved users from LDAP directory.sync.full INFO Retrieved groups from LDAP directory.sync.full INFO Syncing users and groups to cluster storage directory.sync.full WARN Failed to store user directory.sync.full WARN Failed to store group directory.sync.full INFO Full sync completed
Delta sync (periodic incremental): directory.sync.delta INFO Delta sync loop started directory.sync.delta ERROR Delta sync failed directory.sync.delta INFO Starting delta sync from LDAP directory.sync.delta WARN (dynamic license enforcement message) directory.sync.delta DEBUG Retrieved modified users from LDAP directory.sync.delta DEBUG Retrieved modified groups from LDAP directory.sync.delta DEBUG No changes detected directory.sync.delta INFO Syncing modified objects to cluster storage directory.sync.delta INFO Delta sync completed
Single-user sync (on-demand): directory.syncuser ERROR Failed to call LDAP GetUser directory.syncuser ERROR Failed to get LDAP response directory.syncuser ERROR Invalid LDAP response type directory.syncuser DEBUG User not found in LDAP directory.syncuser ERROR Failed to broadcast cache update directory.syncuser WARN Cache update had errors directory.syncuser INFO User synced successfully
Admin: directory.admin INFO Manual full sync requested
Index maintenance (OnUserSet / OnUserDelete callbacks): directory.index WARN Failed to update email index directory.index WARN Failed to update user-groups index directory.index WARN Failed to update group-members index directory.index WARN Failed to update disabled index directory.index INFO User disabled, revoking OIDC tokens and sessions directory.index WARN Failed to initiate OIDC token revocation directory.index WARN Failed to initiate session revocation directory.index DEBUG Calling user disabled callback via hexdcall directory.index WARN Failed to call user disabled callback directory.index WARN Failed to call user updated callback directory.index WARN Failed to remove from email index directory.index WARN Failed to remove from user-groups index directory.index WARN Failed to remove from disabled index
Bulk index (after full sync): directory.index.bulk INFO Bulk indexes builtLDAP Provider
Initialization: ldap.init INFO LDAP provider disabled - no URL configured ldap.init INFO Initializing LDAP connection pool ldap.init ERROR Failed to initialize LDAP connection pool ldap.init INFO LDAP provider initialized and ready
Connection Pool: ldap.pool DEBUG Initializing connection pool ldap.pool DEBUG Creating connection N/M (per-connection progress) ldap.pool DEBUG Connection N/M created successfully ldap.pool INFO Connection pool initialized successfully ldap.pool WARN Transient error, retrying in Xs (attempt N) ldap.pool ERROR Permanent error during connection - refusing to start ldap.pool ERROR Exceeded max retry duration - refusing to start
Connection Lifecycle: ldap.conn DEBUG Using custom CA certificate / Using system CA certificates ldap.conn DEBUG Attempting to connect with HA failover ldap.conn DEBUG Dialing LDAP URL ldap.conn DEBUG Successfully connected ldap.conn DEBUG Binding with service account ldap.conn DEBUG Successfully bound with service account ldap.conn WARN LDAP server failed, trying next ldap.conn ERROR Failed to dial LDAP ldap.conn ERROR Failed to bindOIDC Relying Party
Init (module startup): identity.oidc.init DEBUG No OIDC providers configured, module inactive identity.oidc.init ERROR AUDIT Invalid OIDC provider configuration identity.oidc.init INFO OIDC RP module initialized
Authorize (build authorization URL): identity.oidc.authorize DEBUG Building authorization URL identity.oidc.authorization INFO AUDIT Authorization URL built identity.oidc.authorization WARN Failed to delete auth session
Callback (authorization code callback): identity.oidc.callback WARN AUDIT IdP returned error identity.oidc.callback WARN AUDIT State validation failed identity.oidc.callback DEBUG Processing authorization callback
Discovery (OIDC discovery metadata): identity.oidc.discovery DEBUG Fetching discovery metadata identity.oidc.discovery WARN Dev mode enabled - endpoint validation relaxed identity.oidc.discovery INFO AUDIT Discovery metadata fetched and validated identity.oidc.discovery WARN Failed to cache discovery metadata identity.oidc.discovery WARN Invalid cached metadata type
JWKS (JSON Web Key Set): identity.oidc.jwks DEBUG Fetching JWKS identity.oidc.jwks INFO AUDIT JWKS fetched identity.oidc.jwks WARN Failed to cache JWKS
Token (exchange, refresh, revocation, introspection): identity.oidc.token DEBUG Exchanging code for tokens identity.oidc.token INFO AUDIT Token exchange successful identity.oidc.token DEBUG Refreshing access token identity.oidc.token INFO AUDIT Token refresh successful identity.oidc.token WARN AUDIT Provider does not support token revocation identity.oidc.token INFO AUDIT Token revocation acknowledged identity.oidc.token DEBUG Token introspection completed
Validate ID Token: identity.oidc.validate_id_token DEBUG Validating ID token identity.oidc.validate_id_token WARN AUDIT ID token validation failed
UserInfo (fetch user claims): identity.oidc.userinfo DEBUG Fetching user info from external IdP identity.oidc.userinfo DEBUG Fetching user info identity.oidc.userinfo INFO AUDIT User info fetched
DPoP (Demonstration of Proof-of-Possession): identity.oidc.dpop WARN AUDIT Failed to check/store DPoP JTI identity.oidc.dpop WARN DPoP JTI SetNX wait failed identity.oidc.dpop ERROR Unexpected SetNX response type
PAR (Pushed Authorization Requests): identity.oidc.par DEBUG Pushing authorization request to IdP identity.oidc.par WARN AUDIT PAR endpoint returned error with unparseable body identity.oidc.par WARN Non-standard request_uri format from IdP identity.oidc.par WARN PAR expires_in missing, using default identity.oidc.par WARN PAR expires_in outside RFC 9126 recommended range identity.oidc.par INFO AUDIT PAR request successful identity.oidc.par WARN Discovery failed, falling back to standard authorization identity.oidc.par DEBUG PAR not supported, using standard authorization identity.oidc.par WARN PAR request failed, falling back to standard authorization identity.oidc.par INFO Authorization URL built with PAR
Provider info: identity.oidc.get_provider DEBUG Fetching provider metadata identity.oidc.list_providers DEBUG Listed OIDC providers
Health: identity.oidc.health_check DEBUG Health check completed
Refresh (entry point): identity.oidc.refresh DEBUG Refreshing access token
Revoke (entry point): identity.oidc.revoke DEBUG Revoking token with external IdP
Introspect (entry point): identity.oidc.introspect DEBUG Introspecting token with external IdPSCIM Identity Provider
Init (module startup): identity.scim.init INFO SCIM provider disabled - no providers configured identity.scim.init INFO Initializing SCIM provider identity.scim.init ERROR Failed to initialize SCIM provider identity.scim.init INFO SCIM provider initialized identity.scim.init INFO SCIM identity provider ready
Hexdcall operations: identity.scim.sync_all DEBUG Starting sync identity.scim.sync INFO Sync completed identity.scim.get_sync_status DEBUG Getting sync status identity.scim.get_all_users DEBUG Getting all users identity.scim.get_all_users ERROR Failed to list users identity.scim.get_all_users INFO Retrieved users identity.scim.get_all_groups DEBUG Getting all groups identity.scim.get_all_groups ERROR Failed to list groups identity.scim.get_all_groups INFO Retrieved groups identity.scim.get_user DEBUG Getting user identity.scim.get_group DEBUG Getting group identity.scim.health_check DEBUG Checking health identity.scim.process_webhook DEBUG Processing webhook
SCIM client (HTTP communication): scim.client.list DEBUG Starting paginated user list scim.client.list DEBUG Starting paginated group list scim.client.list WARN Pagination safety limit reached scim.client.list INFO Completed paginated user list scim.client.list INFO Completed paginated group list scim.client.retry WARN Retrying request scim.client.oauth2 DEBUG Refreshing OAuth2 token scim.client.oauth2 INFO OAuth2 token refreshed
Sync orchestrator: identity.scim.sync INFO Starting full sync identity.scim.sync INFO AUDIT Full sync completed identity.scim.sync INFO Starting incremental sync identity.scim.sync INFO Incremental sync completed
Background sync manager: identity.scim.sync INFO Starting background sync manager identity.scim.sync ERROR Initial sync failed for provider identity.scim.sync INFO Initial sync completed identity.scim.sync.delta INFO Delta sync loop started identity.scim.sync.delta INFO Delta sync loop stopping identity.scim.sync.delta INFO No previous sync time, falling back to full sync identity.scim.sync.delta ERROR Delta sync failed identity.scim.sync.delta INFO Delta sync completed identity.scim.sync.full INFO Full sync loop started identity.scim.sync.full INFO Full sync loop stopping identity.scim.sync.full ERROR Full sync failed identity.scim.sync.full INFO Full sync completed identity.scim.sync.full ERROR Cumulative 24h deletion threshold exceeded identity.scim.sync.full WARN Per-sync deletion threshold exceeded identity.scim.sync.full WARN Cannot get client for current state - treating as initial sync
Circuit breaker: identity.scim.sync ERROR Circuit breaker opened - provider disabled after consecutive failures identity.scim.sync INFO Circuit breaker closed - provider recovered identity.scim.sync INFO Circuit breaker manually reset identity.scim.sync.delta WARN Skipping delta sync - circuit open identity.scim.sync.full WARN Skipping full sync - circuit open
Deprovisioning: identity.scim.deprovisioning ERROR AUDIT Deletion threshold exceeded - blocking hard deletions identity.scim.deprovisioning ERROR Deletion requested with zero current users - blocking identity.scim.deprovisioning WARN AUDIT Disabling user identity.scim.deprovisioning WARN AUDIT Deleting user identity.scim.deprovisioning WARN Deleting group
Nested group resolution: identity.scim.nested WARN Max groups per user reached, truncating identity.scim.nested WARN Max nesting depth reached identity.scim.flatten WARN Max nesting depth reached during flattening
Multi-provider merge: identity.scim.merge WARN Skipping user with invalid username identity.scim.merge INFO Merge completed with conflicts identity.scim.merge WARN Skipping group with invalid name identity.scim.merge WARN Group membership truncated
Webhook processing: identity.scim.webhook ERROR Webhook rejected: no webhook_secret configured for provider identity.scim.webhook WARN Webhook payload exceeds size limit identity.scim.webhook WARN Webhook signature verification failed identity.scim.webhook WARN Failed to parse webhook payload identity.scim.webhook INFO Processing webhook event identity.scim.webhook ERROR Webhook event processing had errors identity.scim.webhook INFO Webhook event processed successfully identity.scim.webhook WARN Destructive webhook event missing timestamp identity.scim.webhook WARN Webhook timestamp outside freshness window identity.scim.webhook ERROR Deduplication check failed for destructive event, rejecting identity.scim.webhook WARN Deduplication check failed, proceeding for non-destructive event identity.scim.webhook INFO Duplicate webhook event, skipping identity.scim.webhook WARN Cannot deduplicate destructive event (missing event_id/resource_id), rejecting identity.scim.webhook ERROR Webhook deletion blocked: 24h cumulative threshold exceededSSH & SQL Bastion
SSH Bastion Gateway
Connection & TCP: bastion.tcp.connection DEBUG New TCP connection bastion.tcp.closed DEBUG TCP connection closed bastion.connection.global_limit WARN Global connection limit reached bastion.connection.ip_limit WARN Per-IP connection limit reached bastion.connection.rate_limited WARN Connection rate limit exceeded
Authentication: bastion.auth.success DEBUG Authentication successful bastion.auth.failure WARN Authentication failure bastion.auth.banned WARN Client banned due to auth failures bastion.auth.rate_limited WARN Auth rate limit exceeded bastion.auth.pubkey_rejected INFO Public key rejected bastion.auth.cert_accepted INFO Certificate accepted bastion.auth.cert_rejected WARN AUDIT Certificate rejected bastion.auth.cert_no_principals WARN Certificate has no principals bastion.auth.cert_invalid WARN Certificate invalid bastion.auth.password_rejected INFO Password authentication rejected bastion.auth.keyboard_interactive DEBUG/INFO Keyboard-interactive auth flow bastion.auth.cleanup INFO AUDIT Device code cleanup bastion.auth.poll INFO AUDIT Device code polling
Session Lifecycle: bastion.session.connect INFO Session connection established bastion.session.authenticated INFO Session authenticated bastion.session.created INFO AUDIT Session created bastion.session.cert_auth INFO Certificate authentication path bastion.session.disconnect INFO Session disconnected bastion.session.revoked DEBUG Session revoked bastion.session.terminated WARN Session terminated (user disabled) bastion.session.expired INFO Session expired bastion.session.access_denied WARN AUDIT Session access denied bastion.session.auth_failed ERROR/WARN Device code auth failure bastion.session.auth_cancelled INFO User closed SSH before auth bastion.session.rate_limited WARN AUDIT Session creation rate limit exceeded bastion.session.global_limit WARN AUDIT Global session limit exceeded bastion.session.user_limit WARN AUDIT Per-user session limit exceeded bastion.session.ip_limit WARN AUDIT Per-IP session limit exceeded bastion.session.create_failed ERROR Failed to create session bastion.session.panic ERROR Panic in session handler
Shell & Commands: bastion.command.execute INFO AUDIT Command executed bastion.command.access_denied WARN AUDIT Command access denied bastion.command.panic ERROR Panic recovered executing command bastion.shell.rate_limited WARN AUDIT Command rate limit exceeded bastion.shell.read_error ERROR Failed to read input
AI Features: bastion.ai.rate_limited WARN AI features rate limited bastion.ai.thinking_panic WARN Panic during AI thinking bastion.ai.session_closed WARN Session closed during AI operation
SSH Proxy: bastion.ssh INFO AUDIT SSH proxy operation (connect/disconnect) bastion.ssh.recording_failed WARN SSH session recording failed bastion.ssh.recording_close_failed WARN Failed to close SSH recording
SFTP: bastion.sftp.audit INFO AUDIT SFTP operation audit (upload/download/delete/rename) bastion.sftp.connect ERROR SFTP remote connection error bastion.sftp.connect_error WARN SFTP connection error
Port Forwarding: bastion.forward.tcpip_forward INFO TCP/IP forwarding (listen/connect) bastion.forward.direct_tcpip INFO Direct TCP/IP forward bastion.forward.forwarded_tcpip DEBUG Forwarded TCP/IP connection bastion.forward.ssrf WARN AUDIT SSRF protection blocked forward
Session Sharing: bastion.share.started INFO Session sharing started bastion.share.stopped INFO Session sharing stopped bastion.share.collab_enabled INFO Collaboration enabled bastion.share.collab_disabled INFO Collaboration disabled
Recording: bastion.recording.started INFO Session recording started bastion.recording.stopped INFO Session recording completed bastion.recording.size_limit WARN Recording size limit reached bastion.recording.compress_failed WARN Failed to compress recording
SSH CA: bastion.sshca_sign.issued INFO AUDIT SSH CA certificate issued bastion.sshca_sign.denied INFO AUDIT SSH CA signing denied bastion.sshca_sign.error ERROR SSH CA signing error bastion.sshca_setup.denied INFO AUDIT SSH CA setup access denied
SQL Bastion: bastion.sql.query DEBUG SQL query execution bastion.sql.query_completed INFO AUDIT SQL query completed bastion.sql.query_failed ERROR SQL query execution failed bastion.sql.acl_rejected WARN AUDIT SQL query ACL rejected
QR Code: bastion.qr.rate_limited WARN QR code generation rate limited
PAT & TOTP: bastion.pat.create INFO PAT created bastion.pat.revoke INFO PAT revoked bastion.totp.enroll INFO AUDIT TOTP enrolled bastion.totp.revoke INFO AUDIT TOTP revoked
Geo & Time Restrictions: bastion.restrictions.geo WARN AUDIT Geo-IP restriction blocked bastion.restrictions.time WARN AUDIT Time-based restriction blocked
Host Key Verification: bastion.hostkey INFO/WARN Host key verification (first-seen, changed, verified) bastion.hostkey.sftp INFO/WARN SFTP host key verification
Token Refresh: bastion.refresh DEBUG-ERROR Token/userinfo refresh lifecycle
Lifecycle: bastion.init INFO/ERROR Bastion initialization bastion.shutdown INFO Bastion shutdownCertificates & PKI
ACME CA Server
Init & Lifecycle: acme.init INFO ACME CA server disabled in config acme.init WARN JetStream temporarily unavailable, retrying certificate load acme.init ERROR Failed to load certificates after retries acme.init INFO ACME CA server initialized acme.init DEBUG Restored CRL number from persistent storage acme.init INFO CRL signing failed on startup, retrying acme.init ERROR AUDIT Failed to regenerate CRL on startup — revoked certificates may not be enforced acme.init INFO CRL regenerated on startup acme.init INFO Skipping CRL rebuild on startup (not leader)
Periodic CRL Health Check: acme.crl.periodic WARN AUDIT CRL expired or missing — rebuilding acme.crl.periodic INFO AUDIT Periodic CRL rebuild succeeded acme.crl.periodic ERROR AUDIT Periodic CRL rebuild failed — revoked certificates may not be enforced
Certificate Load from Persistent Storage: acme.init.load INFO Persistent storage not enabled, skipping certificate load acme.init.load ERROR Failed to load certificates from persistent storage acme.init.load DEBUG Skipping expired certificate acme.init.load WARN Failed to store certificate in memory cache acme.init.load DEBUG Loaded certificate from persistent storage acme.init.load WARN Failed to load certificate from persistent storage
Certificate Issuance: acme.certificate.issue WARN AUDIT CAA re-check failed at issuance time acme.certificate.issue WARN Failed to get CA chain acme.certificate.issue WARN Serial index replication incomplete - revocation may need retry acme.certificate.issue WARN Failed to save certificate to persistent storage acme.certificate.issue INFO AUDIT Certificate issued acme.certificate.issue WARN Failed to record certificate issuance for rate limiting
Certificate Revocation: acme.certificate.revoke WARN Failed to update revocation in persistent storage acme.certificate.revoke INFO AUDIT Certificate revoked
CAA Checking: acme.caa.check DEBUG Checking CAA records acme.caa.check WARN CAA lookup returned SERVFAIL acme.caa.check DEBUG CAA lookup returned no records acme.caa.check WARN CAA records do not authorize this CA acme.caa.check DEBUG CAA check passed acme.caa.lookup DEBUG CAA records found acme.caa.iodef DEBUG CAA iodef record found
Challenge Response: acme.challenge.respond ERROR Failed to atomically update challenge status acme.challenge.respond ERROR Failed to update authorization acme.challenge.respond INFO AUDIT Challenge response received
Challenge Validation: acme.challenge.validate WARN Async validation cancelled during initial delay acme.challenge.validate ERROR Failed to reload challenge for async validation acme.challenge.validate INFO Challenge no longer in processing state, skipping validation acme.challenge.validate ERROR Failed to reload challenge after validation acme.challenge.validate WARN Failed to record auth failure for rate limiting acme.challenge.validate ERROR Failed to store challenge after validation acme.challenge.validate DEBUG Starting challenge validation acme.challenge.validate INFO Challenge validation completed
Authorization: acme.authorization.update INFO AUDIT Authorization status updated
Deterministic DNS Token: acme.challenge.deterministic ERROR Cluster key not configured for deterministic DNS acme.challenge.deterministic DEBUG Generated deterministic token
CRL: acme.crl.get DEBUG CRL served from memory cache acme.crl.get INFO No CRL found, generating initial CRL acme.crl.get ERROR Failed to load CRL after rebuild acme.crl.get ERROR Failed to load CRL from persistent storage acme.crl.get DEBUG CRL loaded from persistent storage and cached acme.crl.rebuild INFO Rebuilding CRL acme.crl.rebuild ERROR Failed to collect revoked certificates acme.crl.rebuild ERROR Failed to request CRL signing acme.crl.rebuild ERROR Failed to sign CRL acme.crl.rebuild ERROR Unexpected response type from CA module acme.crl.rebuild ERROR CA module failed to sign CRL acme.crl.rebuild WARN Failed to persist CRL to storage acme.crl.rebuild INFO CRL rebuilt successfully acme.crl.rebuild ERROR Background CRL rebuild failed acme.crl.collect WARN Failed to collect ACME revocations, continuing with X.509 acme.crl.collect WARN Failed to collect X.509 revocations acme.crl.collect DEBUG Collected revoked certificates acme.crl.collect WARN Failed to parse certificate serial number, skipping acme.crl.collect WARN Invalid serial number (zero or negative), skipping acme.crl.collect.x509 WARN Failed to parse X.509 certificate serial number, skipping acme.crl.collect.x509 WARN Invalid X.509 serial number (zero or negative), skipping
Nonce: acme.nonce.create ERROR Failed to generate random nonce acme.nonce.create ERROR Failed to store nonce acme.nonce.create ERROR Failed to achieve nonce storage quorum acme.nonce.create DEBUG Created new nonce acme.nonce.validate ERROR Failed to get nonce from cache acme.nonce.validate ERROR Failed to wait for nonce lookup acme.nonce.validate ERROR Unexpected cache response type acme.nonce.validate WARN Nonce not found acme.nonce.validate ERROR Invalid nonce data type in cache acme.nonce.validate WARN Nonce expired acme.nonce.validate ERROR Failed to atomically consume nonce acme.nonce.validate DEBUG Nonce validated and consumed atomically
OCSP: acme.ocsp.handle WARN Invalid OCSP request acme.ocsp.handle WARN Invalid serial number in OCSP request acme.ocsp.handle DEBUG Processing OCSP request acme.ocsp.handle DEBUG OCSP response served from cache acme.ocsp.handle ERROR Failed to check certificate status acme.ocsp.handle ERROR Failed to request OCSP signing acme.ocsp.handle ERROR Failed to sign OCSP response acme.ocsp.handle ERROR Unexpected response type from CA module acme.ocsp.handle ERROR CA module failed to sign OCSP response acme.ocsp.handle INFO OCSP response generated acme.ocsp.x509 DEBUG Failed to query X.509 module acme.ocsp.flush INFO OCSP cache flushed on startup
Order: acme.order.create ERROR Failed to generate order ID acme.order.create ERROR Failed to create authorization acme.order.create ERROR Failed to store order acme.order.create ERROR Failed to achieve order storage quorum acme.order.create INFO Created new order acme.order.create WARN Failed to record order for rate limiting acme.order.finalize INFO Order finalization started acme.order.issue ERROR Failed to reload order for async certificate issuance acme.order.issue INFO Order no longer in processing state, skipping certificate issuance acme.order.issue WARN Context cancelled before certificate issuance acme.order.issue ERROR Failed to issue certificate acme.order.issue WARN Failed to record finalization failure for rate limiting acme.order.issue ERROR Failed to reload order after certificate issuance acme.order.issue ERROR Failed to update order after certificate issuance acme.order.issue INFO AUDIT Certificate issued successfully
Legacy Order Rate Limit: acme.order.ratelimit WARN Failed to check rate limit, allowing request acme.order.ratelimit WARN Rate limit optimistic lock failed after retries, allowing request acme.order.ratelimit WARN Order rate limit exceeded
Validation HTTP-01: acme.validation.http01 DEBUG Validating HTTP-01 challenge acme.validation.http01 WARN HTTP-01 validation failed: connection error acme.validation.http01 WARN HTTP-01 validation failed: wrong status code acme.validation.http01 WARN HTTP-01 validation failed: invalid key authorization format acme.validation.http01 WARN HTTP-01 validation failed: key authorization hash mismatch acme.validation.http01 INFO HTTP-01 validation successful acme.validation.http01.dns ERROR Failed to resolve hostname via DNS module acme.validation.http01.dns ERROR DNS returned no addresses acme.validation.http01.dns DEBUG Resolved hostname via DNS module acme.validation.http01.dns DEBUG Connected to validation target acme.validation.http01.dns WARN Failed to connect to IP, trying next acme.validation.http01.dns ERROR Failed to connect to any resolved IP
Validation DNS-01: acme.validation.dns01 DEBUG Validating DNS-01 challenge acme.validation.dns01 WARN DNS-01 validation failed: DNS lookup error acme.validation.dns01 ERROR DNS-01 validation failed: no expected value computed acme.validation.dns01 INFO DNS-01 validation successful acme.validation.dns01 WARN DNS-01 validation failed: no matching TXT record
Validation TLS-ALPN-01: acme.validation.tlsalpn01 DEBUG Validating TLS-ALPN-01 challenge acme.validation.tlsalpn01 WARN TLS-ALPN-01 validation failed: connection error acme.validation.tlsalpn01 WARN TLS-ALPN-01 validation failed: wrong ALPN protocol acme.validation.tlsalpn01 WARN TLS-ALPN-01 validation failed: certificate doesn't contain identifier acme.validation.tlsalpn01 WARN TLS-ALPN-01 validation failed: no acmeIdentifier extension acme.validation.tlsalpn01 WARN TLS-ALPN-01 validation failed: acmeIdentifier mismatch acme.validation.tlsalpn01 INFO TLS-ALPN-01 validation successful acme.validation.tlsalpn01.dns ERROR Failed to resolve hostname via DNS module acme.validation.tlsalpn01.dns ERROR DNS returned no addresses acme.validation.tlsalpn01.dns DEBUG Resolved hostname via DNS module acme.validation.tlsalpn01.dns DEBUG TLS connection established acme.validation.tlsalpn01.dns WARN Failed to connect to IP, trying next acme.validation.tlsalpn01.dns ERROR Failed to establish TLS connection to any resolved IP
Validation Deterministic DNS: acme.validation.deterministic DEBUG Failed to resolve domain for deterministic DNS check
Comprehensive Rate Limiting: acme.ratelimit.circuitbreaker ERROR Rate limit circuit breaker open — blocking requests acme.ratelimit.check DEBUG Rate limit checks passed acme.ratelimit.warn WARN Approaching rate limit capacity acme.ratelimit.blocked INFO AUDIT Rate limit check blocked operation acme.ratelimit.error WARN Rate limit state access error acme.ratelimit.record DEBUG Recorded certificate issuance acme.ratelimit.record WARN Recorded authorization failure acme.ratelimit.record WARN Recorded finalization failure
SPIFFE Account: spiffe.account.create WARN Unknown account key - no matching workload found spiffe.account.create WARN Client IP not allowed for workload spiffe.account.create ERROR Failed to store SPIFFE account spiffe.account.create INFO Created SPIFFE account spiffe.account.deactivate INFO SPIFFE account deactivated
SPIFFE Order: spiffe.order.create WARN Client IP not allowed for workload spiffe.order.create WARN SAN not allowed for workload spiffe.order.create ERROR Failed to generate order ID spiffe.order.create ERROR Failed to store SPIFFE order spiffe.order.create INFO Created SPIFFE order (auto-approved) spiffe.order.get WARN Client IP not allowed for workload spiffe.order.finalize DEBUG Using workload snapshot from order creation (hot-reload safe) spiffe.order.finalize WARN Workload removed from config during order lifetime spiffe.order.finalize DEBUG Using current workload config (v1 order - upgrade for hot-reload safety) spiffe.order.finalize WARN Client IP not allowed for workload spiffe.order.finalize WARN Certificate issuance queue full, waiting for slot spiffe.order.finalize ERROR Failed to revert order status after timeout spiffe.order.finalize INFO SPIFFE order finalization started spiffe.order.issue ERROR Failed to reload SPIFFE order for certificate issuance spiffe.order.issue INFO SPIFFE order no longer in processing state, skipping spiffe.order.issue ERROR Failed to issue SPIFFE certificate spiffe.order.issue ERROR Failed to reload SPIFFE order after certificate issuance spiffe.order.issue ERROR Failed to update SPIFFE order after certificate issuance spiffe.order.issue INFO AUDIT SPIFFE certificate issued successfully
SPIFFE Certificate: spiffe.certificate.issue WARN Failed to get CA chain spiffe.certificate.issue WARN Serial index replication incomplete spiffe.certificate.issue WARN Failed to save SPIFFE certificate to persistent storage spiffe.certificate.get.error WARN Account not found for certificate retrieval spiffe.certificate.get.cidr WARN Client IP not allowed for workload during certificate retrieval spiffe.certificate.revoke.cidr WARN Client IP not allowed for workload during revocation spiffe.certificate.revoke INFO AUDIT SPIFFE certificate revoked
SPIFFE Rate Limiting: spiffe.ratelimit.check WARN Failed to check rate limit, allowing request spiffe.ratelimit.blocked WARN SPIFFE rate limit exceeded spiffe.ratelimit.record WARN Failed to get rate limit state spiffe.ratelimit.record WARN Failed to store rate limit stateACME Client
Init (module startup): acmeclient.init INFO Registered ACME certificates readiness check with HexonReady acmeclient.init INFO Static TLS certificate configured - ACME client inactive acmeclient.init INFO ACME client disabled in config acmeclient.init INFO Initializing ACME client acmeclient.init ERROR Failed to initialize ACME client acmeclient.init INFO ACME client initialized successfully acmeclient.init WARN Persistent storage is memory-only (cluster_path not set). Certificates will NOT survive cluster restart. acmeclient.init WARN Failed to load issuance state, starting fresh acmeclient.init WARN Failed to load some certificates from storage acmeclient.init WARN Service certificate acquisition issue - will retry via recovery
Reset (data reset on startup): acmeclient.reset WARN ACME reset requested - deleting all ACME data (account, certificates, issuance state) acmeclient.reset ERROR Failed to reset ACME data acmeclient.reset WARN ACME data reset complete - starting fresh
Fallback (bootstrap fallback): acmeclient.fallback WARN ACME initialization failed, attempting bootstrap fallback acmeclient.fallback ERROR Failed to generate bootstrap certificate - server cannot start with TLS acmeclient.fallback WARN Using bootstrap certificate - ACME unavailable
Startup (service certificate acquisition): acmeclient.startup WARN No service hostname configured - skipping certificate check acmeclient.startup INFO Using existing valid certificate acmeclient.startup INFO No valid certificate found - attempting ACME issuance with leader detection acmeclient.startup INFO Certificate now available acmeclient.startup DEBUG No leader available, waiting... acmeclient.startup INFO Attempting ACME certificate issuance acmeclient.startup WARN Certificate issuance request failed acmeclient.startup WARN Certificate issuance wait failed acmeclient.startup INFO Certificate issued successfully during startup acmeclient.startup WARN ACME issuance timeout, falling back to bootstrap certificate acmeclient.startup WARN Using bootstrap certificate - ACME recovery will be attempted
Account (ACME account management): acmeclient.account INFO Loaded existing ACME account from persistent storage acmeclient.account INFO No existing account found, waiting before creation to prevent race acmeclient.account INFO Account was created by another node during wait, using existing acmeclient.account INFO Creating new ACME account acmeclient.account WARN Failed to save ACME account to persistent storage acmeclient.account INFO Saved new ACME account to persistent storage acmeclient.account INFO Created new ACME account successfully
Request (signed ACME requests): acmeclient.request DEBUG Retrying ACME request after transient error
Rate limit (CA rate limit handling and client-side tracking): acmeclient.ratelimit WARN Rate limited by CA, waiting before retry acmeclient.ratelimit WARN Rate limited by CA, scheduling for later retry acmeclient.ratelimit WARN Rate limited by CA without valid Retry-After, using exponential backoff acmeclient.ratelimit DEBUG Rate limit checking disabled, skipping pre-flight checks acmeclient.ratelimit INFO Starting pre-flight rate limit checks acmeclient.ratelimit INFO All rate limit checks passed acmeclient.ratelimit WARN Failed to check rate limit state acmeclient.ratelimit WARN Approaching rate limit capacity acmeclient.ratelimit WARN Rate limit check blocked operation acmeclient.ratelimit DEBUG Rate limit check passed acmeclient.ratelimit WARN Failed to record last order time acmeclient.ratelimit WARN Failed to retrieve account order state acmeclient.ratelimit ERROR Failed to store account order state acmeclient.ratelimit INFO Recorded order creation acmeclient.ratelimit WARN Failed to retrieve domain state, creating new acmeclient.ratelimit WARN IssuedAt array exceeded max entries, truncating acmeclient.ratelimit ERROR Failed to store domain issuance state acmeclient.ratelimit INFO Recorded domain certificate issuance acmeclient.ratelimit WARN Failed to retrieve exact set state, creating new acmeclient.ratelimit WARN IssuedAt array exceeded max entries, truncating acmeclient.ratelimit ERROR Failed to store exact set issuance state acmeclient.ratelimit INFO Recorded exact set certificate issuance acmeclient.ratelimit WARN Failed to retrieve domain state for auth failure recording acmeclient.ratelimit ERROR Failed to store authorization failure state acmeclient.ratelimit WARN Recorded authorization failure acmeclient.ratelimit ERROR Failed to store Retry-After state acmeclient.ratelimit WARN Stored Retry-After delay from CA
Issue (certificate issuance): acmeclient.issue WARN Certificate issuance attempted but ACME client not initialized acmeclient.issue INFO Starting certificate issuance acmeclient.issue ERROR Certificate issuance failed acmeclient.issue INFO Certificate issued successfully acmeclient.issue INFO All domains covered by static certificate, no ACME issuance needed acmeclient.issue WARN Rate limit check failed, delaying issuance acmeclient.issue WARN Certificate issuance already in progress for domain acmeclient.issue DEBUG Starting ACME certificate issuance acmeclient.issue WARN Failed to record order creation for rate limiting acmeclient.issue INFO Starting challenge listeners cluster-wide acmeclient.issue WARN Node failed to start challenge listener acmeclient.issue INFO Stopping challenge listeners cluster-wide acmeclient.issue WARN Failed to broadcast stop challenge listener acmeclient.issue WARN Failed to save certificate to persistent storage acmeclient.issue WARN Failed to install certificate locally acmeclient.issue WARN Failed to record certificate issuance for rate limiting
Challenge (HTTP-01 challenge handling): acmeclient.challenge WARN Invalid ACME token format acmeclient.challenge WARN Failed to lookup challenge token acmeclient.challenge WARN Failed to wait for challenge lookup acmeclient.challenge ERROR Unexpected response type from memorystorage acmeclient.challenge DEBUG Challenge token not found acmeclient.challenge WARN Challenge token has invalid value type acmeclient.challenge WARN Failed to write challenge response acmeclient.challenge INFO Served ACME challenge acmeclient.challenge DEBUG Challenge token stored, responding to ACME server acmeclient.challenge INFO Authorization validated acmeclient.challenge WARN Failed to record authorization failure for rate limiting
Listener (challenge listener lifecycle): acmeclient.listener DEBUG Challenge listener already running acmeclient.listener WARN Failed to resolve interface IP, falling back to 0.0.0.0 acmeclient.listener ERROR Failed to create challenge listener acmeclient.listener ERROR Failed to start challenge listener acmeclient.listener INFO Challenge listener started acmeclient.listener DEBUG Challenge listener not running, nothing to stop acmeclient.listener WARN Challenge listener shutdown error acmeclient.listener INFO Challenge listener stopped
Bootstrap (bootstrap certificate generation): acmeclient.bootstrap INFO Generated CA-signed bootstrap certificate acmeclient.bootstrap WARN CA signing failed, falling back to self-signed acmeclient.bootstrap WARN Generated temporary bootstrap certificate - ACME certificate pending
Renewal (certificate renewal): acmeclient.renewal INFO Scheduling renewal checks acmeclient.renewal ERROR Failed to schedule renewal checks acmeclient.renewal INFO Renewal check scheduler registered acmeclient.renewal INFO Running startup certificate check acmeclient.renewal ERROR Failed to trigger startup renewal check acmeclient.renewal ERROR Startup renewal check failed acmeclient.renewal INFO Startup certificate check completed acmeclient.renewal INFO Cleaned up old failure records acmeclient.renewal INFO Cleaned up stale inProgress entries acmeclient.renewal WARN Failed to fetch ARI info acmeclient.renewal INFO ARI suggests certificate renewal acmeclient.renewal DEBUG ARI window not yet open, skipping renewal acmeclient.renewal INFO Certificate needs renewal acmeclient.renewal INFO Certificate missing for domain acmeclient.renewal INFO Skipping certificate renewal - retry not allowed acmeclient.renewal INFO Renewing certificate acmeclient.renewal ERROR Failed to renew certificate acmeclient.renewal DEBUG Domain covered by static certificate, skipping acmeclient.renewal INFO ARI-guided certificate renewal completed acmeclient.renewal INFO Certificate renewed successfully
Renewals (hexdcall renewal check operation): acmeclient.renewals WARN Renewal check skipped - ACME client not initialized acmeclient.renewals INFO Starting renewal check acmeclient.renewals INFO Renewal check completed
Domains (domain collection for certificate issuance): acmeclient.domains DEBUG Added service hostname to domain list acmeclient.domains DEBUG Added additional domains from config acmeclient.domains DEBUG Added proxy mapping hosts acmeclient.domains DEBUG Added proxy landing page hostname acmeclient.domains DEBUG Added forward proxy hostname acmeclient.domains DEBUG Added connector hostname acmeclient.domains INFO Collected domains for ACME certificates acmeclient.domains INFO Domains skipped (covered by static TLS certificate) acmeclient.domains WARN No domains configured for ACME. Set service.hostname, acme_client.additional_domains, or configure proxy mappings
Load (certificate loading from storage): acmeclient.load WARN Certificate load skipped - ACME client not initialized acmeclient.load INFO Loading certificates from storage acmeclient.load INFO Loaded certificate from persistent storage
Coverage (static certificate coverage checking): acmeclient.coverage WARN Failed to read static certificate for coverage check acmeclient.coverage WARN Failed to decode static certificate PEM acmeclient.coverage WARN Failed to parse static certificate acmeclient.coverage INFO Parsed static certificate for coverage check acmeclient.coverage DEBUG Domain covered by static certificate, skipping ACME
ARI (ACME Renewal Information - RFC 8739): acmeclient.ari WARN Invalid ARI window: end not after start, using window start acmeclient.ari WARN ARI window exceeds maximum, capping duration acmeclient.ari WARN Failed to generate random offset for ARI window, using window start acmeclient.ari WARN CA suggests early renewal - check explanation URL acmeclient.ari DEBUG Using cached ARI info acmeclient.ari ERROR Failed to fetch ARI info from CA acmeclient.ari WARN Failed to cache ARI info acmeclient.ari INFO Fetched and cached ARI info from CA acmeclient.ari DEBUG No ARI info available for domain acmeclient.ari WARN Failed to retrieve ARI info for marking as replaced acmeclient.ari DEBUG No ARI info found to mark as replaced acmeclient.ari ERROR Failed to store ARI replaced state acmeclient.ari INFO Marked ARI renewal as completed
Recovery (bootstrap recovery routine): acmeclient.recovery INFO Starting ACME recovery routine acmeclient.recovery INFO Bootstrap certificate replaced - recovery complete acmeclient.recovery INFO Waiting for next recovery attempt acmeclient.recovery INFO Bootstrap certificate replaced during wait - recovery complete acmeclient.recovery WARN Initial recovery schedule exhausted - switching to normal renewal cycle acmeclient.recovery INFO Attempting ACME recovery acmeclient.recovery WARN ACME client not fully initialized - attempting reinitialization acmeclient.recovery WARN ACME reinitialization failed acmeclient.recovery WARN ACME recovery request failed acmeclient.recovery WARN ACME recovery wait failed acmeclient.recovery WARN ACME recovery got unexpected response type acmeclient.recovery WARN ACME recovery issuance failed acmeclient.recovery INFO ACME recovery successful - real certificate obtained
Watch (PersistentWatch certificate sync): acmeclient.watch WARN PersistentWatch disconnected, will retry acmeclient.watch ERROR Failed to start PersistentWatch acmeclient.watch INFO Started PersistentWatch for certificate updates acmeclient.watch INFO PersistentWatch channel closed acmeclient.watch WARN Received invalid envelope type acmeclient.watch ERROR Failed to decrypt certificate from watch event acmeclient.watch WARN Module state not ready, skipping certificate install acmeclient.watch ERROR Failed to install certificate from watch event acmeclient.watch INFO AUDIT Certificate installed via PersistentWatch acmeclient.watch INFO AUDIT Certificate removed via PersistentWatch
Status, List, Get (certificate queries): acmeclient.status DEBUG Certificate status check - ACME client not initialized acmeclient.list DEBUG Certificate list requested - ACME client not initialized acmeclient.get DEBUG Certificate requested - ACME client not initialized acmeclient.get WARN Failed to load certificate from storage acmeclient.get DEBUG Certificate retrieved
State (issuance state persistence): acmeclient.state WARN Failed to delete issuance state acmeclient.state WARN Failed to save issuance state acmeclient.state INFO Loaded issuance state from persistent storage
Cleanup (stale data removal): acmeclient.cleanup WARN Failed to delete old issuance state acmeclient.cleanup WARN Removed stale inProgress entry
Shutdown: acmeclient.shutdown WARN Shutdown timed out waiting for watch goroutine acmeclient.shutdown INFO ACME client shutdown completeAutoTLS Certificate Management
Init & Lifecycle: autotls.init ERROR AutoTLS init panic recovered: <detail> autotls.init INFO Static TLS certificate configured, AutoTLS skipping autotls.init INFO AutoTLS enabled, issuing <type> certificate autotls.init INFO Certificate signing failed on startup, retrying autotls.init ERROR AutoTLS initialization failed, will retry in renewal loop autotls.init INFO AutoTLS initialized successfully
Certificate Issuance: autotls.issue INFO Issuing deterministic certificate autotls.issue WARN Failed to store wildcard certificate, hostname cert is still active autotls.issue WARN Failed to set default certificate, hostname cert is still active autotls.issue INFO AutoTLS certificate issued
Renewal: autotls.renew INFO Manual certificate renewal requested autotls.renew WARN Hostname changed, issuing certificate for new hostname autotls.renew INFO Renewing AutoTLS certificate autotls.renew ERROR AutoTLS certificate renewal failed autotls.renew INFO AutoTLS certificate renewed successfully
Epoch Parsing: autotls.epoch WARN invalid epoch "<value>", falling back to default <default> autotls.epoch WARN ACME CA epoch is in the future, certificate cycle will be 0 until epoch is reachedCertificate Management
SetCertificate: certmanager.set ERROR Failed to parse certificate certmanager.set ERROR Certificate does not match domain certmanager.set ERROR Rejecting expired certificate certmanager.set ERROR Rejecting not-yet-valid certificate certmanager.set ERROR Failed to store certificate in memorystorage certmanager.set INFO Certificate stored successfully
SetDefaultCertificate: certmanager.setdefault ERROR Failed to parse default certificate certmanager.setdefault INFO Default certificate set successfully
DeleteCertificate: certmanager.delete INFO Certificate deleted
OnCertificateExpired: certmanager.expired ERROR Panic in OnCertificateExpired callback certmanager.expired WARN Certificate expired from cache - renewal may have failed
ClearCache: certmanager.clearcache INFO Certificate cache cleared
Shutdown: certmanager.shutdown INFO Certificate manager shutdown completeSPIFFE Workload Identity
Route registration: spiffe.routes INFO Registering SPIFFE ACME routes spiffe.routes INFO SPIFFE ACME routes registered successfully
CIDR enforcement: spiffe.cidr.validate WARN Invalid CIDR in AllowedCIDRs, skipping spiffe.cidr.blocked WARN AUDIT SPIFFE request blocked by CIDR policy
Error responses: spiffe.handler.error WARN SPIFFE ACME error responseProtection
Data Loss Prevention
Compilation: dlp.compile INFO DLP engine compiled successfully dlp.compile WARN DLP compiled with warnings (e.g. detectors without keywords) dlp.compile ERROR DLP compilation failed — config validation error
Scan — Clean: dlp.scan INFO DLP scan clean (no violations found) Fields: correlation_id, direction, policy, content_type, body_size, scan_duration_ms, method, path, remote_addr, mapping, user
Scan — Violation: dlp.violation WARN AUDIT DLP violation detected Fields: correlation_id, direction, policy, action (log/redact/block), content_type, body_size, scan_duration_ms, method, path, remote_addr, mapping, user, violations ([{"detector":"credit_card","action":"redact","count":2}]) NOTE: violations field NEVER contains matched content — only detector names and counts
Scan — Error: dlp.error WARN AUDIT DLP scan error (fail_closed blocks, fail_open passes) Fields: correlation_id, direction, policy, method, path, remote_addr, mapping, user, error
Scan — Skipped: dlp.skip DEBUG DLP scan skipped Fields: correlation_id, direction, reason, method, path, remote_addr, mapping, user Reasons: disabled_per_mapping, excluded_group, no_policyGeo/IP and ASN Access Control
Database initialization (init goroutine — bridge.Log): geoaccess.init INFO Geo access module initialized but DISABLED via config geoaccess.init WARN Geo database file not found, trying embedded database geoaccess.init WARN Failed to open geo database, trying embedded database geoaccess.init INFO Geo database loaded successfully from external file geoaccess.init ERROR Failed to load embedded geo database - DISABLING geo restrictions geoaccess.init WARN Using EMBEDDED geo database - may be outdated. Configure geo_database path for up-to-date data geoaccess.init ERROR No geo database available (external or embedded) - DISABLING geo restrictions
ASN database initialization (init goroutine — bridge.Log): geoaccess.init WARN ASN database file not found, trying embedded database geoaccess.init WARN Failed to open ASN database, trying embedded database geoaccess.init INFO ASN database loaded successfully from external file geoaccess.init WARN Failed to load embedded ASN database - ASN filtering disabled geoaccess.init WARN Using EMBEDDED ASN database - may be outdated. Configure geo_asn_database path for up-to-date data geoaccess.init INFO No ASN database available - ASN filtering disabled
Final status (init goroutine — bridge.Log): geoaccess.init INFO Geo access module initialized
Access check blocks (Check — safeLog): geoaccess.check INFO Request blocked by ASN deny list geoaccess.check INFO Request blocked - ASN not in allow list geoaccess.check INFO Request blocked by country deny list geoaccess.check INFO Request blocked - country not in allow list
None of the log entries in this module are marked as AUDIT.Init-phase entries are emitted via bridge.Log. Check-phase entries use safeLog(which calls bridge.GetClusterOp().Local) and carry a traceID for correlation.Proof-of-Work Challenge
Challenge Generation: pow.generate DEBUG Using default difficulty pow.generate ERROR Failed to generate random challenge pow.generate ERROR Failed to generate challenge ID pow.generate WARN Invalid TTL config, using default pow.generate ERROR Failed to broadcast PoW token to cluster pow.generate DEBUG PoW token stored in cluster pow.generate INFO PoW challenge issued
Challenge Creation with Anti-Automation: pow.create ERROR Failed to broadcast PoW token to cluster pow.create DEBUG PoW challenge created with anti-automation features
Validation: pow.validate ERROR Failed to query PoW token from storage pow.validate ERROR Failed to retrieve PoW token pow.validate WARN Invalid challenge ID pow.validate ERROR Invalid token type in storage pow.validate ERROR Failed to delete expired PoW token pow.validate DEBUG Challenge expired pow.validate DEBUG PoW solution failed pow.validate ERROR Failed to delete used PoW token pow.validate DEBUG PoW token deleted after successful validation pow.validate INFO Valid PoW solution
Timing Validation: pow.timing DEBUG Validating PoW timing pow.timing WARN PoW submitted too quickly (bot detection)
Honeypot Validation: pow.honeypot DEBUG Validating honeypot fields pow.honeypot WARN Decoy field filled (bot detection) pow.honeypot DEBUG Honeypot validation passed
Hash Difficulty Check: pow.hash TRACE Hash difficulty check failed at full byte pow.hash TRACE Hash difficulty check failed at partial byte pow.hash TRACE Hash difficulty check passedRate Limiting
Initialization: ratelimit.init INFO Rate limiting module initialized but DISABLED via config ratelimit.init ERROR Rate limiting module initialized with INVALID config ratelimit.init INFO AUDIT Rate limiting module initialized and ENABLED
Request Check: ratelimit.check ERROR Invalid rate limit configuration ratelimit.check WARN Request blocked - client banned ratelimit.check WARN Request blocked - rate limiter at memory capacity ratelimit.check TRACE Request allowed - new window ratelimit.check WARN Request blocked - rate limit exceeded, client banned ratelimit.check TRACE Request allowed
Manual Ban: ratelimit.ban ERROR Failed to ban client ratelimit.ban WARN Client manually banned
Manual Unban: ratelimit.unban ERROR Failed to unban client ratelimit.unban INFO Client manually unbannedRequest Size Limiting
Initialization: sizelimit.init INFO Size limiting module initialized but DISABLED via config sizelimit.init ERROR Size limiting module initialized with INVALID config sizelimit.init WARN Invalid size limit exception - SKIPPED sizelimit.init WARN Invalid regex in size limit exception - SKIPPED sizelimit.init INFO Size limiting module initialized and ENABLED sizelimit.init INFO Size limit exception loadedTime-Based Access Control
Initialization: timeaccess.init INFO Time access module initialized but DISABLED via config timeaccess.init INFO Time access module initialized and ENABLED
Access Check: timeaccess.check INFO Request blocked by time restrictionWeb Application Firewall
Initialization: waf.init INFO AUDIT WAF disabled in configuration waf.init INFO AUDIT Using self-contained blocking mode (each rule blocks immediately) waf.init INFO AUDIT Using anomaly scoring mode (blocks based on accumulated score) waf.init WARN Invalid paranoia level (< 1), clamping to 1 waf.init WARN Invalid paranoia level (> 4), clamping to 4 waf.init WARN WAF running in DETECTION ONLY mode - requests will NOT be blocked waf.init INFO WAF engine initialized successfully
Custom Rules: waf.custom_rule ERROR Rejected invalid custom WAF rule waf.custom_rule ERROR Rejected custom WAF rule with invalid directive waf.custom_rule DEBUG Loaded custom WAF rule
Request Inspection: waf.bypass INFO AUDIT WAF bypassed for route waf.client_ip WARN AUDIT Failed to extract or validate client IP address waf.uri DEBUG Processing request URI waf.args DEBUG Adding query parameters to WAF ARGS waf.phase1 DEBUG Phase 1 (request headers) complete waf.body WARN Request body exceeds maximum size limit waf.body ERROR Failed to read request body waf.body ERROR Failed to inspect request body waf.body ERROR Failed to process request body waf.pass TRACE Request passed WAF inspection
Blocking: waf.block WARN Request blocked by WAF
Metrics Recording: waf.metrics TRACE WAF inspection completeEnd-to-Origin Encryption
Channel init: e2oe.init DEBUG E2OE channel init: no valid session e2oe.init ERROR Failed to generate ECDH key pair e2oe.init ERROR ECDH key derivation failed e2oe.init WARN AUDIT E2OE rebind: decode failed — treating as no rebind e2oe.init INFO AUDIT E2OE Tier 1 rebind failed — downgrade to baseline e2oe.init INFO AUDIT E2OE channel established (dynamic — see below) e2oe.init DEBUG E2OE channel rekeyed
The "E2OE channel established" audit entry uses a dynamic message (auditMsg variable): - "E2OE Tier 1 channel rebound" — rebind proof verified, Tier 1 preserved on page reload - "E2OE Tier 1 channel established" — first Tier 1 from WebAuthn ECDH state in session - "E2OE channel established" — baseline channel (no WebAuthn state)
A separate audit entry signals that Tier 1 promotion was DECLINED for a sessionthat holds a prior WebAuthn-bound secret but provided no rebind proof: e2oe.init INFO AUDIT E2OE channel attached to session with prior Tier 1 — staying Baseline (no rebind proof)This is expected on cross-origin navigation when the user moves from the authorigin to another origin sharing the session cookie. The channel encrypts atBaseline; auth-origin channels can still rebind to Tier 1 via the existingsession secret.
PRF-wrapped per-origin Tier 1 (when enabled — see config below): - "E2OE Tier 1 channel established (PRF-wrapped relay)" cross-origin Tier 1 via wrapped material + relay - e2oe.init INFO AUDIT E2OE Tier 1 PRF-wrapped rebind failed — downgrade to baseline - e2oe.tier1_relay INFO AUDIT E2OE Tier 1 wrap-relay served - e2oe.tier1_wrap_upload INFO AUDIT E2OE Tier 1 wrap-upload accepted - e2oe.tier1_wrap_upload WARN AUDIT E2OE Tier 1 wrap-upload: credential ID mismatch — rejecting
WebSocket encryption: e2oe.websocket INFO AUDIT E2OE WebSocket encryption active e2oe.websocket WARN AUDIT E2OE WebSocket frame too short e2oe.websocket WARN AUDIT E2OE WebSocket decryption failed e2oe.websocket ERROR AUDIT E2OE WebSocket encryption failed
HTTP middleware: e2oe.middleware DEBUG request encrypted e2oe.decrypt INFO AUDIT E2OE decryption failed e2oe.middleware WARN AUDIT E2OE buffer overflow — response served unencrypted e2oe.middleware WARN AUDIT E2OE passthrough — response advertises streaming Content-Type but request did not; stream served unencrypted e2oe.middleware WARN AUDIT E2OE passthrough — backend body failed decompression; serving unencrypted
HTML shell: e2oe.shell WARN AUDIT E2OE shell buffer overflow — HTML served unencrypted e2oe.shell WARN AUDIT E2OE shell passthrough — response advertises streaming Content-Type; stream served unencrypted e2oe.shell DEBUG HTML wrapped in E2OE shell
WebSocket strict-monotonic gate: e2oe.websocket WARN AUDIT E2OE WebSocket non-monotonic seq — rejecting (replay or reorder)
PRF-wrapped Tier 1 (when e2oe_tier1_pre_provision is on): e2oe.tier1_relay INFO AUDIT E2OE Tier 1 wrap-relay served e2oe.tier1_wrap_upload INFO AUDIT E2OE Tier 1 wrap-upload accepted e2oe.tier1_wrap_upload WARN AUDIT E2OE Tier 1 wrap-upload: missing credential ID — rejecting e2oe.tier1_wrap_upload WARN AUDIT E2OE Tier 1 wrap-upload: credential ID mismatch — rejecting e2oe.tier1_wrap_relay WARN AUDIT E2OE Tier 1 endpoint rate-limited (layer=session|ip) e2oe.tier1_wrap_upload WARN AUDIT E2OE Tier 1 endpoint rate-limited (layer=session|ip) e2oe.tier1_wrap_state WARN AUDIT E2OE Tier 1 endpoint rate-limited (layer=session|ip)
Auth-time provisioning: signin.tier1.provision INFO AUDIT Tier 1 pre-provisioning issued signin.tier1.provision ERROR CSPRNG failure deriving Tier 1 origin secret signin.tier1.provision ERROR AUDIT Tier 1 pre-provisioning: failed to persist origin secrets — falling back to legacy Baseline
E2OE HTTP middleware is applied globally on path-based service routes(signin, console, OIDC IdP, SCIM) and by the proxy for each proxiedhostname.Connectivity
DNS Resolution
Init & Lifecycle: dns.init INFO DNS module initialized dns.health INFO DNS resolvers not configured, using cluster resolvers for health checking dns.health WARN Failed to initialize resolver health manager dns.health INFO Resolver health manager started dns.health INFO Health checking enabled but no resolvers configured dns.health INFO Resolver health checking disabled dns.adaptive INFO Adaptive resolver selector initialized dns.adaptive INFO Adaptive selector enabled but no resolvers configured dns.adaptive INFO Adaptive resolver selector disabled
Resolution: dns.resolve DEBUG DNS resolution request dns.resolve DEBUG DNS cache hit dns.resolve DEBUG Waiting for concurrent DNS lookup to complete dns.resolve ERROR DNS lookup panicked dns.resolve ERROR DNS resolution failed dns.resolve INFO DNS resolution succeeded - no records found dns.resolve INFO DNS resolution succeeded
Hostname Validation: dns.validate WARN Hostname validation failed
Health Status: dns.gethealth DEBUG DNS health status requested
Cache Operations: dns.cache WARN Invalid cache entry type dns.cache WARN Failed to broadcast DNS cache update dns.cache DEBUG DNS result cached
DNSSEC Core: dns.dnssec DEBUG Using DNS-over-TLS dns.dnssec WARN DNS query failed dns.dnssec DEBUG DNS query returned error dns.dnssec.full DEBUG RRSIG present but AD bit not set - performing full validation dns.dnssec.full ERROR Full DNSSEC validation failed dns.dnssec.full INFO Full DNSSEC validation succeeded dns.dnssec ERROR DNSSEC validation failed: RRSIG present but AD bit not set dns.dnssec ERROR DNSSEC strict mode: zone not signed dns.dnssec WARN DNSSEC validation skipped: zone not signed dns.dnssec DEBUG DNSSEC validation succeeded (resolver-trust mode)
DNSSEC Validation: dns.dnssec.validate WARN RRSIG signature verification failed dns.dnssec.validate WARN RRSIG signature expired or not yet valid dns.dnssec.validate DEBUG RRSIG signature validated successfully dns.dnssec.dnskey WARN Failed to query DNSKEY dns.dnssec.dnskey WARN DNSKEY query returned error dns.dnssec.dnskey WARN No DNSKEY records found in zone dns.dnssec.dnskey DEBUG DNSKEY records fetched successfully dns.dnssec.validate ERROR DNSSEC strict mode: RRset not signed dns.dnssec.validate DEBUG RRset has no RRSIG (zone not signed) dns.dnssec.validate ERROR No matching DNSKEY found for RRSIG dns.dnssec.validate INFO DNSSEC validation completed
DNSSEC Cache: dns.dnssec.cache DEBUG DNSKEY cache hit dns.dnssec.cache DEBUG DNSKEY cache expired dns.dnssec.cache DEBUG DNSKEY cached dns.dnssec.cache DEBUG DS cache hit dns.dnssec.cache DEBUG DS cache expired dns.dnssec.cache DEBUG DS cached dns.dnssec.cache INFO DNSSEC cache cleared
DNSSEC Chain of Trust: dns.dnssec WARN DEPRECATED: SHA-1 used in DNSSEC validation dns.dnssec.ds WARN Failed to query DS dns.dnssec.ds WARN DS query returned error dns.dnssec.ds DEBUG No DS records found (zone may be unsigned or at root) dns.dnssec.ds DEBUG DS records fetched successfully dns.dnssec.chain WARN Failed to compute DS digest dns.dnssec.chain DEBUG DNSKEY validated successfully using DS dns.dnssec.chain ERROR DNSKEY validation failed: no matching DS found dns.dnssec.chain DEBUG Validating chain of trust dns.dnssec.chain INFO Root DNSKEY validated against trust anchor dns.dnssec.chain ERROR Root DNSKEY validation failed: no matching trust anchor
DNSSEC NSEC/NSEC3: dns.dnssec.nsec DEBUG No NSEC records found in response dns.dnssec.nsec DEBUG Found NSEC records for validation dns.dnssec.nsec INFO NSEC authenticated denial validated dns.dnssec.nsec WARN NSEC validation failed: name not in range dns.dnssec.nsec3 DEBUG No NSEC3 records found in response dns.dnssec.nsec3 DEBUG Found NSEC3 records for validation dns.dnssec.nsec3 WARN Unsupported NSEC3 hash algorithm dns.dnssec.nsec3 ERROR Failed to compute NSEC3 hash dns.dnssec.nsec3 INFO NSEC3 authenticated denial validated dns.dnssec.nsec3 WARN NSEC3 validation failed: hash not in range
Resolver: dns.resolve WARN Hostname validation failed dns.ttl DEBUG Cache override enabled, using configured TTL dns.ttl DEBUG Using DNS server TTL dns.ttl DEBUG DNS server TTL not available, using fallback dns.health DEBUG Filtered unhealthy resolvers dns.resolve DEBUG DNS resolution succeeded dns.resolve DEBUG DNS resolution failed, trying next resolver dns.resolve ERROR All DNS resolvers failed dns.resolve DEBUG Using system DNS resolver dns.resolve DEBUG Using configured DNS cache TTL for system resolver dns.resolve DEBUG DNS resolution succeeded dns.dnssec DEBUG DNSSEC resolution succeeded dns.dnssec WARN DNSSEC lookup failed, trying next resolver dns.cname DEBUG Resolving CNAME target dns.cname DEBUG CNAME record found dns.cname WARN Failed to resolve CNAME target dns.cname DEBUG CNAME chain returned (flatten disabled) dns.query DEBUG Using DNS-over-TLS dns.query WARN DNS query failed dns.query DEBUG DNS query returned error dns.query DEBUG DNS query completed dns.query WARN DNS query returned SERVFAIL
Adaptive Resolver: dns.adaptive ERROR Failed to create adaptive selector dns.adaptive INFO Cleaned up performance data for removed resolvers dns.adaptive INFO Adaptive resolver selector initialized dns.adaptive TRACE Resolver performance updated dns.adaptive INFO Adaptive selector learning phase completed, switching to intelligent selection dns.adaptive DEBUG Adaptive DNS resolution succeeded dns.adaptive DEBUG Adaptive DNS resolution failed, selecting another resolver dns.adaptive ERROR All adaptive DNS resolution attempts failed
Health Manager: dns.health INFO Initializing resolver health checks dns.health ERROR Invalid resolver address format dns.health WARN Initial health check failed dns.health INFO Initial health check passed dns.health ERROR No healthy DNS resolvers available dns.health INFO Resolver health initialization complete dns.health DEBUG Starting health check dns.health WARN Health check query failed dns.health WARN Health check returned nil response dns.health DEBUG Health check returned error response dns.health DEBUG Health check successful dns.health DEBUG GetHealthyResolvers called dns.fallback WARN All custom DNS resolvers unhealthy, falling back to system DNS dns.fallback INFO Custom DNS resolver recovered, switching back from system DNS dns.health WARN RecordSuccess called for unknown resolver dns.health INFO Resolver recovered dns.fallback INFO Custom DNS resolver recovered, switching back from system DNS dns.health WARN RecordFailure called for unknown resolver dns.health WARN Resolver marked unhealthy dns.health INFO Starting resolver health checker dns.health INFO Stopping resolver health checker dns.health DEBUG Performing health checks dns.health DEBUG Health check still failing dns.health INFO Resolver recovered via health check dns.health INFO Removed resolvers no longer in configurationForward Proxy Engine
Initialize: forwardproxy.init INFO Forward proxy disabled in config forwardproxy.init ERROR Failed to initialize forward proxy forwardproxy.init INFO Initializing forward proxy module
Access Control: forwardproxy.checkaccess ERROR Failed to resolve user groups forwardproxy.checkaccess ERROR Failed to call firewall.CheckProxyAccess forwardproxy.checkaccess ERROR Invalid response type from firewall
Allowed Targets: forwardproxy.getallowedtargets ERROR Failed to resolve user groups forwardproxy.getallowedtargets ERROR Failed to call firewall.GetAllowedTargets forwardproxy.getallowedtargets ERROR Invalid response type from firewall
PAC Generation: forwardproxy.generatepac WARN PAC requested without authentication forwardproxy.generatepac DEBUG Generated PAC file
Authentication: forwardproxy.auth WARN Token validation failed forwardproxy.auth WARN User account is disabled forwardproxy.auth INFO AUDIT Token authentication successful forwardproxy.auth DEBUG Invalidated fingerprint binding
Token Generation: forwardproxy.token ERROR Failed to generate token forwardproxy.token DEBUG Generated proxy token
Fingerprint Binding: forwardproxy.bind WARN Failed to broadcast fingerprint binding forwardproxy.bind WARN Failed to achieve quorum for fingerprint binding forwardproxy.bind INFO Fingerprint bound to session
Rate Limiting: forwardproxy.ratelimit WARN Rate limit check called without UserID forwardproxy.ratelimit WARN User rate limit exceeded forwardproxy.ratelimit WARN Destination rate limit exceeded forwardproxy.ratelimit WARN User bandwidth limit exceeded
Rate Limit Cleanup: forwardproxy.cleanup DEBUG Cleaned up stale rate limit entries
Geo Restrictions: forwardproxy.restrictions.geo ERROR Geo check failed - denying access (fail-closed) forwardproxy.restrictions.geo ERROR Geo check wait failed - denying access (fail-closed) forwardproxy.restrictions.geo ERROR Invalid geo check response type - denying access (fail-closed) forwardproxy.restrictions.geo INFO Access blocked by geo restriction
Time Restrictions: forwardproxy.restrictions.time ERROR Time check failed - denying access (fail-closed) forwardproxy.restrictions.time ERROR Time check wait failed - denying access (fail-closed) forwardproxy.restrictions.time ERROR Invalid time check response type - denying access (fail-closed) forwardproxy.restrictions.time INFO Access blocked by time restrictionClient Access (HexonClient)
Lifecycle: clientaccess INFO initializing client access subsystem clientaccess ERROR failed to create IP pool clientaccess ERROR TLS config not available, client access listener disabled clientaccess ERROR failed to create client access listener clientaccess ERROR failed to start client access listener clientaccess INFO client access listener started
Connection: clientaccess INFO AUDIT client connected (VIP, routes, hostname) clientaccess INFO AUDIT client disconnected (duration, traffic stats) clientaccess WARN client rejected: max clients reached clientaccess WARN unexpected first message type
Registration: clientaccess INFO client registered (session, VIP, hostname) clientaccess INFO client unregistered (session, duration, traffic counters)
Authentication — JWT: clientaccess INFO/WARN client auth failed (INFO for PAT rejection, WARN otherwise) clientaccess WARN channel binding failed
Authentication — Device Code: clientaccess WARN device code auth rejected: concurrency limit reached clientaccess WARN device code authorization request failed clientaccess INFO device code challenge sent, waiting for authorization clientaccess INFO client disconnected during device code auth clientaccess INFO device code authorized clientaccess INFO device code denied by user clientaccess INFO device code expired
Authorization: clientaccess WARN group access denied
Token Refresh: clientaccess WARN token refresh failed: invalid token clientaccess WARN token refresh failed: channel binding clientaccess WARN group access revoked on refresh clientaccess INFO token refreshed with group change clientaccess DEBUG token refreshed
PAT Revocation: clientaccess INFO disconnected clients after PAT revocation
Dial: clientaccess WARN dial denied by ACL clientaccess DEBUG dial failed clientaccess DEBUG udp dial failed clientaccess DEBUG dial accept stream error
Traffic: clientaccess DEBUG client traffic
Hexdcall Module: clientaccess.list_sessions WARN Registry not initialized clientaccess.list_sessions DEBUG Listed client access sessions clientaccess.disconnect_session WARN Username missing in disconnect request clientaccess.disconnect_session WARN Registry not initialized clientaccess.disconnect_session INFO Session not found on this node clientaccess.disconnect_session INFO Disconnected client access session clientaccess.disconnect_session INFO Disconnected all client access sessions for userQUIC Connector
Initialization: connectors INFO initializing connector subsystem connectors ERROR TLS config not available, connector listener disabled connectors ERROR failed to create connector listener connectors ERROR failed to start connector listener connectors INFO connector listener started
Authentication: connectors.handler WARN AUDIT connector auth failed: invalid proof connectors.handler WARN AUDIT connector auth failed: unknown site connectors.handler WARN AUDIT connector auth failed: source IP not allowed
Connection lifecycle: connectors.handler INFO AUDIT connector connected connectors.handler INFO AUDIT connector disconnected
Registry: connectors.registry INFO AUDIT Connector instance registered connectors.registry INFO AUDIT Connector instance unregistered
Session management: connectors WARN failed to create session connectors WARN session create wait failed connectors WARN unexpected session create response type connectors DEBUG failed to extend session connectors DEBUG session extend wait failed connectors WARN failed to revoke session connectors WARN session revoke wait failed
Config reload: connectors.reload INFO disconnected instances for removed site
Relay: connectors.relay WARN AUDIT relay rejected: source IP not a cluster peer connectors.relay DEBUG relay connection accepted connectors.relay WARN relay fallback also failed after local exhaustionNetwork Listener
HTTP Errors: listener.http.error DEBUG/WARN HTTP server errors (DEBUG for client TLS/connection failures, WARN otherwise)
Proxy Mode: listener.proxy_validation WARN Rejected connection not from trusted proxy listener.proxy_validation ERROR Client IP header missing in proxy mode listener.proxy_cert WARN Oversized cert header (DoS) / parse failed listener.proxy_cert DEBUG/INFO Client cert injected / invalid PEM block
CORS: listener.cors WARN AUDIT CORS origin rejected
Sessions: listener.session DEBUG Session created / validated / expired listener.session ERROR/WARN Session creation/validation failures
Proof-of-Work: listener.pow INFO PoW challenge passed / application session valid / body restored listener.pow WARN Body too large / session validation failures / invalid body format listener.pow ERROR PoW handler not registered / body encryption failures listener.pow DEBUG Session checks, challenge served, body stored
Rate Limiting: listener.ratelimit WARN AUDIT Request blocked by rate limit listener.ratelimit WARN Config fallback (invalid rate_limit_type) listener.ratelimit ERROR Ratelimit module call/response failures / no fingerprint listener.ratelimit DEBUG Fingerprint fallback to IP listener.ratelimit TRACE Per-entity rate limiting applied listener.ratelimit.status DEBUG Rate limit check passed listener.ratelimit.circuitbreaker ERROR Circuit breaker open — blocking request
Size Limiting: listener.sizelimit WARN AUDIT Request blocked — size limit exceeded listener.sizelimit ERROR Sizelimit module call/response failures listener.sizelimit TRACE Size limit applied / exception / within limit
Compression: listener.compression DEBUG Response compressed
Geo Restrictions: listener.geo INFO AUDIT Request blocked by geo restriction listener.geo ERROR Geo check failed (allowing request)
Time Restrictions: listener.time INFO AUDIT Request blocked by time restriction listener.time ERROR Time check failed (allowing request)
ECH (Encrypted Client Hello): ech.generate INFO ECH key pair derived from cluster key
PoW Body Preservation: pow.body DEBUG POST body stored / retrieved / deleted / restored pow.body WARN Body not found (expired) / cleanup failures pow.body ERROR Storage / retrieval / decryption failuresForward Proxy
Lifecycle & Middleware: forwardproxy.service.init INFO Forward proxy service disabled in config forwardproxy.service.init INFO Forward proxy service initialized forwardproxy.middleware INFO Forward proxy disabled, passing CONNECT to next handler forwardproxy.middleware WARN CONNECT request rejected on main service port
PAC & Config Endpoints: forwardproxy.pac DEBUG Generating PAC file for authenticated user forwardproxy.pac ERROR Failed to generate PAC forwardproxy.config DEBUG Generating proxy config for extension forwardproxy.config WARN Access blocked by restriction forwardproxy.config ERROR Failed to generate PAC forwardproxy.config ERROR Failed to generate proxy token forwardproxy.config INFO Proxy config generated successfully forwardproxy.setup INFO Proxy setup authorized
Restrictions: forwardproxy.restrictions ERROR Failed to call restrictions check
SSRF Protection: forwardproxy.ssrf WARN AUDIT blocked non-routable IP from DNS resolution forwardproxy.ssrf WARN AUDIT all resolved IPs are non-routable — request blocked
DNS & Connectivity: forwardproxy.dns DEBUG Resolving hostname via DNS module forwardproxy.dns DEBUG DNS resolution successful forwardproxy.dns DEBUG Using system DNS resolver forwardproxy.dns DEBUG Successfully connected to backend forwardproxy.dns WARN DNS module failure - falling back to system DNS forwardproxy.dns WARN DNS resolution timeout - falling back to system DNS forwardproxy.dns WARN DNS module returned error - falling back to system DNS forwardproxy.dns WARN Failed to connect to IP, trying next forwardproxy.connector DEBUG Dialing via connector site forwardproxy.connector DEBUG Connected via connector site
TCP CONNECT Authentication: forwardproxy.tcp.auth INFO AUDIT Missing or invalid Proxy-Authorization header forwardproxy.tcp.auth INFO AUDIT Token too long forwardproxy.tcp.auth INFO AUDIT Authentication failed
TCP CONNECT ACL & Rate Limiting: forwardproxy.tcp.acl WARN AUDIT ACL denied forwardproxy.tcp.ratelimit ERROR Rate limit service unavailable forwardproxy.tcp.ratelimit ERROR Rate limit check failed forwardproxy.tcp.ratelimit WARN AUDIT Rate limit exceeded
TCP CONNECT Connection: forwardproxy.tcp.connect INFO Proxy connection established forwardproxy.tcp.dial ERROR Failed to connect to backend forwardproxy.tcp.http2 DEBUG Using HTTP/2+ full duplex CONNECT stream forwardproxy.tcp.http2 ERROR Failed to enable full duplex mode forwardproxy.tcp.http2 ERROR Failed to flush response forwardproxy.tcp.hijack ERROR ResponseWriter does not support hijacking forwardproxy.tcp.hijack ERROR Failed to hijack connection forwardproxy.tcp.error ERROR Request validation or service errors (dynamic message)
HTTP Proxy Authentication: forwardproxy.http.auth INFO AUDIT Missing or invalid Proxy-Authorization header forwardproxy.http.auth INFO AUDIT Token too long forwardproxy.http.auth INFO AUDIT Authentication failed
HTTP Proxy ACL & Rate Limiting: forwardproxy.http.acl WARN AUDIT ACL denied forwardproxy.http.ratelimit ERROR Rate limit service unavailable forwardproxy.http.ratelimit ERROR Rate limit check failed forwardproxy.http.ratelimit WARN AUDIT Rate limit exceeded
HTTP Proxy Forwarding: forwardproxy.http.forward INFO HTTP proxy request forwarded forwardproxy.http.forward ERROR Failed to forward request forwardproxy.http.copy DEBUG Response body copy error forwardproxy.http.error ERROR Request validation or service errors (dynamic message)
UDP/MASQUE Authentication: forwardproxy.udp.auth INFO AUDIT Missing or invalid Proxy-Authorization header forwardproxy.udp.auth INFO AUDIT Token too long forwardproxy.udp.auth INFO AUDIT Authentication failed
UDP/MASQUE ACL & Rate Limiting: forwardproxy.udp.acl WARN AUDIT ACL denied forwardproxy.udp.ratelimit ERROR Rate limit service unavailable forwardproxy.udp.ratelimit ERROR Rate limit check failed forwardproxy.udp.ratelimit WARN Rate limit exceeded
UDP/MASQUE Connection & Session: forwardproxy.udp.parse WARN Failed to parse CONNECT-UDP request forwardproxy.udp.parse WARN Invalid CONNECT-UDP request forwardproxy.udp.parse WARN Invalid target hostname forwardproxy.udp.connect INFO UDP proxy session authorized forwardproxy.udp.ssrf WARN AUDIT SSRF blocked: UDP target resolves to non-routable IP forwardproxy.udp.dial WARN Failed to dial UDP IP, trying next forwardproxy.udp.dial ERROR All UDP dial attempts failed forwardproxy.udp.proxy ERROR UDP proxy error forwardproxy.udp.complete INFO UDP proxy session completed forwardproxy.udp.error ERROR Request validation or service errors (dynamic message)
Shared (TCP, HTTP, UDP): forwardproxy.ratelimit.status DEBUG Rate limit check passedCluster & Operations
Git Configuration Management
Related logs from other modules: - config: logs git fetch, hard reset, and reload results - cluster: logs broadcast delivery to member nodesHot Reload
Related logs from other modules: - config: logs file watcher start/stop, hash comparison, reload success/failure - cluster: logs broadcast delivery to member nodesModule Data Storage
Initialization: moduledata.init WARN module_data_storage=ldap is deprecated and no longer supported; using hexon KV backend. Migrate existing module data to hexon KV before upgrading. moduledata.init WARN cluster_path not set - module data may be lost on restart moduledata.init WARN Persistent storage not enabled - module data will NOT survive restarts moduledata.init INFO Module data storage initialized (hexon KV)
Get Operation: moduledata.get DEBUG Getting module data moduledata.get ERROR Backend.Get failed
Set Operation: moduledata.set INFO Setting module data moduledata.set ERROR Backend.Set failed moduledata.set.preferences WARN Failed to store language preference
Delete Operation: moduledata.delete INFO Deleting module data moduledata.delete ERROR Backend.Delete failed
GetAllForUser Operation: moduledata.getallforuser DEBUG Getting all module data for user moduledata.getallforuser ERROR Backend.GetAllForUser failed
LoadAll Operation: moduledata.loadall INFO Loading all module data moduledata.loadall ERROR Backend.LoadAll failed
Exists Operation: moduledata.exists ERROR Backend.Exists failed
Hexon KV Backend — Get: moduledata.hexon.get ERROR PersistentGet failed moduledata.hexon.get WARN Unexpected value type in KV
Hexon KV Backend — Set: moduledata.hexon.set ERROR PersistentSet failed moduledata.hexon.set DEBUG Module data stored in Hexon KV
Hexon KV Backend — Delete: moduledata.hexon.delete DEBUG Key not found in Hexon KV (nothing to delete) moduledata.hexon.delete ERROR PersistentDelete failed moduledata.hexon.delete DEBUG Module data deleted from Hexon KV
Hexon KV Backend — GetAllForUser: moduledata.hexon.getallforuser DEBUG Retrieved all module data for user
Hexon KV Backend — LoadAll: moduledata.hexon.loadall INFO Loaded all module data from Hexon KVNotification Service
Send — single event delivery: notify.send.email_failed WARN Email notification failed notify.send.webhook_failed WARN Webhook notification failed notify.send.webhook_ok DEBUG Webhook notification sent notify.send.render_fallback WARN Email template rendering failed, using plain text fallback
Digest — batched digest delivery: notify.digest.email_failed WARN Digest email failed notify.digest.webhook_failed WARN Digest webhook failed notify.digest.render_fallback WARN Digest template rendering failed, using plain text fallback
Health check: notify.healthcheck DEBUG Health check completedDistributed Sessions
Session Create: sessions.create INFO Session created (type, module_key, TTL) sessions.create WARN TTL capped to certificate validity / DurableKV not available sessions.create ERROR Failed to generate ID / store session / update index
Session Validate: sessions.validate DEBUG Session validated (type, module_key) sessions.validate ERROR Invalid session type in storage
Session Extend: sessions.extend DEBUG Session TTL extended sessions.extend WARN Extension rejected by validator / cert expired / TTL capped
Session Revoke: sessions.revoke INFO Session revoked sessions.revoke WARN Failed to broadcast deletion sessions.revoke_all INFO All sessions revoked for module_key
Session Regenerate: sessions.regenerate INFO Session ID regenerated successfully sessions.regenerate WARN Session not found for regeneration sessions.regenerate ERROR Fetch/generate/store/index/delete failures
Activity Tracking: sessions.persist_activity ERROR Panic recovered persisting LastActivity
Callbacks & Validators: sessions.validator INFO Session extend validator registered sessions.callback INFO Session create/delete/delete_v2 callback registered sessions.callback ERROR Callback panicked (create/delete/delete_v2)
Index: sessions.index DEBUG Index cleanup / session removed / index deletedSMTP Email Delivery
PAT expiry callback (init): smtp.pat_expiry INFO AUDIT Personal Access Token expired
TLS certificate warnings (sendViaSSL / sendViaSTARTTLS): smtp.send WARN TLS certificate verification failed, retrying with skip_tls=true — not recommended for production, configure a valid certificate smtp.send WARN STARTTLS certificate verification failed, retrying with skip_tls=true — not recommended for production, configure a valid certificate
Magic link validation (SendMagicLinkEmail): smtp.magiclink WARN AUDIT Magic link email blocked — invalid sealed return URL
Skip notifications (SendPasskeyExpirationEmail / SendVPNPSKExpirationEmail): smtp.send DEBUG Skipping email for expired passkey smtp.send DEBUG Skipping email for expired PSK
Generic email (SendEmail): smtp.send ERROR SMTP send failed smtp.send INFO Email sent successfully
OTP email (SendOTPEmail): smtp.send ERROR SMTP send failed smtp.send INFO Email sent successfully
Certificate renewal email (SendCertRenewalEmail): smtp.send ERROR SMTP cert renewal send failed smtp.send INFO Certificate renewal email sent
Passkey expiration email (SendPasskeyExpirationEmail): smtp.send ERROR SMTP passkey expiration send failed smtp.send INFO Passkey expiration email sent
Magic link email (SendMagicLinkEmail): smtp.send ERROR SMTP send failed smtp.send INFO Magic link email sent
Test email (SendTestEmail): smtp.test ERROR SMTP test email failed smtp.test INFO SMTP test email sent
PAT created email (SendPATCreatedEmail): smtp.pat_created ERROR PAT creation notification email failed smtp.pat_created INFO PAT creation notification email sent
PAT revoked email (SendPATRevokedEmail): smtp.pat_revoked ERROR PAT revocation notification email failed smtp.pat_revoked INFO PAT revocation notification email sent
PAT expired email (SendPATExpiredEmail): smtp.pat_expired ERROR PAT expiration notification email failed smtp.pat_expired INFO PAT expiration notification email sent
Passkey created email (SendPasskeyCreatedEmail): smtp.passkey_created ERROR Passkey creation notification email failed smtp.passkey_created INFO Passkey creation notification email sent
Passkey revoked email (SendPasskeyRevokedEmail): smtp.passkey_revoked ERROR Passkey revocation notification email failed smtp.passkey_revoked INFO Passkey revocation notification email sent
TOTP created email (SendTOTPCreatedEmail): smtp.totp_created ERROR TOTP creation notification email failed smtp.totp_created INFO TOTP creation notification email sent
TOTP revoked email (SendTOTPRevokedEmail): smtp.totp_revoked ERROR TOTP revocation notification email failed smtp.totp_revoked INFO TOTP revocation notification email sent
Certificate created email (SendCertCreatedEmail): smtp.cert_created ERROR Certificate creation notification email failed smtp.cert_created INFO Certificate creation notification email sent
Certificate revoked email (SendCertRevokedEmail): smtp.cert_revoked ERROR Certificate revocation notification email failed smtp.cert_revoked INFO Certificate revocation notification email sentPersistent File Storage
Save Operation: storage.filesystem WARN Path traversal attempt blocked storage.filesystem ERROR Failed to create directory storage.filesystem ERROR Failed to marshal JSON storage.filesystem ERROR Failed to save file storage.filesystem DEBUG File saved
Load Operation: storage.filesystem WARN Path traversal attempt blocked storage.filesystem DEBUG File not found storage.filesystem ERROR Failed to read file storage.filesystem ERROR Failed to unmarshal JSON storage.filesystem DEBUG File loaded
Delete Operation: storage.filesystem WARN Path traversal attempt blocked storage.filesystem DEBUG File not found for deletion storage.filesystem ERROR Failed to delete file storage.filesystem DEBUG File deleted
Move Operation: storage.filesystem WARN Path traversal attempt blocked (source) storage.filesystem WARN Path traversal attempt blocked (target) storage.filesystem ERROR Failed to create target directory storage.filesystem ERROR Failed to move file storage.filesystem DEBUG File moved
List Operation: storage.filesystem WARN Path traversal attempt blocked storage.filesystem DEBUG Directory not found storage.filesystem ERROR Failed to read directory storage.filesystem DEBUG Directory listed
Exists Operation: storage.filesystem WARN Path traversal attempt blocked storage.filesystem DEBUG File existence checkedDistributed Memory Storage
Bootstrap — KV: memory.bootstrap.start INFO Starting JetStream KV bootstrap memory.kv.init DEBUG Requesting JetStream KV bucket memory.kv.retry DEBUG JetStream not ready, retrying in {duration} (attempt N/M) memory.bootstrap.kv_unavailable INFO JetStream KV unavailable after retries, falling back to peer broadcast memory.kv.ready DEBUG JetStream KV bucket ready memory.bootstrap.cold INFO Cold mode enabled — skipping bootstrap warmup, cache will populate on demand memory.bootstrap.read_keys DEBUG Reading keys from JetStream KV memory.bootstrap.empty INFO JetStream KV bucket is empty, nothing to restore memory.bootstrap.failed ERROR Failed to read KV keys memory.bootstrap.keys_found DEBUG Found N keys in JetStream KV memory.bootstrap.process_key DEBUG Processing KV key memory.bootstrap.retry_transient INFO Retrying N keys after transient NATS errors (JetStream leader stabilizing) memory.bootstrap.complete INFO Bootstrap complete (loaded, skipped, errors, duration)
Bootstrap — Key Processing: memory.bootstrap.get_tombstone DEBUG KV key listed but not found (tombstone) memory.bootstrap.get_transient DEBUG Transient NATS error, will retry memory.bootstrap.get_error WARN Failed to get KV entry memory.bootstrap.decode_error WARN Failed to decode KV entry, deleting corrupted key memory.bootstrap.decode_error_cleanup WARN Failed to delete corrupted KV entry memory.bootstrap.parse_error WARN Failed to parse KV key format memory.bootstrap.skip_expired DEBUG Skipping expired entry memory.bootstrap.skip_exists DEBUG Skipping key (already in memory from broadcast) memory.bootstrap.skip_deleted DEBUG Skipping key (deleted during bootstrap) memory.bootstrap.loaded DEBUG Loaded entry from KV memory.bootstrap.tracking_stopped DEBUG Stopped tracking deletes, bootstrap complete / peer bootstrap complete memory.bootstrap.track_delete DEBUG Tracking delete during bootstrap
Bootstrap — Peer Fallback: memory.bootstrap.peers_encryption_timeout WARN Encryption not ready after timeout, proceeding with bootstrap anyway memory.bootstrap.peers_wait_encryption DEBUG Waiting for encryption to be ready (X3DH/shared key sync) memory.bootstrap.peers_start INFO Starting peer-to-peer bootstrap via Broadcast memory.bootstrap.peers_failed ERROR Failed to broadcast BootstrapGetAll memory.bootstrap.peers_responses INFO Collected responses from N peers memory.bootstrap.peers_timeout WARN Failed to collect all peer responses memory.bootstrap.peers_operation_error WARN Operation error from node memory.bootstrap.peers_invalid_response WARN Invalid response type from node memory.bootstrap.peers_merge DEBUG Merging snapshot from node memory.bootstrap.peers_complete INFO Peer bootstrap complete (loaded, skipped, duration)
KV Persistence: memory.kv.encode_error WARN Failed to encode entry for KV memory.kv.put_error WARN Failed to write to KV memory.kv.persist_success DEBUG Entry persisted to KV memory.kv.delete_error WARN Failed to delete from KV memory.kv.delete_success DEBUG Entry deleted from KV
CRUD Operations: memory DEBUG Memory storage Set memory DEBUG Triggering OnSet callback memory WARN OnSet callback failed memory DEBUG Memory storage Delete memory DEBUG Triggering OnDelete callback memory WARN OnDelete callback failed memory DEBUG Memory storage All memory DEBUG Memory storage Touch memory DEBUG Memory storage SetNX memory DEBUG Memory storage SyncSet memory DEBUG Memory storage SyncGet (lazy-loaded from KV)
Bootstrap Snapshot: memory.bootstrap DEBUG BootstrapGetAll returning snapshot
Cold Cache: memory.cold WARN Corrupted KV entry, deleting memory.cold DEBUG Cold eviction sweep
Eviction: memory.eviction INFO Eviction loop shutting down gracefullyTelemetry & Logging
Stderr diagnostics (not structured LogEntry calls): [TELEMETRY] Failed to initialize OTLP exporter: <err> (falling back to stdout) — Startup: OTLP gRPC connection failed, output mode reverts to stdout Failed to marshal log entry: <err> — Runtime: JSON encoding of a log entry failed (entry is dropped) [TELEMETRY] OTLP provider shutdown error: <err> — Shutdown: OTLP provider flush/close returned an error [TELEMETRY] Shutdown complete: N logs processed, N logs dropped due to overflow — Shutdown: final stats when logs were dropped (includes audit count if any)
These messages appear only in stderr, never in the structured log stream orring buffer. They indicate infrastructure-level issues with the telemetrypipeline itself.AI Assistant
Query lifecycle: llm.query.start INFO Starting LLM query llm.query.complete INFO LLM query completed llm.query.api_error ERROR LLM API call failed llm.query.max_rounds WARN LLM query exceeded maximum tool rounds
Tool execution: llm.tool.execute INFO Executing tool via hexdcall llm.tool.approved INFO AUDIT Write operation approved by operator llm.tool.denied INFO AUDIT Write operation denied by operatorAdmin Unix Socket
No structured log entries. A single console message is emitted on startup.Command execution logging is handled by the admin CLI module.Threshold Signing & Cluster Cryptography
Threshold State Changes: threshold INFO AUDIT Threshold signing ready threshold WARN AUDIT Threshold signing unavailable threshold WARN AUDIT Threshold signing degraded threshold INFO AUDIT DKG initiated threshold INFO AUDIT DKG complete threshold ERROR AUDIT DKG failed threshold ERROR AUDIT DKG timed out threshold ERROR AUDIT CRITICAL: DKG failed after max retries threshold INFO AUDIT Threshold share persisted to KV threshold WARN AUDIT Corrupt threshold share deleted threshold ERROR AUDIT Threshold signing failed threshold ERROR AUDIT Threshold signing timed out threshold INFO AUDIT Threshold CA birth complete threshold INFO AUDIT CA resharing initiated threshold INFO AUDIT CA resharing complete threshold ERROR AUDIT CA resharing failed threshold ERROR AUDIT CA resharing timed out threshold ERROR AUDIT CRITICAL: CA public key changed during resharing threshold INFO AUDIT Threshold share migration pending threshold ERROR AUDIT TSS replay attack detected threshold ERROR AUDIT TSS envelope signature verification failed threshold ERROR AUDIT TSS mandatory signature missing
Key Rotation Events: keyrotation ERROR AUDIT Key rotation aborted keyrotation ERROR AUDIT Key rotation spk_failed keyrotation WARN AUDIT Key rotation retry keyrotation WARN AUDIT Key rotation commit_quorum keyrotation WARN AUDIT Key rotation abort_received keyrotation INFO AUDIT Key rotation <event> (initiated, deferred, commit_all, completed, activated, spk_completed)
Hexon Readiness: hexdcall INFO AUDIT HexonReady: All subsystems operational - Hexon is ready to serve traffic
CA Module — GetCABundle: ca.getcabundle ERROR Failed to get ACME CA bundle ca.getcabundle DEBUG ACME CA bundle retrieved successfully
CA Module — SignCertificate: ca.signcertificate WARN Certificate template is required ca.signcertificate WARN Public key DER is required ca.signcertificate WARN Failed to parse public key DER ca.signcertificate ERROR Failed to sign certificate with ACME CA ca.signcertificate INFO AUDIT Certificate signed successfully with ACME CA
CA Module — SignCRL: ca.signcrl WARN CRL number is required ca.signcrl WARN CRL number must be positive ca.signcrl WARN NextUpdate must be after ThisUpdate ca.signcrl ERROR Failed to sign CRL with ACME CA ca.signcrl INFO AUDIT CRL signed successfully with ACME CA
CA Module — SignOCSPResponse: ca.signocspresponse WARN Serial number is required ca.signocspresponse WARN Serial number must be positive ca.signocspresponse WARN Invalid OCSP status ca.signocspresponse WARN NextUpdate must be after ThisUpdate ca.signocspresponse ERROR Failed to sign OCSP response with ACME CA ca.signocspresponse INFO AUDIT OCSP response signed successfully with ACME CAConfiguration System
Console output categories:
startup and reload: fmt.Printf "[CONFIG] Warning: Failed to start hot-reload system: %v" fmt.Printf "[CONFIG] Loading configuration from directory: %s" fmt.Printf "[%s] %s" (license periodic check callback) fmt.Fprintf "[CONFIG] DEPRECATED: %s is deprecated — %s" fmt.Fprintf "[CONFIG] Warning: %s: expected %s, got %s — %s" (type mismatch auto-correction)
cross-module validation: fmt.Fprintf "[CONFIG] WARNING: signin.magiclink.enabled=true but SMTP is not configured — magic link disabled" fmt.Fprintf "[CONFIG] INFO: auto-enabling authentication.devicecode (required by signin.magiclink)"
git clone and metadata: fmt.Printf "[CONFIG] Git TLS config: ..." fmt.Printf "[CONFIG] Loading configuration from git repository: %s (branch: %s)" fmt.Printf "[CONFIG] Git configuration loaded successfully: ..." fmt.Printf "[CONFIG] Warning: Failed to extract git metadata: %v" fmt.Printf "[CONFIG] Using HTTP basic authentication" fmt.Printf "[CONFIG] Using SSH authentication"
file watching and reload (via logHotReloadEvent helper): fmt.Printf "[CONFIG-HOTRELOAD] Hot reload system started" fmt.Printf "[CONFIG-HOTRELOAD] Hot reload system stopped" fmt.Printf "[CONFIG-HOTRELOAD] Config file changed, triggering reload" fmt.Printf "[CONFIG-HOTRELOAD] Config reload successful" fmt.Printf "[CONFIG-HOTRELOAD] Config reload failed - keeping previous config" fmt.Printf "[CONFIG-HOTRELOAD] ALERT: Config file deleted - running with last valid config" fmt.Printf "[CONFIG-HOTRELOAD] Config file restored" fmt.Printf "[CONFIG-HOTRELOAD] Config still invalid - not retrying same broken config" fmt.Printf "[CONFIG-HOTRELOAD] ALERT: Config parse failure" fmt.Printf "[CONFIG-HOTRELOAD] ALERT: Config validation failure" fmt.Printf "[CONFIG-HOTRELOAD] ALERT: Config file missing" fmt.Printf "[CONFIG-HOTRELOAD] Config reload triggered by cluster broadcast" fmt.Printf "[CONFIG-HOTRELOAD] Config reload from cluster successful" fmt.Printf "[CONFIG-HOTRELOAD] Config reload from cluster failed" fmt.Printf "[CONFIG-HOTRELOAD] Cluster notified of config reload" fmt.Printf "[CONFIG-HOTRELOAD] Config changes detected" fmt.Printf "[CONFIG-HOTRELOAD] Config reloaded with no detected changes" fmt.Printf "[CONFIG-HOTRELOAD] Config callback panicked" fmt.Printf "[CONFIG-HOTRELOAD] WARN: Legacy config callback timed out (goroutine leaked)" fmt.Printf "[CONFIG-HOTRELOAD] WARN: Config callback timed out (context cancelled)" fmt.Printf "[CONFIG-HOTRELOAD] WARN: Context-aware callback not respecting cancellation" fmt.Printf "[CONFIG-HOTRELOAD] Config cache invalidated" fmt.Printf "[CONFIG-HOTRELOAD] Hot reload configuration optimized"
None of these are queryable via 'logs search'. They appear only in process stdout/stderr.The infrastructure/hotreload module wraps some of this via hexdcall manager logger (slog).Kubernetes CRD Configuration
CRD Definition Management: CRD definition ensure failed WARN Schema ensure error for a single CRD kind CRD definition created INFO New CRD definition created in cluster CRD definition updated INFO Existing CRD definition updated with new schema CRD definitions ensured INFO Summary: created/updated/unchanged counts for all CRDs
Manager Lifecycle: CRD auto-apply failed, using existing definitions WARN CRD ensure failed (RBAC or network); continues with existing starting K8s CRD informers INFO Informer startup with namespace and CRD count K8s API watch interrupted, will retry WARN Transient network error on watch stream (auto-retries) K8s API watch failed ERROR Non-network watch error (permissions, API server issue) failed to set watch error handler WARN Could not install custom watch error handler informer cache sync failed WARN Individual informer cache did not sync K8s informers synced INFO All informer caches synced, ready to process events K8s manager stopped INFO Manager shutdown complete K8s manager restarting after CRD definitions applied INFO Manager restart after CRD sync timeout recovery
Config Apply: failed to convert CRD to config ERROR UnstructuredToConfig failed for a CRD change skipping CRD change with unresolved sensitive fields DEBUG SecretKeyRef not yet populated, skip to avoid empty overwrite failed to apply singleton change ERROR Config mutation failed for singleton CRD failed to apply array change ERROR Config mutation failed for array/map CRD item failed to apply delete ERROR Config deletion failed for array/map item CRD config validation failed, reload skipped ERROR Config.Validate() failed after applying CRD changes applied CRD config changes INFO Config updated from CRD changes with apply/skip/error counts all CRD changes matched current config, reload skipped DEBUG All CRD changes identical to running config
Bootstrap Reconciliation: bootstrap singleton failed ERROR Failed to reconcile a singleton CRD from config bootstrap array failed ERROR Failed to reconcile an array CRD type from config bootstrap reconciliation complete INFO Summary: created/updated/skipped/pruned counts bootstrap array item failed ERROR Failed to create/update a single array item CRD bootstrap map item failed ERROR Failed to create/update a single map-keyed CRD failed to prune bootstrap CRD ERROR Could not delete orphaned bootstrap-owned CRD pruned bootstrap CRD removed from config INFO Deleted bootstrap CRD no longer in TOML config failed to delete companion Secret during prune WARN Companion Secret cleanup failed during CRD prune failed to create companion Secret ERROR Could not create K8s Secret for sensitive fields
Secrets: created companion Secret for CRD INFO New K8s Secret created for sensitive fields updated companion Secret for CRD DEBUG Existing K8s Secret updated with new sensitive data failed to resolve Secret for sensitive field WARN Could not read SecretKeyRef value from K8s Secret
Status: status update: failed to write status WARN Could not write status condition to CRD instance
Health Sync: health status synced INFO Health status written to CRD resources (with update count) cluster status sync: failed to get resource WARN Could not read cluster CRD for status update cluster status sync: failed to write status WARN Could not write leader/nodes/health to cluster CRD connector status sync: failed to get resource WARN Could not read connector site CRD for status update connector status sync: failed to write status WARN Could not write rich status to connector site CRD health sync: failed to get resource WARN Could not read CRD resource for health update health sync: failed to write status WARN Could not write health field to CRD resource
Resource Apply: CRD resource created INFO CRD instance created via CLI apply CRD resource updated INFO CRD instance updated via CLI apply (may include ownership transfer)
Watcher: unexpected object type in informer event WARN Informer delivered non-Unstructured object