Alertmanager Configuration for VaultDb Injector

This configuration defines a set of alerts for monitoring the VaultDb Injector within a Kubernetes environment. Each alert is designed to notify the team of potential issues that could impact the availability, security, or functionality of the services relying on Vault for secret management.

1. Alerts Configuration

1.1. Service Account Denied

- alert: VaultDbInjectorServiceAccountDenied
  annotations:
    description: "Service Account (SA) `{{ $labels.service_account_name }}` in namespace `{{ $labels.exported_namespace }}` was denied access to db_role `{{ $labels.db_role }}` due to `{{ $labels.cause }}` on cluster `{{ $labels.k8s_cluster }}`. Immediate investigation is recommended to ensure proper access controls and service configurations."
    summary: "Service Account `{{ $labels.service_account_name }}` in namespace `{{ $labels.exported_namespace }}` was denied by the injector."
  expr: increase(vault_injector_service_account_denied_count{}[2m]) > 0
  for: 1m
  labels:
    severity: critical

Response Actions: - Verify the service account permissions and roles. - Check the db_role configurations to ensure they are correctly set up. - Investigate the cause for denial to prevent future occurrences.

1.2. Token Renewal Failure

- alert: VaultDbInjectorFailToRenewToken
  annotations:
    description: "VaultDbInjector encountered an error while attempting to renew a token. This might affect the continuous operation of dependent services. Check for errors and ensure the token renewal process is configured correctly."
    summary: "VaultDbInjector token renewal failure for namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`."
  expr: increase(vault_injector_renew_token_count_error{}[2m]) > 0
  for: 1m
  labels:
    severity: warning

Response Actions: - Review the injector logs for errors related to token renewal. - Ensure the Vault policies allow for token renewal by the injector. - Check for network issues that might prevent the injector from communicating with Vault.

1.3. Lease Renewal Failure

- alert: VaultDbInjectorFailToRenewLease
  annotations:
    description: "VaultDbInjector encountered an error while attempting to renew a lease. Similar to token renewal failures, this can disrupt service operations if not addressed."
    summary: "VaultDbInjector lease renewal failure for namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`."
  expr: increase(vault_injector_renew_lease_count_error{}[2m]) > 0
  for: 1m
  labels:
    severity: warning

Response Actions: - Inspect the injector logs for specific errors related to lease renewal. - Confirm that the Vault configuration allows the injector to renew leases. - Investigate any network or configuration issues that might affect communication with Vault.

1.4. Token Expiration Warnings

- alert: VaultDbInjectorTokenExpirationLessThan14Days
  annotations:
    description: "A token is nearing expiration (less than 2 weeks). Renewing or rotating the token promptly ensures continuous service operation without interruption."
    summary: "Token nearing expiration in namespace `{{ $labels.exported_namespace }}` on cluster `{{ $labels.k8s_cluster }}`."
  expr: vault_injector_token_expiration - time() < 1209600
  for: 90m
  labels:
    severity: warning

- alert: VaultDbInjectorTokenExpirationLessThan7Days
  annotations:
    description: "A token will expire in less than 7 days. Immediate action is required to renew or rotate the token to avoid service disruption."
    summary: "Urgent: Token expiration warning for namespace `{{ $labels.exported_namespace }}`."
  expr: vault_injector_token_expiration - time() < 604800
  for: 5m
  labels:
    severity: critical

Response Actions: - For both alerts, identify the service or application using the token. - Initiate the token renewal or rotation process. - Review token policies to ensure they're aligned with security and operational requirements.

1.5. Lease Expiration Warnings

- alert: VaultDbInjectorLeaseExpirationLessThan4Days
  annotations:
    description: "A lease is nearing expiration (less than 4 days). Addressing this promptly can prevent potential access issues for services relying on leased credentials or secrets."
    summary: "Lease nearing expiration for namespace `{{ $labels.namespace }}` on cluster `{{ $labels.k8s_cluster }}`."
  expr: vault_injector_lease_expiration - time() < 345600
  for: 3m
  labels:
    severity: warning

- alert: VaultDbInjectorLeaseExpirationLessThan1Day
  annotations:
    description: "A lease will expire in less than 1 day. Immediate renewal is critical to maintaining access for the dependent services."
    summary: "Critical: Lease expiration imminent for namespace `{{ $labels.namespace }}`."
  expr: vault_injector_lease_expiration - time() < 86400
  for: 3m
  labels:
    severity: critical

Response Actions: - Quickly identify and renew the leases for the affected services or credentials. - Review the lease durations and renewal policies to prevent future alerts.

2. Conclusion

Monitoring VaultDb Injector with these alerts helps ensure the reliability and security of services depending on Vault for secret management and access control. Each alert is designed to provide actionable insights for maintaining operational efficiency and security compliance. Responding promptly to these alerts will mitigate potential risks and disruptions to your services.