Patterns
Observability
Cloud Infrastructure Google Cloud Platform DevOps & SRE Architecture

Observability

Comprehensive guide to Cloud Monitoring, Logging, and Trace for application observability on Google Cloud Platform

Observability

Google Cloud Platform provides comprehensive observability services to monitor, debug, and optimize your applications. This section covers Cloud Monitoring, Cloud Logging, and Cloud Trace for end-to-end application visibility. This guide provides everything from basic monitoring setup to advanced observability patterns.

Prerequisites

Before working with observability on GCP, ensure you have:

  • A GCP project with appropriate permissions
  • Understanding of monitoring and logging concepts
  • Knowledge of application performance monitoring
  • Familiarity with alerting and incident response
  • Understanding of distributed tracing concepts

Cloud Observability

Comprehensive monitoring, logging, and tracing for applications. Cloud Observability combines Cloud Monitoring, Cloud Logging, and Cloud Trace for comprehensive application visibility. It provides metrics collection, log analysis, distributed tracing, alerting, and dashboards.

Core Components

Cloud Monitoring

  • Metrics collection and visualization
  • Custom metrics and dashboards
  • Alerting and uptime checks
  • Integration with 150+ GCP services

Cloud Logging

  • Log ingestion and storage
  • Log analysis and query
  • Log-based metrics
  • Export and routing capabilities

Cloud Trace

  • Distributed tracing
  • Latency analysis
  • Performance profiling
  • End-to-end request tracking

Cloud Monitoring

Metrics Types

  • Standard Metrics: Built-in metrics from GCP services
  • Custom Metrics: Application-specific metrics
  • Log-based Metrics: Derived from log entries
  • Workload Metrics: Kubernetes and GKE specific

Dashboard Features

  • Pre-built dashboards for GCP services
  • Custom dashboard creation
  • Real-time data visualization
  • Shareable dashboard links

Alerting

  • Alert policies based on conditions
  • Multiple notification channels
  • Alert grouping and escalation
  • Incident management integration

Cloud Logging

Log Types

  • System Logs: GCP service-generated logs
  • Audit Logs: Administrative activity logs
  • Platform Logs: Compute and application logs
  • Custom Logs: Application-generated logs

Log Analysis

  • Advanced query language
  • Regular expression support
  • Log aggregation and filtering
  • Real-time log streaming

Log Management

  • Log retention policies
  • Log routing and export
  • Log sinks for external destinations
  • IAM-based access control

Cloud Trace

Tracing Concepts

  • Spans: Individual operations within a trace
  • Traces: Collection of spans for a request
  • Trace ID: Unique identifier for a trace
  • Span ID: Unique identifier for a span

Performance Analysis

  • Latency waterfall views
  • Hotspot identification
  • Bottleneck detection
  • Performance baseline comparison

Integration

  • Automatic instrumentation for GCP services
  • OpenTelemetry support
  • Custom instrumentation libraries
  • Service map visualization

Use Cases

  • Monitoring application health
  • Debugging distributed systems
  • Implementing observability best practices
  • Performance optimization
  • Capacity planning

Pros

  • Unified monitoring platform
  • Integrated across GCP services
  • Rich visualization options
  • Automated alerting capabilities

Cons

  • Cost can grow with data volume
  • Complex query language for advanced use
  • Requires instrumentation setup
  • Learning curve for effective use

Monitoring Setup

# Create custom metric descriptor
gcloud monitoring metric-descriptors create \
  custom.googleapis.com/my-metric \
  --metric-kind=GAUGE \
  --value-type=DOUBLE \
  --description="My custom metric"

# Create alert policy
gcloud alpha monitoring policies create \
  --policy-from-file=alert-policy.yaml

# Create dashboard
gcloud monitoring dashboards create \
  --config-from-file=dashboard.json

Logging Setup

# Create log sink
gcloud logging sinks create my-sink \
  bigquery.googleapis.com/projects/my-project/datasets/my_dataset \
  --log-filter='resource.type="gce_instance"'

# Create log-based metric
gcloud logging metrics create my-metric \
  --description="My log-based metric" \
  --log-filter='resource.type="gce_instance" AND severity>=ERROR'

# Export logs
gcloud logging logs list
gcloud logging read 'resource.type="gce_instance"' \
  --limit=10 \
  --format=json

Trace Setup

# trace_example.py
from opentelemetry import trace
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Setup tracing
trace.set_tracer_provider(TracerProvider())
cloud_trace_exporter = CloudTraceSpanExporter()
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(cloud_trace_exporter)
)

tracer = trace.get_tracer(__name__)

def my_function():
    with tracer.start_as_current_span("my-span"):
        # Your code here
        pass

Best Practices

Monitoring Strategy

Golden Signals

  • Latency: Request processing time
  • Traffic: Request rate and volume
  • Errors: Error rate and types
  • Saturation: Resource utilization
  • Availability: Service uptime

SLO and SLI

  • Define Service Level Objectives
  • Measure Service Level Indicators
  • Implement error budgets
  • Create SLO-based alerting

Logging Strategy

Log Levels

  • DEBUG: Detailed diagnostic information
  • INFO: General informational messages
  • WARNING: Warning conditions
  • ERROR: Error events
  • CRITICAL: Critical conditions

Structured Logging

  • Use JSON format for logs
  • Include contextual information
  • Use consistent field names
  • Enable log correlation

Tracing Strategy

Span Design

  • Create meaningful span names
  • Add relevant attributes
  • Include events and annotations
  • Set appropriate span kinds

Sampling

  • Implement appropriate sampling rates
  • Balance cost vs visibility
  • Use head-based sampling
  • Consider tail-based sampling

Advanced Features

Monitoring Filters

# metric-filter.yaml
displayName: "High CPU Usage"
conditions:
  - displayName: "CPU > 80%"
    conditionThreshold:
      filter: 'resource.type="gce_instance" AND metric.type="compute.googleapis.com/instance/cpu/utilization"'
      comparison: COMPARISON_GT
      thresholdValue: 0.8
      duration: 300s
aggregation:
  alignmentPeriod: 300s
  perSeriesAligner: ALIGN_MEAN

Log Queries

-- Advanced log query
resource.type="gce_instance"
AND severity>=WARNING
AND jsonPayload.service="my-service"
AND timestamp >= "2024-01-01T00:00:00Z"
AND timestamp <= "2024-01-02T00:00:00Z"

Trace Analysis

# Trace analysis example
from google.cloud import trace_v2

client = trace_v2.TraceServiceClient()

def analyze_traces(project_id):
    project_name = f"projects/{project_id}"
    
    for trace in client.list_traces(project_name):
        span_count = len(trace.spans)
        total_latency = sum(span.end_time.seconds - span.start_time.seconds 
                          for span in trace.spans)
        print(f"Trace {trace.trace_id}: {span_count} spans, {total_latency}s total")

Integration Examples

Cloud Run Integration

# service.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-service
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/target: "100"
    spec:
      containers:
      - image: gcr.io/my-project/my-app
        env:
        - name: ENABLE_TRACING
          value: "true"

GKE Integration

# deployment-with-metrics.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: my-app
        image: gcr.io/my-project/my-app
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/key.json
        volumeMounts:
        - name: google-cloud-key
          mountPath: /var/secrets/google
      volumes:
      - name: google-cloud-key
        secret:
          secretName: gcp-key

Cost Optimization

Monitoring Costs

  • Sample high-volume metrics
  • Use appropriate retention periods
  • Optimize custom metric cardinality
  • Review alert policy efficiency

Logging Costs

  • Implement log filtering
  • Use appropriate retention
  • Exclude debug logs in production
  • Aggregate similar log entries

Tracing Costs

  • Implement trace sampling
  • Limit span size and duration
  • Exclude non-critical spans
  • Regular span cleanup

Common Issues and Troubleshooting

Monitoring Data Not Appearing

  • Verify metric descriptor configuration
  • Check service account permissions
  • Review metric ingestion endpoints
  • Monitor metric collection latency

Log Query Failures

  • Validate log query syntax
  • Check log filter expressions
  • Review log retention policies
  • Ensure proper log indexing

Trace Data Gaps

  • Verify tracing agent configuration
  • Check sampling rate settings
  • Review trace export configuration
  • Monitor trace ingestion latency

Alert Policy Issues

  • Validate alert condition logic
  • Check notification channel configuration
  • Review alert evaluation windows
  • Monitor alert policy execution

Cleanup Commands

# Delete custom metric
gcloud monitoring metric-descriptors delete custom.googleapis.com/my-metric

# Delete alert policy
gcloud alpha monitoring policies delete my-alert-policy

# Delete log sink
gcloud logging sinks delete my-sink

# Delete log-based metric
gcloud logging metrics delete my-metric

# Clean up dashboards
gcloud monitoring dashboards delete my-dashboard

# Remove trace configurations
# Tracing configuration is managed through application code

Jump to other sections

Additional Resources