amo-server/docs/docker-deployment.md

Docker Deployment Guide

This guide covers deploying the AMO CRM service with FastStream workers using Docker and Docker Compose.

Architecture

The Docker deployment consists of the following services:

• app: FastAPI application server
• faststream-worker: FastStream message processing workers
• faststream-scheduler: Scheduled task processor
• redis: Redis message broker and cache
• nginx (production): Reverse proxy and load balancer

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+
• At least 2GB RAM available
• AMO CRM access token

Quick Start

1. Setup

# Clone the repository
git clone <repository-url>
cd amo-server

# Run setup script
python scripts/docker_setup.py setup

2. Configuration

Edit the .env file with your settings:

# AMO CRM Configuration
AMO_CRM_DOMAIN=your-domain.amocrm.ru
AMO_CRM_ACCESS_TOKEN=your-access-token

# Google Sheets (optional)
GOOGLE_SERVICE_ACCOUNT_FILE=/app/credentials/google.json
GOOGLE_SCOPES=https://www.googleapis.com/auth/spreadsheets

# Database
DATABASE_URL=sqlite:///./data/amo_data.db

# Redis
REDIS_URL=redis://redis:6379/0

# Logging
LOG_LEVEL=INFO

3. Start Services

# Development mode (with hot reload)
python scripts/docker_setup.py start

# Production mode
python scripts/docker_setup.py start --mode prod

4. Verify Deployment

# Check service status
python scripts/docker_setup.py status

# View logs
python scripts/docker_setup.py logs

# Follow logs for specific service
python scripts/docker_setup.py logs --service faststream-worker --follow

Service Details

FastAPI Application (app)

Development:

• Hot reload enabled
• Debug logging
• Source code mounted as volume

Production:

• 4 worker processes
• Resource limits: 512MB RAM, 0.5 CPU
• Health checks enabled

FastStream Worker (faststream-worker)

Development:

• Hot reload enabled
• Single worker process

Production:

• 4 worker processes per container
• 2 container replicas (8 total workers)
• Resource limits: 256MB RAM, 0.5 CPU
• Automatic restart on failure

FastStream Scheduler (faststream-scheduler)

Handles periodic tasks:

• Deals refresh: Every 6 hours
• Contacts refresh: Every 4 hours
• Companies refresh: Every 8 hours
• Users refresh: Every 12 hours
• Pipelines refresh: Daily
• Events refresh: Every 2 hours

Redis (redis)

Development:

• 512MB memory limit
• Port 6379 exposed for debugging

Production:

• 1GB memory limit
• Persistence enabled with AOF
• Not exposed externally

Nginx (nginx) - Production Only

• Rate limiting (10 req/s general, 1 req/s exports)
• Gzip compression
• Security headers
• SSL termination support
• Health check bypass

Commands

Docker Setup Script

# Setup environment
python scripts/docker_setup.py setup

# Build images
python scripts/docker_setup.py build

# Start services
python scripts/docker_setup.py start [--mode dev|prod]

# Stop services
python scripts/docker_setup.py stop

# Restart services
python scripts/docker_setup.py restart [--mode dev|prod]

# View logs
python scripts/docker_setup.py logs [--service SERVICE] [--follow]

# Check status
python scripts/docker_setup.py status

Manual Docker Compose

# Development
docker-compose up -d
docker-compose logs -f

# Production
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Stop
docker-compose down

# Rebuild
docker-compose build --no-cache

Monitoring

Health Checks

All services include health checks:

# Check all services
docker-compose ps

# Check specific service health
docker-compose exec app curl http://localhost:8000/health

Logs

# All services
docker-compose logs

# Specific service
docker-compose logs faststream-worker

# Follow logs
docker-compose logs -f app

# Last 100 lines
docker-compose logs --tail=100 faststream-scheduler

Resource Usage

# Container stats
docker stats

# Service resource usage
docker-compose exec app ps aux
docker-compose exec faststream-worker free -h

Scaling

Horizontal Scaling

Scale FastStream workers:

# Scale to 4 worker containers
docker-compose up -d --scale faststream-worker=4

# Production scaling (in docker-compose.prod.yml)
# Edit replicas value for faststream-worker service

Vertical Scaling

Edit resource limits in docker-compose.prod.yml:

services:
  faststream-worker:
    deploy:
      resources:
        limits:
          memory: 512M  # Increase from 256M
          cpus: '1.0'   # Increase from 0.5

Troubleshooting

Common Issues

1. Redis Connection Failed

   # Check Redis health
   docker-compose exec redis redis-cli ping
   
   # Check Redis logs
   docker-compose logs redis
   

2. Worker Not Processing Jobs

   # Check worker logs
   docker-compose logs faststream-worker
   
   # Restart workers
   docker-compose restart faststream-worker
   

3. High Memory Usage

   # Check memory usage
   docker stats
   
   # Reduce worker processes or add memory limits
   

4. Permission Issues

   # Fix data directory permissions
   sudo chown -R 1000:1000 ./data
   sudo chown -R 1000:1000 ./credentials
   

Debug Mode

Enable debug logging:

# Set in .env file
LOG_LEVEL=DEBUG

# Restart services
docker-compose restart

Database Issues

# Access SQLite database
docker-compose exec app sqlite3 /app/data/amo_data.db

# Run migrations
docker-compose exec app alembic upgrade head

Backup and Recovery

Database Backup

# Backup SQLite database
docker-compose exec app cp /app/data/amo_data.db /app/data/amo_data.db.backup

# Copy to host
docker cp $(docker-compose ps -q app):/app/data/amo_data.db ./backup/

Redis Backup

# Redis automatically saves to /data/dump.rdb
# Volume is mounted to redis_data

# Manual backup
docker-compose exec redis redis-cli BGSAVE

Configuration Backup

# Backup configuration files
tar -czf backup/config-$(date +%Y%m%d).tar.gz .env credentials/

Security

Production Security

1. Environment Variables: Never commit .env files
2. Credentials: Store in secure volume, not in image
3. Network: Use internal networks, don't expose Redis
4. SSL: Configure SSL certificates for HTTPS
5. Rate Limiting: Nginx provides rate limiting
6. Updates: Regularly update base images

SSL Configuration

1. Place certificates in ./ssl/ directory
2. Update nginx.conf for HTTPS
3. Restart nginx service

Performance Tuning

Redis Optimization

# In docker-compose.prod.yml
command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru --tcp-backlog 511

Worker Optimization

• Adjust worker count based on CPU cores
• Monitor memory usage and adjust limits
• Use connection pooling for database

Database Optimization

• Regular VACUUM for SQLite
• Consider PostgreSQL for high load
• Index optimization

Migration from Celery

If migrating from a Celery-based deployment:

1. Stop Celery workers: docker-compose stop celery-worker celery-beat
2. Update code to use FastStream
3. Start FastStream services: docker-compose up -d faststream-worker faststream-scheduler
4. Remove Celery services from docker-compose.yml

The FastStream workers will process the same job queues through Redis.