Maxim Snesarev
33d6bb7ebd
- Updated database configuration to switch from SQLite to PostgreSQL, including changes to alembic.ini, Docker Compose, and environment settings. - Refactored application code to utilize PostgreSQL database adapters, ensuring compatibility with the new database structure. - Enhanced API routes and data handling to support the new database, including adjustments in data models and query logic. - Introduced new job processing mechanisms for full synchronization of AMO CRM entities, leveraging FastStream for background tasks. - Improved logging and error handling across the application to facilitate better monitoring and debugging. - Removed obsolete SQLite adapter files and migrations, streamlining the project structure for PostgreSQL integration.
386 lines
7.3 KiB
Markdown
386 lines
7.3 KiB
Markdown
# Docker Deployment Guide
|
|
|
|
This guide covers deploying the AMO CRM service with FastStream workers using Docker and Docker Compose.
|
|
|
|
## Architecture
|
|
|
|
The Docker deployment consists of the following services:
|
|
|
|
- **app**: FastAPI application server
|
|
- **faststream-worker**: FastStream message processing workers
|
|
- **faststream-scheduler**: Scheduled task processor
|
|
- **redis**: Redis message broker and cache
|
|
- **nginx** (production): Reverse proxy and load balancer
|
|
|
|
## Prerequisites
|
|
|
|
- Docker 20.10+
|
|
- Docker Compose 2.0+
|
|
- At least 2GB RAM available
|
|
- AMO CRM access token
|
|
|
|
## Quick Start
|
|
|
|
### 1. Setup
|
|
|
|
```bash
|
|
# Clone the repository
|
|
git clone <repository-url>
|
|
cd amo-server
|
|
|
|
# Run setup script
|
|
python scripts/docker_setup.py setup
|
|
```
|
|
|
|
### 2. Configuration
|
|
|
|
Edit the `.env` file with your settings:
|
|
|
|
```env
|
|
# AMO CRM Configuration
|
|
AMO_CRM_DOMAIN=your-domain.amocrm.ru
|
|
AMO_CRM_ACCESS_TOKEN=your-access-token
|
|
|
|
# Google Sheets (optional)
|
|
GOOGLE_SERVICE_ACCOUNT_FILE=/app/credentials/google.json
|
|
GOOGLE_SCOPES=https://www.googleapis.com/auth/spreadsheets
|
|
|
|
# Database
|
|
DATABASE_URL=sqlite:///./data/amo_data.db
|
|
|
|
# Redis
|
|
REDIS_URL=redis://redis:6379/0
|
|
|
|
# Logging
|
|
LOG_LEVEL=INFO
|
|
```
|
|
|
|
### 3. Start Services
|
|
|
|
```bash
|
|
# Development mode (with hot reload)
|
|
python scripts/docker_setup.py start
|
|
|
|
# Production mode
|
|
python scripts/docker_setup.py start --mode prod
|
|
```
|
|
|
|
### 4. Verify Deployment
|
|
|
|
```bash
|
|
# Check service status
|
|
python scripts/docker_setup.py status
|
|
|
|
# View logs
|
|
python scripts/docker_setup.py logs
|
|
|
|
# Follow logs for specific service
|
|
python scripts/docker_setup.py logs --service faststream-worker --follow
|
|
```
|
|
|
|
## Service Details
|
|
|
|
### FastAPI Application (`app`)
|
|
|
|
**Development:**
|
|
- Hot reload enabled
|
|
- Debug logging
|
|
- Source code mounted as volume
|
|
|
|
**Production:**
|
|
- 4 worker processes
|
|
- Resource limits: 512MB RAM, 0.5 CPU
|
|
- Health checks enabled
|
|
|
|
### FastStream Worker (`faststream-worker`)
|
|
|
|
**Development:**
|
|
- Hot reload enabled
|
|
- Single worker process
|
|
|
|
**Production:**
|
|
- 4 worker processes per container
|
|
- 2 container replicas (8 total workers)
|
|
- Resource limits: 256MB RAM, 0.5 CPU
|
|
- Automatic restart on failure
|
|
|
|
### FastStream Scheduler (`faststream-scheduler`)
|
|
|
|
Handles periodic tasks:
|
|
- Deals refresh: Every 6 hours
|
|
- Contacts refresh: Every 4 hours
|
|
- Companies refresh: Every 8 hours
|
|
- Users refresh: Every 12 hours
|
|
- Pipelines refresh: Daily
|
|
- Events refresh: Every 2 hours
|
|
|
|
### Redis (`redis`)
|
|
|
|
**Development:**
|
|
- 512MB memory limit
|
|
- Port 6379 exposed for debugging
|
|
|
|
**Production:**
|
|
- 1GB memory limit
|
|
- Persistence enabled with AOF
|
|
- Not exposed externally
|
|
|
|
### Nginx (`nginx`) - Production Only
|
|
|
|
- Rate limiting (10 req/s general, 1 req/s exports)
|
|
- Gzip compression
|
|
- Security headers
|
|
- SSL termination support
|
|
- Health check bypass
|
|
|
|
## Commands
|
|
|
|
### Docker Setup Script
|
|
|
|
```bash
|
|
# Setup environment
|
|
python scripts/docker_setup.py setup
|
|
|
|
# Build images
|
|
python scripts/docker_setup.py build
|
|
|
|
# Start services
|
|
python scripts/docker_setup.py start [--mode dev|prod]
|
|
|
|
# Stop services
|
|
python scripts/docker_setup.py stop
|
|
|
|
# Restart services
|
|
python scripts/docker_setup.py restart [--mode dev|prod]
|
|
|
|
# View logs
|
|
python scripts/docker_setup.py logs [--service SERVICE] [--follow]
|
|
|
|
# Check status
|
|
python scripts/docker_setup.py status
|
|
```
|
|
|
|
### Manual Docker Compose
|
|
|
|
```bash
|
|
# Development
|
|
docker-compose up -d
|
|
docker-compose logs -f
|
|
|
|
# Production
|
|
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
|
|
|
|
# Stop
|
|
docker-compose down
|
|
|
|
# Rebuild
|
|
docker-compose build --no-cache
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
### Health Checks
|
|
|
|
All services include health checks:
|
|
|
|
```bash
|
|
# Check all services
|
|
docker-compose ps
|
|
|
|
# Check specific service health
|
|
docker-compose exec app curl http://localhost:8000/health
|
|
```
|
|
|
|
### Logs
|
|
|
|
```bash
|
|
# All services
|
|
docker-compose logs
|
|
|
|
# Specific service
|
|
docker-compose logs faststream-worker
|
|
|
|
# Follow logs
|
|
docker-compose logs -f app
|
|
|
|
# Last 100 lines
|
|
docker-compose logs --tail=100 faststream-scheduler
|
|
```
|
|
|
|
### Resource Usage
|
|
|
|
```bash
|
|
# Container stats
|
|
docker stats
|
|
|
|
# Service resource usage
|
|
docker-compose exec app ps aux
|
|
docker-compose exec faststream-worker free -h
|
|
```
|
|
|
|
## Scaling
|
|
|
|
### Horizontal Scaling
|
|
|
|
Scale FastStream workers:
|
|
|
|
```bash
|
|
# Scale to 4 worker containers
|
|
docker-compose up -d --scale faststream-worker=4
|
|
|
|
# Production scaling (in docker-compose.prod.yml)
|
|
# Edit replicas value for faststream-worker service
|
|
```
|
|
|
|
### Vertical Scaling
|
|
|
|
Edit resource limits in `docker-compose.prod.yml`:
|
|
|
|
```yaml
|
|
services:
|
|
faststream-worker:
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 512M # Increase from 256M
|
|
cpus: '1.0' # Increase from 0.5
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Redis Connection Failed**
|
|
```bash
|
|
# Check Redis health
|
|
docker-compose exec redis redis-cli ping
|
|
|
|
# Check Redis logs
|
|
docker-compose logs redis
|
|
```
|
|
|
|
2. **Worker Not Processing Jobs**
|
|
```bash
|
|
# Check worker logs
|
|
docker-compose logs faststream-worker
|
|
|
|
# Restart workers
|
|
docker-compose restart faststream-worker
|
|
```
|
|
|
|
3. **High Memory Usage**
|
|
```bash
|
|
# Check memory usage
|
|
docker stats
|
|
|
|
# Reduce worker processes or add memory limits
|
|
```
|
|
|
|
4. **Permission Issues**
|
|
```bash
|
|
# Fix data directory permissions
|
|
sudo chown -R 1000:1000 ./data
|
|
sudo chown -R 1000:1000 ./credentials
|
|
```
|
|
|
|
### Debug Mode
|
|
|
|
Enable debug logging:
|
|
|
|
```bash
|
|
# Set in .env file
|
|
LOG_LEVEL=DEBUG
|
|
|
|
# Restart services
|
|
docker-compose restart
|
|
```
|
|
|
|
### Database Issues
|
|
|
|
```bash
|
|
# Access SQLite database
|
|
docker-compose exec app sqlite3 /app/data/amo_data.db
|
|
|
|
# Run migrations
|
|
docker-compose exec app alembic upgrade head
|
|
```
|
|
|
|
## Backup and Recovery
|
|
|
|
### Database Backup
|
|
|
|
```bash
|
|
# Backup SQLite database
|
|
docker-compose exec app cp /app/data/amo_data.db /app/data/amo_data.db.backup
|
|
|
|
# Copy to host
|
|
docker cp $(docker-compose ps -q app):/app/data/amo_data.db ./backup/
|
|
```
|
|
|
|
### Redis Backup
|
|
|
|
```bash
|
|
# Redis automatically saves to /data/dump.rdb
|
|
# Volume is mounted to redis_data
|
|
|
|
# Manual backup
|
|
docker-compose exec redis redis-cli BGSAVE
|
|
```
|
|
|
|
### Configuration Backup
|
|
|
|
```bash
|
|
# Backup configuration files
|
|
tar -czf backup/config-$(date +%Y%m%d).tar.gz .env credentials/
|
|
```
|
|
|
|
## Security
|
|
|
|
### Production Security
|
|
|
|
1. **Environment Variables**: Never commit `.env` files
|
|
2. **Credentials**: Store in secure volume, not in image
|
|
3. **Network**: Use internal networks, don't expose Redis
|
|
4. **SSL**: Configure SSL certificates for HTTPS
|
|
5. **Rate Limiting**: Nginx provides rate limiting
|
|
6. **Updates**: Regularly update base images
|
|
|
|
### SSL Configuration
|
|
|
|
1. Place certificates in `./ssl/` directory
|
|
2. Update `nginx.conf` for HTTPS
|
|
3. Restart nginx service
|
|
|
|
## Performance Tuning
|
|
|
|
### Redis Optimization
|
|
|
|
```bash
|
|
# In docker-compose.prod.yml
|
|
command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru --tcp-backlog 511
|
|
```
|
|
|
|
### Worker Optimization
|
|
|
|
- Adjust worker count based on CPU cores
|
|
- Monitor memory usage and adjust limits
|
|
- Use connection pooling for database
|
|
|
|
### Database Optimization
|
|
|
|
- Regular VACUUM for SQLite
|
|
- Consider PostgreSQL for high load
|
|
- Index optimization
|
|
|
|
## Migration from Celery
|
|
|
|
If migrating from a Celery-based deployment:
|
|
|
|
1. Stop Celery workers: `docker-compose stop celery-worker celery-beat`
|
|
2. Update code to use FastStream
|
|
3. Start FastStream services: `docker-compose up -d faststream-worker faststream-scheduler`
|
|
4. Remove Celery services from docker-compose.yml
|
|
|
|
The FastStream workers will process the same job queues through Redis.
|
|
|