Maxim Snesarev
33d6bb7ebd
- Updated database configuration to switch from SQLite to PostgreSQL, including changes to alembic.ini, Docker Compose, and environment settings. - Refactored application code to utilize PostgreSQL database adapters, ensuring compatibility with the new database structure. - Enhanced API routes and data handling to support the new database, including adjustments in data models and query logic. - Introduced new job processing mechanisms for full synchronization of AMO CRM entities, leveraging FastStream for background tasks. - Improved logging and error handling across the application to facilitate better monitoring and debugging. - Removed obsolete SQLite adapter files and migrations, streamlining the project structure for PostgreSQL integration.
374 lines
13 KiB
Markdown
374 lines
13 KiB
Markdown
# AMO CRM Service - Implementation Status
|
|
|
|
## ✅ COMPLETED IMPLEMENTATION
|
|
|
|
All core functionality has been implemented and is ready for testing and deployment.
|
|
|
|
---
|
|
|
|
## 🎯 Completed Components
|
|
|
|
### 1. **AMO CRM API Client** ✅
|
|
- **File**: `adapters/amocrm_client.py`
|
|
- **Features**:
|
|
- Generic `fetch_entity_data()` method for all entity types
|
|
- Support for pagination, incremental updates, and filtering
|
|
- Proper type annotations
|
|
- All entity-specific methods (deals, contacts, companies, users, pipelines, events)
|
|
|
|
### 2. **Google Sheets Integration** ✅
|
|
- **File**: `adapters/google_sheets_client.py`
|
|
- **Features**:
|
|
- Service account authentication
|
|
- Write, clear, and append data operations
|
|
- Automatic sheet creation
|
|
- Header row formatting (bold, frozen)
|
|
- Batch updates support
|
|
|
|
### 3. **Sync Server** ✅
|
|
- **File**: `servers/sync_server.py`
|
|
- **Features**:
|
|
- Process sync jobs from FastStream queue
|
|
- Full sync and incremental sync support
|
|
- Integration with data ingestion endpoints
|
|
- Incremental update tracking via entity timestamps
|
|
- Batch processing with configurable sizes
|
|
|
|
### 4. **Job Server** ✅
|
|
- **File**: `servers/job_server.py`
|
|
- **Features**:
|
|
- Queue export and sync jobs via Redis/FastStream
|
|
- Job status tracking in database
|
|
- List and filter jobs
|
|
- Update job progress and status
|
|
|
|
### 5. **Export Server** ✅
|
|
- **File**: `servers/export_server.py`
|
|
- **Features**:
|
|
- Process export jobs from queue
|
|
- Retrieve data from database with date filtering
|
|
- Format data according to field mappings
|
|
- Export to Google Sheets with proper formatting
|
|
- Support for custom fields
|
|
- Job progress tracking
|
|
|
|
### 6. **API Routers** ✅
|
|
All routers are complete and functional:
|
|
|
|
#### a. AMO CRM Router (`routers/amocrm.py`)
|
|
- Fetch entities directly from AMO CRM
|
|
- Fetch custom field metadata
|
|
- **Sync endpoint** - fetches from AMO CRM and stores in database
|
|
- Connection info endpoint
|
|
|
|
#### b. Data Ingestion Router (`routers/data.py`)
|
|
- Endpoints for all entity types (deals, contacts, companies, users, pipelines, events)
|
|
- Support for custom fields and relationships
|
|
- Upsert, insert, and replace modes
|
|
|
|
#### c. Export Router (`routers/export.py`)
|
|
- Create and manage export configurations
|
|
- Start export jobs (async processing)
|
|
- Track job status and progress
|
|
- Queue sync jobs
|
|
- Schedule refresh jobs
|
|
|
|
### 7. **Background Workers** ✅
|
|
- **File**: `workers/broker.py`
|
|
- **Features**:
|
|
- Process export jobs
|
|
- Process sync jobs
|
|
- Process scheduled refresh jobs
|
|
- Dead letter queue handling
|
|
- Logging and error handling
|
|
|
|
### 8. **Task Scheduler** ✅
|
|
- **File**: `workers/scheduler.py`
|
|
- **Features**:
|
|
- Periodic data refresh for all entity types
|
|
- Configurable schedules (cron-based)
|
|
- Different intervals per entity type
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ AMO CRM API │
|
|
│ (wecheap.amocrm.ru) │
|
|
└─────────────────────┬───────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ AMO CRM API Client │
|
|
│ (adapters/amocrm_client.py) │
|
|
└─────────────────────┬───────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ FastAPI Routers │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ AMO CRM │ │ Data │ │ Export │ │
|
|
│ │ Router │ │ Ingestion │ │ Router │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└───────────┬────────────────┬────────────────┬──────────────────┘
|
|
│ │ │
|
|
↓ ↓ ↓
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ SQLite Database │
|
|
│ (Users, Pipelines, Deals, Contacts, Companies, Events, │
|
|
│ Custom Fields, Export Configurations, Jobs) │
|
|
└───────────────────┬─────────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ FastStream/Redis Message Broker │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Export Queue │ │ Sync Queue │ │Refresh Queue │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└───────────┬────────────────┬────────────────┬──────────────────┘
|
|
│ │ │
|
|
↓ ↓ ↓
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Background Workers │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │Export Server │ │ Sync Server │ │ Scheduler │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└───────────┬─────────────────────────────────────────────────────┘
|
|
│
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Google Sheets API │
|
|
│ (Google Sheets Client) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 How to Use
|
|
|
|
### 1. Setup Environment Variables
|
|
Create a `.env` file with:
|
|
```env
|
|
# AMO CRM Configuration
|
|
AMO_CRM_DOMAIN=wecheap.amocrm.ru
|
|
AMO_CRM_ACCESS_TOKEN=your_token_here
|
|
|
|
# Google Sheets Configuration
|
|
GOOGLE_SERVICE_ACCOUNT_FILE=credentials/google.json
|
|
GOOGLE_SCOPES=https://www.googleapis.com/auth/spreadsheets
|
|
|
|
# Redis Configuration
|
|
REDIS_URL=redis://localhost:6379/0
|
|
|
|
# API Configuration
|
|
API_V1_STR=/api/v1
|
|
LOG_LEVEL=INFO
|
|
```
|
|
|
|
### 2. Run Database Migrations
|
|
```bash
|
|
cd adapters/sqlite
|
|
alembic upgrade head
|
|
```
|
|
|
|
### 3. Start Services
|
|
|
|
#### Option A: Development (separate terminals)
|
|
```bash
|
|
# Terminal 1: FastAPI server
|
|
uvicorn app:app --reload --host 0.0.0.0 --port 8000
|
|
|
|
# Terminal 2: FastStream worker
|
|
python -m workers.broker
|
|
|
|
# Terminal 3 (optional): Task scheduler
|
|
python -m workers.scheduler
|
|
```
|
|
|
|
#### Option B: Docker Compose
|
|
```bash
|
|
docker-compose up -d
|
|
```
|
|
|
|
### 4. Test the Service
|
|
|
|
#### Fetch data from AMO CRM:
|
|
```bash
|
|
curl http://localhost:8000/api/v1/amocrm/fetch/deals?limit=10
|
|
```
|
|
|
|
#### Sync data to database:
|
|
```bash
|
|
curl -X POST http://localhost:8000/api/v1/amocrm/sync/deals?limit=250
|
|
```
|
|
|
|
#### Create export configuration:
|
|
```bash
|
|
curl -X POST http://localhost:8000/api/v1/export/configure \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"name": "Monthly Export",
|
|
"sheet_id": "YOUR_GOOGLE_SHEET_ID",
|
|
"entity_mappings": {
|
|
"deals": {
|
|
"sheet_name": "Deals",
|
|
"is_enabled": true,
|
|
"field_mapping": [
|
|
{"field_name": "id", "column": "A", "order": 1},
|
|
{"field_name": "name", "column": "B", "order": 2},
|
|
{"field_name": "price", "column": "C", "order": 3}
|
|
]
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
#### Start export job:
|
|
```bash
|
|
curl -X POST http://localhost:8000/api/v1/export/start \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"configuration_id": 1}'
|
|
```
|
|
|
|
#### Check job status:
|
|
```bash
|
|
curl http://localhost:8000/api/v1/export/status/1
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 API Endpoints Summary
|
|
|
|
### AMO CRM Routes (`/api/v1/amocrm`)
|
|
- `GET /fetch/{entity_type}` - Fetch data directly from AMO CRM
|
|
- `GET /fetch/custom_fields/{entity_type}` - Fetch custom field metadata
|
|
- `GET /fetch/all` - Fetch all data for testing
|
|
- `POST /sync/{entity_type}` - Sync data from AMO CRM to database
|
|
- `GET /info` - Get connection info
|
|
|
|
### Data Ingestion Routes (`/api/v1/data`)
|
|
- `POST /users` - Ingest users
|
|
- `POST /pipelines` - Ingest pipelines
|
|
- `POST /companies` - Ingest companies
|
|
- `POST /contacts` - Ingest contacts
|
|
- `POST /deals` - Ingest deals
|
|
- `POST /events` - Ingest events
|
|
|
|
### Export Routes (`/api/v1/export`)
|
|
- `POST /configure` - Create export configuration
|
|
- `GET /configurations` - List configurations
|
|
- `POST /start` - Start export job
|
|
- `GET /status/{job_id}` - Get job status
|
|
- `GET /jobs` - List all jobs
|
|
- `POST /sync` - Queue sync job
|
|
- `POST /refresh/{entity_type}` - Schedule refresh
|
|
|
|
---
|
|
|
|
## 🔧 Configuration
|
|
|
|
### Google Sheets Setup
|
|
1. Create a service account in Google Cloud Console
|
|
2. Download the JSON key file
|
|
3. Save it to `credentials/google.json`
|
|
4. Share your Google Sheet with the service account email
|
|
|
|
### Redis Setup
|
|
For Docker:
|
|
```yaml
|
|
services:
|
|
redis:
|
|
image: redis:7-alpine
|
|
ports:
|
|
- "6379:6379"
|
|
```
|
|
|
|
For local development:
|
|
```bash
|
|
# Install Redis
|
|
# Ubuntu/Debian
|
|
sudo apt-get install redis-server
|
|
|
|
# macOS
|
|
brew install redis
|
|
|
|
# Start Redis
|
|
redis-server
|
|
```
|
|
|
|
---
|
|
|
|
## ⏰ Scheduled Tasks
|
|
|
|
The scheduler runs periodic data refreshes:
|
|
|
|
| Entity Type | Schedule | Cron Expression |
|
|
|------------|----------|----------------|
|
|
| Events | Every 2 hours | `0 */2 * * *` |
|
|
| Contacts | Every 4 hours | `0 */4 * * *` |
|
|
| Deals | Every 6 hours | `0 */6 * * *` |
|
|
| Companies | Every 8 hours | `0 */8 * * *` |
|
|
| Users | Every 12 hours | `0 */12 * * *` |
|
|
| Pipelines | Daily | `0 */24 * * *` |
|
|
|
|
---
|
|
|
|
## 🐛 Known Issues & Notes
|
|
|
|
### Type Checker Warnings
|
|
Some linter warnings about SQLAlchemy Column assignments are false positives. SQLAlchemy uses the descriptor protocol which confuses static type checkers. These don't affect runtime behavior.
|
|
|
|
### Google Sheets API
|
|
- Requires service account credentials
|
|
- Rate limits: 100 requests per 100 seconds per user
|
|
- Each sheet can have max 10 million cells
|
|
|
|
### Database
|
|
- SQLite is used for simplicity
|
|
- For production, consider PostgreSQL for better concurrency
|
|
- Regular backups recommended
|
|
|
|
---
|
|
|
|
## 📊 Next Steps
|
|
|
|
1. **Testing**:
|
|
- Test with real AMO CRM credentials
|
|
- Verify Google Sheets export
|
|
- Load testing with large datasets
|
|
|
|
2. **Deployment**:
|
|
- Set up production environment variables
|
|
- Configure NGINX reverse proxy
|
|
- Set up monitoring and logging
|
|
|
|
3. **Enhancements**:
|
|
- Add authentication/authorization
|
|
- Implement webhook receivers for real-time updates
|
|
- Add more entity types if needed
|
|
- Implement data validation and error recovery
|
|
|
|
4. **Documentation**:
|
|
- API documentation with OpenAPI/Swagger
|
|
- User guide for configuration
|
|
- Troubleshooting guide
|
|
|
|
---
|
|
|
|
## 🎉 Summary
|
|
|
|
The AMO CRM Data Collection Service is now **fully implemented** with:
|
|
|
|
✅ Complete AMO CRM API integration
|
|
✅ SQLite database with all entity types
|
|
✅ Background job processing with FastStream/Redis
|
|
✅ Google Sheets export functionality
|
|
✅ Scheduled data synchronization
|
|
✅ RESTful API with comprehensive endpoints
|
|
✅ Docker support for easy deployment
|
|
|
|
The service is ready for testing and deployment!
|
|
|