AMO CRM Data Collection Service

A FastAPI service that collects data from AMO CRM API and stores it in SQLite database for further export to Google Sheets.

Features

• Data Collection: Collects Deals, Contacts, Companies, Pipelines, Users, and Events from AMO CRM
• HAL+JSON Support: Handles AMO CRM's HAL+JSON response format
• Custom Fields: Extracts and stores custom fields with proper typing
• Relationships: Maintains proper relationships between entities
• Google Sheets Export: Unified configuration for exporting all entities to Google Sheets
• Background Processing: Async job processing for exports
• SQLite Database: Optimized storage with proper indexing

Project Structure

amo-server/
├── adapters/           # Database and external service adapters
│   └── sqlite/         # SQLite database models and connection
├── routers/            # FastAPI route handlers
├── servers/            # Business logic (to be implemented)
├── utils/              # Utilities and configuration
├── workers/            # Background workers (to be implemented)
├── tests/              # Test files and fixtures
│   └── fixtures/       # AMO CRM response examples
└── docs/               # Documentation

Setup

Prerequisites

• Python 3.12+
• uv package manager

Installation

1. Clone the repository:

git clone <repository-url>
cd amo-server

2. Install dependencies using uv:

uv sync

3. Create environment file:

cp env.example .env

4. Configure your environment variables in .env:

# Database
DATABASE_URL=sqlite:///./amo_data.db

# AMO CRM API
AMO_CRM_DOMAIN=wecheap.amocrm.ru
AMO_CRM_ACCESS_TOKEN=your-longterm-access-token

# Google Sheets API
GOOGLE_SERVICE_ACCOUNT_FILE=path/to/service-account.json

# Redis (for Celery)
REDIS_URL=redis://localhost:6379/0

# API Settings
API_V1_STR=/api/v1

# Logging
LOG_LEVEL=INFO

Running the Service

1. Start the development server:

uv run uvicorn app:app --reload

2. Open your browser to http://localhost:8000 to see the API

3. Visit http://localhost:8000/docs for the interactive API documentation

Fetching Real AMO CRM Data

To fetch real responses from your AMO CRM for testing:

1. Set your access token in .env:

AMO_CRM_ACCESS_TOKEN=your-longterm-token

2. Run the fetch script:

uv run python scripts/fetch_amocrm_data.py

This will:

• Fetch real data from wecheap.amocrm.ru
• Save individual JSON files to tests/fixtures/real_responses/
• Generate Python fixtures at tests/fixtures/real_amocrm_responses.py
• Show a summary of fetched data

3. Test the API connection:

curl http://localhost:8000/api/v1/amocrm/info
curl http://localhost:8000/api/v1/amocrm/fetch/users?limit=5

API Endpoints

Entity Management

• GET /api/v1/entities - List all entities with statistics
• GET /api/v1/entities/{entity_type}/fields - List fields for specific entity

AMO CRM Integration

• GET /api/v1/amocrm/info - Get AMO CRM connection info
• GET /api/v1/amocrm/fetch/{entity_type} - Fetch data directly from AMO CRM
• GET /api/v1/amocrm/fetch/custom_fields/{entity_type} - Fetch custom fields metadata
• GET /api/v1/amocrm/fetch/all - Fetch all data from AMO CRM (for testing)
• POST /api/v1/amocrm/sync/{entity_type} - Fetch and sync data from AMO CRM

Export Management

• POST /api/v1/export/configure - Create unified export configuration
• GET /api/v1/export/configurations - List export configurations
• POST /api/v1/export/start - Start export job
• GET /api/v1/export/status/{job_id} - Get export job status
• GET /api/v1/export/jobs - List all export jobs

Data Ingestion (Worker API)

• POST /api/v1/data/users - Import users data
• POST /api/v1/data/pipelines - Import pipelines data
• POST /api/v1/data/companies - Import companies data
• POST /api/v1/data/contacts - Import contacts data
• POST /api/v1/data/deals - Import deals data
• POST /api/v1/data/events - Import events data

Database Schema

The service uses SQLite with the following main tables:

• amo_users - CRM users
• amo_pipelines - Sales pipelines
• amo_pipeline_stages - Pipeline stages
• amo_companies - Companies
• amo_contacts - Contacts
• amo_deals - Deals/Leads
• amo_events - Activity events
• amo_custom_fields - Universal custom fields storage
• export_configuration - Export configurations
• export_entity_mappings - Entity-specific export mappings
• export_jobs - Export job tracking

AMO CRM Response Examples

The project includes real AMO CRM API response examples in tests/fixtures/amocrm_responses.py for:

• Users (USERS_RESPONSE)
• Pipelines (PIPELINES_RESPONSE)
• Companies (COMPANIES_RESPONSE)
• Contacts (CONTACTS_RESPONSE)
• Deals (DEALS_RESPONSE)
• Events (EVENTS_RESPONSE)
• Custom Fields Metadata (CUSTOM_FIELDS_RESPONSE)

These examples demonstrate:

• HAL+JSON format structure
• Custom fields with different types (text, select, numeric, date)
• Embedded relationships between entities
• Proper date formats (Unix timestamps)
• Event types filtering (incoming_call, outgoing_call, lead_status_changed)

Testing

Run tests using pytest:

uv run pytest

Run tests with coverage:

uv run pytest --cov=. --cov-report=html

Export Configuration Example

Create a unified export configuration for all entities:

{
  "name": "Q1 2024 AMO CRM Export",
  "sheet_id": "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms",
  "date_range_start": "2024-01-01T00:00:00Z",
  "date_range_end": "2024-03-31T23:59:59Z",
  "entity_mappings": {
    "deals": {
      "sheet_name": "Deals",
      "is_enabled": true,
      "field_mapping": [
        {"field_name": "name", "column": "A", "order": 1},
        {"field_name": "price", "column": "B", "order": 2},
        {"field_name": "created_at", "column": "C", "order": 3}
      ]
    },
    "contacts": {
      "sheet_name": "Contacts",
      "is_enabled": true,
      "field_mapping": [
        {"field_name": "name", "column": "A", "order": 1},
        {"field_name": "email", "column": "B", "order": 2}
      ]
    }
  }
}

Custom Fields Handling

The service automatically extracts custom fields from AMO CRM responses:

• Field Types: text, numeric, select, multiselect, date, checkbox, textarea, url
• Type Detection: Automatically detects and converts field values
• Storage: Stores in universal amo_custom_fields table with proper typing
• Relationships: Links custom fields to their parent entities

Date Validation

All dates are validated to be between 2017-2026 (Unix timestamps: 1483228800 - 1767225600).

Development

Code Style

The project uses:

• Black for code formatting
• isort for import sorting
• flake8 for linting
• mypy for type checking

Run code formatting:

uv run black .
uv run isort .

Project Conventions

Following the established patterns:

• Adapters: Handle external integrations (database, APIs)
• Servers: Contain business logic
• Routers: Define API routes only
• Workers: Background processing
• Utils: Shared utilities and configuration

Next Steps

1. Implement Background Workers: Set up Celery workers for data collection
2. Add Google Sheets Integration: Implement actual Google Sheets export
3. Add AMO CRM Client: Create HTTP client for AMO CRM API
4. Add Authentication: Implement API authentication
5. Add Monitoring: Set up logging and metrics
6. Add Rate Limiting: Implement API rate limiting
7. Add Caching: Use Redis for caching frequently accessed data

License

MIT License