amo-server/README.md

# AMO CRM Data Collection Service

A FastAPI service that collects data from AMO CRM API and stores it in SQLite database for further export to Google Sheets.

## Features

- **Data Collection**: Collects Deals, Contacts, Companies, Pipelines, Users, and Events from AMO CRM
- **HAL+JSON Support**: Handles AMO CRM's HAL+JSON response format
- **Custom Fields**: Extracts and stores custom fields with proper typing
- **Relationships**: Maintains proper relationships between entities
- **Google Sheets Export**: Unified configuration for exporting all entities to Google Sheets
- **Background Processing**: Async job processing for exports
- **SQLite Database**: Optimized storage with proper indexing

## Project Structure

```
amo-server/
├── adapters/           # Database and external service adapters
│   └── sqlite/         # SQLite database models and connection
├── routers/            # FastAPI route handlers
├── servers/            # Business logic (to be implemented)
├── utils/              # Utilities and configuration
├── workers/            # Background workers (to be implemented)
├── tests/              # Test files and fixtures
│   └── fixtures/       # AMO CRM response examples
└── docs/               # Documentation
```

## Setup

### Prerequisites

- Python 3.12+
- [uv](https://docs.astral.sh/uv/) package manager

### Installation

1. Clone the repository:
```bash
git clone <repository-url>
cd amo-server
```

2. Install dependencies using uv:
```bash
uv sync
```

3. Create environment file:
```bash
cp env.example .env
```

4. Configure your environment variables in `.env`:
```env
# Database
DATABASE_URL=sqlite:///./amo_data.db

# AMO CRM API
AMO_CRM_DOMAIN=wecheap.amocrm.ru
AMO_CRM_ACCESS_TOKEN=your-longterm-access-token

# Google Sheets API
GOOGLE_SERVICE_ACCOUNT_FILE=path/to/service-account.json

# Redis (for Celery)
REDIS_URL=redis://localhost:6379/0

# API Settings
API_V1_STR=/api/v1

# Logging
LOG_LEVEL=INFO
```

### Running the Service

1. Start the development server:
```bash
uv run uvicorn app:app --reload
```

2. Open your browser to http://localhost:8000 to see the API

3. Visit http://localhost:8000/docs for the interactive API documentation

### Fetching Real AMO CRM Data

To fetch real responses from your AMO CRM for testing:

1. Set your access token in `.env`:
```env
AMO_CRM_ACCESS_TOKEN=your-longterm-token
```

2. Run the fetch script:
```bash
uv run python scripts/fetch_amocrm_data.py
```

This will:
- Fetch real data from wecheap.amocrm.ru
- Save individual JSON files to `tests/fixtures/real_responses/`
- Generate Python fixtures at `tests/fixtures/real_amocrm_responses.py`
- Show a summary of fetched data

3. Test the API connection:
```bash
curl http://localhost:8000/api/v1/amocrm/info
curl http://localhost:8000/api/v1/amocrm/fetch/users?limit=5
```

## API Endpoints

### Entity Management

- `GET /api/v1/entities` - List all entities with statistics
- `GET /api/v1/entities/{entity_type}/fields` - List fields for specific entity

### AMO CRM Integration

- `GET /api/v1/amocrm/info` - Get AMO CRM connection info
- `GET /api/v1/amocrm/fetch/{entity_type}` - Fetch data directly from AMO CRM
- `GET /api/v1/amocrm/fetch/custom_fields/{entity_type}` - Fetch custom fields metadata
- `GET /api/v1/amocrm/fetch/all` - Fetch all data from AMO CRM (for testing)
- `POST /api/v1/amocrm/sync/{entity_type}` - Fetch and sync data from AMO CRM

### Export Management

- `POST /api/v1/export/configure` - Create unified export configuration
- `GET /api/v1/export/configurations` - List export configurations
- `POST /api/v1/export/start` - Start export job
- `GET /api/v1/export/status/{job_id}` - Get export job status
- `GET /api/v1/export/jobs` - List all export jobs

### Data Ingestion (Worker API)

- `POST /api/v1/data/users` - Import users data
- `POST /api/v1/data/pipelines` - Import pipelines data
- `POST /api/v1/data/companies` - Import companies data
- `POST /api/v1/data/contacts` - Import contacts data
- `POST /api/v1/data/deals` - Import deals data
- `POST /api/v1/data/events` - Import events data

## Database Schema

The service uses SQLite with the following main tables:

- `amo_users` - CRM users
- `amo_pipelines` - Sales pipelines
- `amo_pipeline_stages` - Pipeline stages
- `amo_companies` - Companies
- `amo_contacts` - Contacts
- `amo_deals` - Deals/Leads
- `amo_events` - Activity events
- `amo_custom_fields` - Universal custom fields storage
- `export_configuration` - Export configurations
- `export_entity_mappings` - Entity-specific export mappings
- `export_jobs` - Export job tracking

## AMO CRM Response Examples

The project includes real AMO CRM API response examples in `tests/fixtures/amocrm_responses.py` for:

- **Users** (`USERS_RESPONSE`)
- **Pipelines** (`PIPELINES_RESPONSE`)
- **Companies** (`COMPANIES_RESPONSE`)
- **Contacts** (`CONTACTS_RESPONSE`)
- **Deals** (`DEALS_RESPONSE`)
- **Events** (`EVENTS_RESPONSE`)
- **Custom Fields Metadata** (`CUSTOM_FIELDS_RESPONSE`)

These examples demonstrate:
- HAL+JSON format structure
- Custom fields with different types (text, select, numeric, date)
- Embedded relationships between entities
- Proper date formats (Unix timestamps)
- Event types filtering (incoming_call, outgoing_call, lead_status_changed)

## Testing

Run tests using pytest:

```bash
uv run pytest
```

Run tests with coverage:

```bash
uv run pytest --cov=. --cov-report=html
```

## Export Configuration Example

Create a unified export configuration for all entities:

```json
{
  "name": "Q1 2024 AMO CRM Export",
  "sheet_id": "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms",
  "date_range_start": "2024-01-01T00:00:00Z",
  "date_range_end": "2024-03-31T23:59:59Z",
  "entity_mappings": {
    "deals": {
      "sheet_name": "Deals",
      "is_enabled": true,
      "field_mapping": [
        {"field_name": "name", "column": "A", "order": 1},
        {"field_name": "price", "column": "B", "order": 2},
        {"field_name": "created_at", "column": "C", "order": 3}
      ]
    },
    "contacts": {
      "sheet_name": "Contacts",
      "is_enabled": true,
      "field_mapping": [
        {"field_name": "name", "column": "A", "order": 1},
        {"field_name": "email", "column": "B", "order": 2}
      ]
    }
  }
}
```

## Custom Fields Handling

The service automatically extracts custom fields from AMO CRM responses:

- **Field Types**: text, numeric, select, multiselect, date, checkbox, textarea, url
- **Type Detection**: Automatically detects and converts field values
- **Storage**: Stores in universal `amo_custom_fields` table with proper typing
- **Relationships**: Links custom fields to their parent entities

## Date Validation

All dates are validated to be between 2017-2026 (Unix timestamps: 1483228800 - 1767225600).

## Development

### Code Style

The project uses:
- **Black** for code formatting
- **isort** for import sorting
- **flake8** for linting
- **mypy** for type checking

Run code formatting:
```bash
uv run black .
uv run isort .
```

### Project Conventions

Following the established patterns:
- **Adapters**: Handle external integrations (database, APIs)
- **Servers**: Contain business logic
- **Routers**: Define API routes only
- **Workers**: Background processing
- **Utils**: Shared utilities and configuration

## Next Steps

1. **Implement Background Workers**: Set up Celery workers for data collection
2. **Add Google Sheets Integration**: Implement actual Google Sheets export
3. **Add AMO CRM Client**: Create HTTP client for AMO CRM API
4. **Add Authentication**: Implement API authentication
5. **Add Monitoring**: Set up logging and metrics
6. **Add Rate Limiting**: Implement API rate limiting
7. **Add Caching**: Use Redis for caching frequently accessed data

## License

MIT License