Dereja M&E System - Docker Architecture
Introduction
This documentation provides a comprehensive overview of the Docker-based architecture for the Dereja M&E System. The system is containerized using Docker and orchestrated with Docker Compose, making it portable and scalable across different environments.
Key Features
- Containerized Architecture: All components run in isolated Docker containers
- Microservices Design: Independent services for web, database, analytics, etc.
- CI/CD Pipeline: Automated build and deployment with GitLab CI
- Scalability: Easily scale individual services as needed
- Environment Consistency: Consistent environments from development to production
- Infrastructure as Code: All infrastructure defined in Docker Compose files
System Overview
Component | Description |
---|---|
Web Application | PHP-based application with Nginx web server |
Database | MariaDB for application data and PostgreSQL for analytics |
Analytics | Metabase for business intelligence and reporting |
Chatbot | Python-based Streamlit application for AI interactions |
Management Tools | phpMyAdmin, pgAdmin, and Portainer for administration |
Getting Started
Prerequisites
Before you begin, ensure you have the following installed:
- Docker (minimum version 20.10.x)
- Docker Compose (minimum version 1.29.x)
- Git (for cloning the repository)
Installation Steps
1 Clone the Repository
2 Start the System
3 Access Services
Service | URL |
---|---|
Main Application | http://localhost:8880 |
Metabase Analytics | http://localhost:4000 |
phpMyAdmin | http://localhost:3307 |
Portainer | http://localhost:9099 |
AI Chatbot | http://localhost:8502 |
System Architecture
The Dereja M&E system follows a containerized microservices architecture. Each component runs in its own Docker container, communicating through a dedicated network.
Data Flow
The system follows this data flow pattern:
- User requests are handled by Nginx
- Static content is served directly by Nginx
- PHP requests are forwarded to the PHP-FPM service
- Application data is stored in MariaDB
- Analytics data is stored in PostgreSQL
- Metabase queries PostgreSQL for reporting
- Chatbot service provides AI-powered interactions
Network Architecture
The system uses a custom bridge network (derejanet
) with a fixed subnet (192.168.160.0/24). Each service has a static IP address within this subnet:
Service | IP Address | Description |
---|---|---|
webdata | 192.168.160.55 | Shared volume service |
php | 192.168.160.10 | Application backend |
nginx | 192.168.160.20 | Web server |
metabase | 192.168.160.30 | Analytics dashboard |
postgres | 192.168.160.40 | Analytics database |
db (MariaDB) | 192.168.160.50 | Application database |
Docker Services
The Docker Compose file defines the following services that make up the Dereja M&E system:
webdata
Shared volume service that syncs source code to the web root.
- Image: php:8.2-fpm-alpine
- Volumes: web_content, source code
- Network: derejanet (192.168.160.55)
php
PHP-FPM service running the application backend.
- Build: Custom Dockerfile
- Exposes: Port 9009
- Depends on: db, webdata
- Network: derejanet (192.168.160.10)
nginx
Web server handling HTTP/HTTPS traffic.
- Build: Custom Dockerfile (nginx_extended)
- Ports: 8880:80, 8443:443
- Depends on: php, webdata, metabase
- Network: derejanet (192.168.160.20)
metabase
Business intelligence and analytics dashboard.
- Image: metabase/metabase:latest
- Ports: 4000:3000
- Depends on: postgres
- Network: derejanet (192.168.160.30)
db (MariaDB)
Primary application database.
- Image: mariadb:latest
- Ports: 3306:3306
- Volumes: mysql_data
- Network: derejanet (192.168.160.50)
chat
AI-powered chatbot service.
- Build: Custom Dockerfile (ai_chatbot)
- Ports: 8502:8502
- Network: derejanet (192.168.160.100)
Build Process
The system is built using a multi-stage Dockerfile that creates optimized images for different components.
Dockerfile Structure
The Dockerfile defines three build targets:
- base: PHP-FPM environment with all dependencies
- nginx_extended: Custom Nginx image with additional tools
- ai_chatbot: Python environment for the Streamlit chatbot
Building the System
To build the entire system:
Building Individual Components
To build specific services:
Deployment
The system can be deployed to different environments using Docker Compose.
Starting the System
To start all services in detached mode:
Stopping the System
To stop all services while preserving data volumes:
Full Cleanup
To stop all services and remove all resources:
Environment-Specific Deployment
The system supports different deployment environments:
- Development: Includes volume syncing for live code updates
- Production: Uses optimized images with proper SSL configuration
Environment | Compose File | Command |
---|---|---|
Development | docker-compose.yml | docker-compose -p derejame up -d |
Production | prod-docker-compose.yml | docker-compose -f prod-docker-compose.yml -p derejame up -d |
Configuration
The system is configured through environment variables and configuration files.
Key Configuration Files
- docker-compose.yml: Main service definitions and configurations
- Dockerfile: Build instructions for custom images
- config/nginx.conf: Nginx server configuration
- config/php.ini: PHP runtime configuration
- .env: Environment variables for development
- .env.prod: Environment variables for production
Environment Variables
Variable | Service | Description | Default |
---|---|---|---|
MYSQL_ROOT_PASSWORD | db | MariaDB root password | merqderejadmin |
MYSQL_DATABASE | db | Application database name | me_dereja |
MB_ADMIN_EMAIL | metabase | Metabase admin email | dev@merqconsultancy.org |
MB_ADMIN_PASSWORD | metabase | Metabase admin password | merqderejadev0 |
PGADMIN_DEFAULT_EMAIL | pgadmin | pgAdmin admin email | dev@merqconsultancy.org |
PGADMIN_DEFAULT_PASSWORD | pgadmin | pgAdmin admin password | merqderejadev |
CI/CD Pipeline
The system uses GitLab CI for automated build and deployment.
Pipeline Stages
- Build: Create Docker images for application and chatbot
- Deploy: Deploy to production environment
Build Stage
Builds application and chatbot images and pushes to Docker Hub:
Deploy Stage
Deploys the application to production environment:
API Documentation
The system uses multiple APIs for different components and resources. For detailed API documentation, see the API Documentation.
The Dereja M&E system provides RESTful APIs for integration with other systems using PHPRunner's REST API implementation.
Security
All API endpoints require authentication. Unauthorized access returns:
HTTP Basic Authentication
Authenticate using username and password:
API Key Authentication
Authenticate using X-Auth-Token header:
Advanced Security
Row-level security from your project automatically applies to API calls. Use inRestApi()
in event handlers:
API Endpoints
All endpoints use format: https://mne.dereja.com/app/api/v1.php?table=candidates&action={action}
1. List Candidates
Retrieve candidate records with pagination and filtering
Parameter | Description | Example |
---|---|---|
records | Number of records to return | records=10 |
skip | Records to skip (pagination) | skip=20 |
q | Advanced search filter | q=(Status~equals~Active) |
qs | Full-text search | qs=Abebe |
2. View Candidate
Get details of a specific candidate
3. Insert Candidate
Create a new candidate record
4. Update Candidate
Update an existing candidate record
5. Delete Candidate
Delete a candidate record
Advanced Functions
sendError($message, $errno)
Send custom error responses from API handlers
inRestApi()
Check if current execution is via REST API
Troubleshooting
Issue | Solution |
---|---|
Authentication failures | Verify credentials and authentication method (Basic vs API Key) |
Unexpected redirects | Wrap redirects in if (!inRestApi()) condition |
Debugging API calls | Use verbose mode: curl -v "http://..." |
Filter not working | Verify filter syntax using Advanced Search in web interface |
Permission issues | Check user roles and row-level security settings |
Database Documentation
The system uses multiple databases for different services. For detailed database schema documentation, see the Database Schema Documentation.
Database | Type | Username | Password | Host | Port |
---|---|---|---|---|---|
me_dereja | MariaDB | me_dereja | me_dereja | db | 3306 |
merqderejadb | PostgreSQL | merqderejadb | merqderejadb | postgres | 5432 |
MongoDB | MongoDB | merqderejadev | merqderejadev | mongo | 27017 |
Database Access Commands
Virtualmin Hosting
Production deployment on Virtualmin requires specific configuration for Docker services.
Virtualmin Server Requirements
- Ubuntu 22.04 LTS
- 4 vCPU cores
- 8GB RAM
- 50GB storage
- Docker and Docker Compose installed
Configuration Steps
1 Create Virtual Server
2 Configure Proxy
3 SSL Configuration
Deployment to Production
Troubleshooting
Common issues and solutions for the Docker-based Dereja M&E system.
Common Problems
Issue | Possible Cause | Solution |
---|---|---|
Containers not starting | Port conflicts | Check for other services using ports 3306, 80, 443, etc. |
Database connection errors | Incorrect credentials | Verify environment variables in .env file |
Permission issues | Volume ownership | Run: chown -R www-data:www-data /var/www/html |
Metabase not loading | PostgreSQL connection issue | Check PostgreSQL logs and connection settings |
Chatbot not responding | Python dependencies | Reinstall requirements: pip install -r requirements.txt |