Skip to content

ROFEO: Automated LMS Production Infrastructure

Internshipshipped

Nine Ansible roles take a fresh Ubuntu server to production, with tested backups and restores.

Two months as ROFEO Academy's only engineer. My build-vs-buy study replaced a from-scratch spec with a customized Moodle 5.

AnsibleNginxPostgreSQLRedisPHPLinuxLet's Encrypt
View on GitHub

Idempotent Ansible playbooks that take a fresh Ubuntu LTS server to a running Moodle 5.x LMS for ROFEO Academy. I tested them on a dev VM on my homelab and on an Azure VM. In production it ran on a single server; ROFEO piloted it with teachers and students and no longer uses it.

The sanitized open-source distribution is published as rofeo-ansible-template. This document details the system architecture, process boundaries, state synchronization pipelines, and failure modes.

Subsystem Boundaries

I engineered the complete Ansible automation topology across nine roles: system_prep, firewall, postgres, redis, php, nginx, moodle, mailpit and backup. The main ones:

  • Web Tier (roles/nginx): Server blocks enforcing Moodle 5.x’s isolated /public webroot, FastCGI buffer allocations, loopback-only FPM metrics endpoints (/fpm-status), and automated Let’s Encrypt ACME renewal hooks.
  • Runtime & Process Pool (roles/php, roles/moodle): Dedicated PHP 8.3-FPM pool isolation (moodle), Zend Opcache parameter tuning, native systemd timer units replacing legacy cron loops, and CLI-driven non-interactive core installations.
  • State & Persistence (roles/postgres, roles/redis): PostgreSQL 16 database provisioning on template0 enforcing strict UTF-8/en_US collation, alongside a dual-database Redis deployment segregating ephemeral cache from stateful student sessions.
  • Plugin Lifecycle Pipeline: A declarative Git synchronization engine handling third-party and custom plugin deployments (theme_rofeo, local_rofeo, theme_moove) paired with automated post-pull database migrations.
  • Disaster Recovery & Backup (roles/backup): A disk-aware backup runner with Azure Blob offsite synchronization via rclone, accompanied by two recovery playbooks: targeted commit-state rollbacks and full bare-metal disaster recovery.

Architecture

INGRESS Internet / Students & Academic Staff inbound HTTPS (TLS 1.3) on TCP 443; filtered by UFW with default incoming deny policy WEB TIER Nginx 1.24+ (Strict /public Webroot) TLS termination (Let's Encrypt / Self-Signed) and dynamic slasharguments routing fastcgi_read_timeout 300s prevents 504 gateway timeouts on long grading exports blocks hidden dotfiles; loopback-only access to /fpm-status and /fpm-ping TCP 443 APPLICATION PHP 8.3-FPM (Pool: moodle) isolated dynamic pool under www-data pm.max_children = 20, max_requests = 500 slowlog captures requests taking over 10s Zend Opcache & Core 128MB shared RAM buffer 16,229 accelerated files max_input_vars = 5000 FastCGI over Unix socket STORES PostgreSQL 16 relational schemas & logs template0 / UTF-8 / en_US unprivileged moodleuser Redis DB 0 (MUC Cache) no expiry, never evicted instance: 256MB, volatile-lru ephemeral strings & cache Redis DB 1 (Sessions) the only evictable keys TTL 7,200s, lock 120s keeps sessions off PostgreSQL TCP 5432 TCP 6379 sessions AUTOMATION Systemd: moodle-cron runs admin/cli/cron.php every 60s Type=oneshot prevents worker overlap isolated from HTTP request threads Systemd: moodle-backup daily backup runner @ 03:00 UTC pre-flight check: aborts if df > 80% dumps pg_dump + dataroot tar.gz archive OFFSITE STORAGE Azure Blob Cloud Storage (via rclone) TLS-encrypted offsite replication with 7-day retention pruning cold disaster recovery source for schema dumps and moodledata rclone sync All internal services bind strictly to loopback (127.0.0.1) or Unix domain sockets. Write access is quarantined to /var/moodledata; PHP source files remain read-only to workers.

The topology decouples application layers onto local Unix Domain Sockets (UDS) and loopback interfaces (127.0.0.1), eliminating inter-process network hops:

  • Ingress: Public traffic terminates at Nginx over TCP 443 with TLS 1.3.
  • FastCGI Transport: Nginx proxies PHP requests directly to PHP-FPM over /run/php/php8.3-fpm-moodle.sock under www-data:www-data permissions (0660).
  • Database & Cache: PHP workers communicate locally with PostgreSQL on TCP 5432 and Redis on TCP 6379 via authenticated loopback bindings.
  • Cron & Background Tasks: Decoupled from web request loops entirely; a native systemd timer executes CLI workers in single-tenant batches.

State Partitioning & In-Memory Isolation

Moodle couples high-frequency relational reads/writes with heavy filesystem caching and volatile session state. The settings below were sized for concurrent exam traffic. The load figures in this section are design targets, not measurements from the pilot:

  • Session Write Load: With sessions in PostgreSQL (mdl_sessions), every active quiz participant generates continuous write transactions, and the design target was hundreds of participants at once. Sessions are offloaded entirely to Redis DB 1, off the database that holds the quiz data.
  • Dual-Database Segregation: Under allkeys-lru, a burst of cache writes (invalidation, language string compilation) can evict session keys and log students out mid-quiz. Separate databases keep a cache purge away from sessions, but they don’t change eviction, which applies to the whole instance (see below).
  • Partitioned Boundaries: Redis DB 0 holds the Moodle Universal Cache (MUC) and DB 1 holds sessions, with explicit TTLs (7,200s) and lock timeouts (120s). A cache purge flushes DB 0 only, so purging caches no longer logs students out. Memory pressure is a different story: both databases share one instance, its 256MB cap and its volatile-lru policy, which only evicts keys that have an expiry. Only the session keys have one, so they are the only keys Redis can evict, and once none are left it rejects new writes. What I’d change: give sessions their own instance with noeviction, sized for the peak number of sessions, and keep the cache on a capped instance with allkeys-lru.

Runtime & Web Process Tuning

Default configurations favor lightweight, short-lived PHP requests. LMS workloads, such as gradebook exports, bulk CSV student enrollments, and SCORM package decompression, trigger heavy compute operations:

  • FastCGI Buffer Tuning: Under standard Nginx timeouts (fastcgi_read_timeout 60s), large report generations drop with 504 Gateway Timeout. Nginx and PHP pool timeouts are aligned to 300 seconds, with client_max_body_size 128M to handle large course asset uploads.
  • Worker Sizing & Memory Recycling: The dedicated moodle FPM pool runs dynamically (pm.max_children = 20) with pm.max_requests = 500. Recycling worker processes after 500 requests mitigates slow memory leaks introduced by complex third-party plugins.
  • Zend Opcache Sizing: The default Ubuntu limit of 10,000 files causes severe cache thrashing on Moodle 5.x (which contains over 12,000 individual PHP files). opcache.max_accelerated_files is tuned to 16,229 with 128MB shared memory and max_input_vars raised to 5,000 to prevent silent form truncation on role permission matrices.
  • Systemd Cron Timers Over Legacy Crontab: Traditional /etc/crontab executions lack process state awareness. If a scheduled cron job stalls due to a heavy background indexing task, subsequent minute-based cron triggers spawn on top of it, creating worker process storms. Deploying a native Type=oneshot systemd timer unit guarantees overlapping executions can never occur.

Declarative Plugin Lifecycle & Drift

Managing extensions via the web UI violates infrastructure immutability. The template manages plugins through the moodle_plugins declarative inventory array:

  • Targeted Git Fetch: Ansible checks out each repository directly into its respective target path inside /var/www/moodle/public/ using pinned branches or tags (depth: 1, umask: 0022).
  • Conditional Schema Migrations: If and only if a plugin repository pulls new commit references, the playbook executes Moodle’s non-interactive CLI upgrade script (admin/cli/upgrade.php --non-interactive) followed by an immediate cache purge (admin/cli/purge_caches.php).
  • Ansible Error Handling Quirks: Certain plugins emit non-fatal informational warnings to stderr during schema validation. In Ansible’s default mode, any stderr stream marks the task as failed even if the process exits with status code 0. The playbook uses explicit output assertions ('System was upgraded successfully' in upgrade_result.stdout) to prevent false deployment aborts.
  • Permission Boundaries: The codebase is owned by root:www-data (0755 directories, 0644 files), restricting PHP-FPM workers to read-only access. Write permissions are quarantined strictly to /var/moodledata (0770 owned by www-data:www-data).

Disaster Recovery & Rollback Mechanics

FAILURE DETECTION Failed Upgrade / Regression schema migration error, broken theme, or fatal PHP exception on update Host Failure / Volume Loss unrecoverable VM corruption, fried disk, or catastrophic host filesystem loss RECOVERY WORKFLOW restore_moodle.yml (Targeted Rollback) reads pre_update_state.yml recorded on run checks out recorded Git SHAs for all repos drops database and imports pre-update SQL purges Redis MUC cache and exits maintenance disaster_recovery.yml (Bare-Metal DR) scans local backups & Azure Blob via rclone prompts operator for timestamp ID to fetch recreates DB with template0 / en_US.UTF-8 restores dataroot archive with 0770 mode automated rollback cloud disaster recovery OBJECTIVES Target: Rapid Rollback RTO: under 2 minutes (no transfer overhead) RPO: back to the pre-update dump scope: code, plugins and database schema Target: Full Bare-Metal Recovery RTO: 15-30 minutes (cloud download ingress) RPO: <= 24 hours (last nightly snapshot) scope: full server rebuild & dataroot sync The rollback path trades off scope for speed: it assumes the host OS and dataroot remain intact. Full disaster recovery rebuilds state from cold storage, provisioning schema and blobs from scratch.

A backup system that cannot perform rapid, automated rollbacks introduces catastrophic operational risk. The infrastructure implements two distinct restoration pipelines:

  • Pre-Update State Rollbacks (restore_moodle.yml): Before applying any plugin or core update, update_moodle.yml records the active Git commit SHAs of core and all plugins, dumps the pre-upgrade PostgreSQL schema, and writes a metadata snapshot to pre_update_state.yml. If an update fails, running restore_moodle.yml reads this file, checks out the exact recorded Git SHAs, drops the database, restores the pre-upgrade SQL dump, and clears caches. Target RTO is under 2 minutes. Anything written after the pre-update dump is lost on rollback, so the data-loss window is the time since that dump.
  • Bare-Metal Disaster Recovery (disaster_recovery.yml): Rebuilds a host from scratch by scanning local archives and remote Azure Blob Storage containers via rclone. The operator is prompted for a target snapshot ID; the playbook then terminates active database connections, drops and recreates the database with template0 / en_US.UTF-8 collations, unpacks the dataroot archive with 0770 permissions, and rebuilds caches.
  • Defensive Backup Runner: The daily backup script (moodle-backup.sh) runs a pre-flight disk check that aborts execution if partition usage exceeds 80%, preventing root volume exhaustion. It wraps execution in trap cleanup EXIT bound to disabling Moodle maintenance mode, ensuring the site is never left locked if a dump process encounters a transient error.

Key Configuration Invariants (Tuning Matrix)

Nginx Subsystem

  • fastcgi_read_timeout: Default: 60s → Hardened: 300s
    Purpose: Prevents 504 timeouts during large grading exports and SCORM backups.
  • client_max_body_size: Default: 1M → Hardened: 128M
    Purpose: Permits SCORM packages and course video uploads.
  • server_tokens: Default: on → Hardened: off
    Purpose: Obfuscates web server version in HTTP response headers.

PHP-FPM Runtime & Pool

  • pm.max_children: Default: 5 → Hardened: 20
    Purpose: Scales concurrent worker capacity.
  • pm.max_requests: Default: 0 (Unlimited) → Hardened: 500
    Purpose: Recycles worker memory to mitigate plugin leaks.
  • request_slowlog_timeout: Default: Disabled → Hardened: 10s
    Purpose: Isolates slow database queries to dedicated logs.

PHP Core Directives

  • memory_limit (FPM / CLI): Default: 128M → Hardened: 512M / 1024M
    Purpose: Accommodates heavy quiz evaluation matrices and CLI migrations.
  • max_input_vars: Default: 1000 → Hardened: 5000
    Purpose: Prevents silent truncation of complex gradebook matrices.
  • upload_max_filesize: Default: 2M → Hardened: 100M
    Purpose: Aligns file upload threshold with institutional requirements.
  • opcache.max_accelerated_files: Default: 10000 → Hardened: 16229
    Purpose: Caches all ~12,000 Moodle 5.x PHP files in RAM without thrashing.

PostgreSQL & Redis Persistence

  • PostgreSQL Collation: Default: OS Default (C) → Hardened: en_US.UTF-8 on template0
    Purpose: Prevents database corruption on multilingual text sorting.
  • Redis Eviction Policy: Default: noeviction → Hardened: volatile-lru
    Purpose: Under memory pressure, evicts the least recently used keys that have an expiry. Here that means session keys, since cache keys have none; once no session keys are left, Redis rejects new writes.
  • Redis Partitioning: Default: DB 0 (Shared) → Hardened: DB 0 (Cache) / DB 1 (Sessions)
    Purpose: Eliminates session drops during course cache purges.

Moodle Core & Filesystem

  • Session Handler: Default: Database (mdl_sessions) → Hardened: \core\session\redis
    Purpose: Offloads session I/O, stopping database connection spikes.
  • Cron Execution: Default: Web allowed (0) → Hardened: CLI Only (1)
    Purpose: Blocks unauthorized external triggers of cron via web HTTP requests.
  • Dataroot Permissions: Default: 0755 → Hardened: 0770 (www-data:www-data)
    Purpose: Locks student submissions exclusively to the runtime process user.
← Back