# HostFluid Production Server — Complete Context

**Last updated:** April 4, 2026
**Purpose:** Master context document for Claude — covers the full production server environment, migration history, security stack, DNS configuration, and all hosted applications.

---

## Server Overview

- **Provider:** Hetzner (dedicated server, EX63)
- **Server #:** 2918925
- **IP:** 77.42.1.80
- **OS:** AlmaLinux 9.7
- **Panel:** cPanel/WHM 132.0.23
- **Hostname:** server.hostfluid.com
- **SSH Port:** 2222 (port 22 closed)
- **SSH Auth:** Key-only (ed25519), password auth disabled
- **SSH Key Location (local):** `C:\Users\User\.ssh\id_ed25519`
- **SSH Client:** Termius (desktop + mobile, free Starter plan)
- **CPU:** Intel Core Ultra 7 265 (20 cores/20 threads)
- **RAM:** 64GB DDR5
- **Storage:** 2× 1TB NVMe (RAID1) — ~905GB usable
- **Disk Usage:** ~422GB used (50%)
- **Location:** Helsinki, Finland
- **Hetzner Client #:** K0118929925

---

## Owner / Operator

Marcus owns and operates **HostFluid**, a web hosting business managing 70+ client domains on this server. He also develops **Geophii**, an AI-powered SEO audit application hosted as one of the cPanel accounts.

- **Primary email:** edwardhadome@gmail.com
- **Admin alerts:** admin@hostfluid.com
- **LFD alerts redirected to:** admin@hostfluid.com

### Key Contacts

- **Daisy Chua** — internal, forwards client issues
- **Anwar Hussein** — client (a.hussein@bv-investment.com, bv-investment.com)

---

### Hostwinds → Hetzner (Feb 2026)

- **Source server:** Hostwinds, IP 23.254.198.242, SSH port 2222
- **Method:** WHM Transfer Tool (Remote Root Account Transfer)
- **Total accounts migrated:** ~188 zones / 70+ client domains
- **Migration completed in batches** — all reported "Success" with standard non-critical warnings (PHP version mismatches, DNS non-authoritative)
- **Old Hostwinds server:** May still be running as fallback — cancel only after full verification

### ConvertPods / BrightStageAI Migration (Mar 2026)

- **Source server:** Bluehost, IP 23.238.34.38
- **Account:** convertpods (cPanel user: convertpods)
- **Domains include:** convertpods.com, convertpods.net, brightstageai.com (and subdomains: app., ai., info., link., notify., discover., help., podprime., results.versiii.com.convertpods.com)
- **Size:** ~51GB (convertpods.com) + 15MB (convertpods.net)
- **Status:** Transferred successfully, SSL pending (AutoSSL DCV failed — domains don't point to 77.42.1.80 yet)
- **Action needed:** Switch nameservers at registrar → ns1/ns2.hostfluid.com, then run `autossl_check --user=convertpods`

---

## DNS Configuration

### Nameservers

- **ns1.hostfluid.com** → 77.42.1.80
- **ns2.hostfluid.com** → 77.42.1.80
- **Glue records:** Registered at Namecheap under hostfluid.com
- **DNS software:** PowerDNS (via cPanel)
- **All 188 zones** have both NS records configured

### DNS Cutover Status

- DNS zones on the new server already point to 77.42.1.80
- Old server also had zones updated to point to 77.42.1.80 (so traffic already flows to new server for domains whose DNS was updated on old server)
- **To complete migration:** Change nameservers at each domain's registrar to ns1/ns2.hostfluid.com
- **hostfluid.com:** Still on Namecheap's default DNS (dns1/dns2.registrar-servers.com) — switch when ready
- **Recommendation:** Switch in batches of 5-10 domains, test, then continue

### Email DNS Records (configured for all domains)

- **SPF:** `v=spf1 +a +mx +ip4:77.42.1.80 ~all` (some still include old IP 192.129.178.42 or 23.254.198.242 — remove after Hostwinds cancellation)
- **DKIM:** Auto-generated by cPanel for all domains
- **DMARC:** `v=DMARC1; p=none; rua=mailto:edwardhadome@gmail.com` — mass-added to all domains

---

## Email Configuration

- **MTA:** Exim (cPanel-managed)
- **IMAP/POP:** Dovecot
- **Ports 25 and 465:** Unblocked by Hetzner (ticket approved)
- **Reverse DNS:** 77.42.1.80 → server.hostfluid.com (set in Hetzner Robot)
- **SMTP restrictions:** Outbound SMTP via Exim only, "nobody" user cannot send
- **Email confirmed working:** Delivered to Gmail with DKIM + TLS 1.3 (verified in exim_mainlog)
- **SpamAssassin:** Running
- **Greylisting:** Enabled

### Email Deliverability Issues (resolved)

- Server IP was listed on **RATS-NoPtr blacklist** ("Worst Offender") due to:
  - Legacy Python queue worker flooding mail queue (Mar 2026)
  - Departed client domain `surevitality.co.uk` generating SPF/DKIM auth failures
- **Remediation:** PTR confirmed correct, surevitality.co.uk removed from `/etc/localdomains` and `/etc/userdomains`, manual delisting email sent to SpamRATS
- **Client issue:** Anwar Hussein (a.hussein@bv-investment.com) had "535 Incorrect authentication data" from Gmail "Send mail as" — fix: reset mailbox password in cPanel, update stored SMTP credentials in Gmail

### Planned: Dedicated Email IP

- Order an additional IP from Hetzner (~€1.70/mo) dedicated to outbound email
- Benefits: separates web/email reputation, prevents blacklist cross-contamination
- Setup steps: add IP to server, set as Exim outgoing interface, set rDNS to mail.hostfluid.com, add to SPF records

---

## Security Stack — ALL OPERATIONAL

### Firewall & Intrusion Detection

| Tool | Status | Purpose |
|------|--------|---------|
| CrowdSec | ✅ Active | Community IP reputation, auto-banning |
| CrowdSec Firewall Bouncer | ✅ Active | iptables enforcement |
| Fail2Ban | ✅ Active | Brute force protection (SSH, Dovecot) |
| cPHulk | ✅ Active | cPanel/WHM/Webmail brute force protection |
| ModSecurity + OWASP CRS 3.0 | ✅ Active | Web application firewall ("Process the rules") |

### Malware & Rootkit Detection

| Tool | Status | Purpose |
|------|--------|---------|
| ClamAV + Freshclam | ✅ Active | Virus scanning, auto-updating signatures |
| Maldet (LMD) | ✅ Active | Real-time inotify monitoring on /home, auto-quarantine |
| RKHunter | ✅ Active | Daily rootkit scans (cron at 4am), email reports |
| ImunifyAV | ✅ Active | cPanel malware scanner (free version) |

### Server Hardening

| Setting | Status |
|---------|--------|
| SSH | Key-only auth, port 2222, MaxAuthTries=3, LoginGraceTime=30s |
| Root login | Via key only (PermitRootLogin without-password) |
| MySQL | Bound to localhost only (bind-address=127.0.0.1) |
| KernelCare | Installed, live kernel patching |
| Shell access | Disabled (noshell) for ALL client cPanel accounts |
| PHP dangerous functions | Disabled: exec, passthru, shell_exec, system, proc_open, popen, curl_multi_exec, parse_ini_file, show_source |
| Symlink protection | Enabled |
| Jail Apache (mod_ruid2) | ✅ On |
| JailManager (TailWatch) | ✅ Enabled |

### Security Config Files

| File | Purpose |
|------|---------|
| /etc/crowdsec/config.yaml | CrowdSec main config |
| /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml | Bouncer config |
| /etc/fail2ban/jail.local | Fail2Ban jails (Hostwinds IP whitelisted) |
| /usr/local/maldetect/conf.maldet | Maldet config |
| /var/log/rkhunter/rkhunter.log | RKHunter scan log |
| /etc/cron.d/rkhunter-daily | RKHunter daily cron |
| /etc/rkhunter.conf | RKHunter config (ALLOW_SSH_ROOT_USER=yes) |
| /var/log/exim_mainlog | Exim mail log |

### Whitelisted IPs

- Hostwinds: 23.254.198.242 (in Fail2Ban + cPHulk)
- cPanel infrastructure IPs: 68.66.x.x, 184.94.x.x auto-allowed

### RKHunter Known False Positives

These are cPanel-modified files, not malicious — baseline updated with `--propupd`:
- /usr/bin/mount (permissions)
- /usr/bin/newgrp (permissions)
- /usr/bin/su (permissions + group)
- Client shell changes to noshell (intentional)
- If they reappear: `echo "PKGMGR_NO_VRFY=/usr/bin/mount:/usr/bin/newgrp:/usr/bin/su" >> /etc/rkhunter.conf && rkhunter --propupd`

---

## Monitoring & Alerts

### Active Monitoring

| Monitor | Details |
|---------|---------|
| RKHunter | Daily email report to root |
| CrowdSec | Auto-banning, alerts via `cscli alerts list` |
| Maldet | Real-time inotify on /home |
| Disk alert cron | Every 6 hours, emails if >80% usage (`/etc/cron.d/disk-alert`) |
| WHM Service Manager | All services monitored, auto-restart on failure |
| WHM Contact Manager | Alerts to edwardhadome@gmail.com |

### Recommended (External)

- **UptimeRobot** (free) — HTTP/port monitoring for main client sites
- **Hetzner Robot Monitoring** — Server offline alerts

### Useful Diagnostic Commands

```bash
# Check recent logins
last -20

# Check shell users (should only show root)
cat /etc/passwd | grep "bash"

# CrowdSec status
cscli alerts list --limit 10
cscli decisions list

# Recently modified PHP files (malware check)
find /home -name "*.php" -mtime -3 -newer /etc/passwd | head -20

# Full malware scan
maldet -a /home

# Check open connections
ss -tulnp | grep LISTEN

# Service status
systemctl status fail2ban crowdsec clamav-freshclam --no-pager
```

---

## PHP Configuration

- **Default PHP:** ea-php82 (PHP 8.2)
- **Available versions:** ea-php81, ea-php82, ea-php83
- **Handler:** PHP-FPM for ALL versions (per-user isolation)
- **PHP-FPM migration completed** — root cause during migration was FPM services not enabled and php.conf requiring direct edits

### Post-Migration PHP Fix (if sites show 503)

```bash
/scripts/rebuildhttpdconf
/scripts/php_fpm_config --rebuild
systemctl restart ea-php81-php-fpm ea-php82-php-fpm ea-php83-php-fpm
systemctl restart httpd
```

---

## Backups

- **JetBackup 5:** Installed, sending backups to Google Drive
- **Recommended:** Add Hetzner 5TB Storage Box (~€12/mo) as primary destination
- **Strategy:** Local (JetBackup on server) + nearby offsite (Storage Box) + far offsite (Google Drive)

---

## Web Server

- **Apache:** 2.4.66 (cPanel EasyApache)
- **Modules:** mod_proxy_fcgi, mod_security2, mod_bwlimited, mod_ruid2
- **LiteSpeed:** Not installed (lsws restart command may fail — normal)

---

## Geophii Application

- **cPanel account:** geophii
- **Web root:** /home/geophii/public_html/geophii/
- **App URL:** https://app.geophii.com
- **Main site:** https://geophii.com
- **Git repo:** github.com/bigchain/geophiistats (dev branch)
- **Database:** geophii_myai (user: geophii_flaviumllc)
- **Python venv:** /home/geophii/public_html/geophii/venv/
- **Queue worker cron (geophii user, NOT root):**
  ```
  * * * * * cd /home/geophii/public_html/geophii && /usr/local/cpanel/3rdparty/bin/php scripts/queue_worker.php >> storage/logs/queue_worker.log 2>&1
  ```
- **CRITICAL:** The `cd` is required — queue_worker.php uses dirname(__DIR__) for BASE_PATH
- **CRITICAL:** Never run python_queue_worker.py via cron — caused 15,000+ zombie processes and MySQL exhaustion (Mar 5, 2026 incident)

### Security Headers (deployed Apr 2026)

CSP headers added to `.htaccess` — includes:
- script-src: cdn.tailwindcss.com, cdn.jsdelivr.net, 'unsafe-inline'
- style-src: fonts.googleapis.com, 'unsafe-inline'
- font-src: fonts.gstatic.com
- connect-src: 'self' (all fetch calls are same-origin)
- HSTS, Referrer-Policy, Permissions-Policy

### Git Deployment

```bash
cd /home/geophii/public_html/geophii && git pull origin dev
```
If local changes block the pull:
```bash
git stash && git pull origin dev && git stash drop
```
After deploying, ensure file ownership is correct:
```bash
chown -R geophii:geophii /home/geophii/public_html/geophii/
```
**Note:** Files owned by root prevent cPanel's "Deploy HEAD Commit" from running.

### ImunifyAV False Positives

The Python venv's MySQL connector bundles `libcom_err.so` files that ImunifyAV flags as malicious. These are legitimate OpenSSL/Kerberos libraries. Whitelisted with:
```bash
imunify360-agent malware ignore add /home/geophii/public_html/geophii/venv/lib/python3.9/site-packages/mysql/vendor/private/libcom_err.so.3 /home/geophii/public_html/geophii/venv/lib/python3.9/site-packages/mysql/vendor/private/libcom_err.so.3.0
```

### Pending Geophii Code Fixes

- AdminController.php:563 — SQL parameter mismatch crashing /admin/queue
- Dashboard "Unknown column 'r.client_id'" error

---

## ConvertPods / BrightStageAI

- **cPanel account:** convertpods
- **Source server:** Bluehost (23.238.34.38)
- **Main domains:** convertpods.com, convertpods.net, brightstageai.com
- **Subdomains:** app.convertpods.com, discover.convertpods.com, help.convertpods.com, podprime.convertpods.com, app.brightstageai.com, ai.brightstageai.com, info.brightstageai.com, link.brightstageai.com, notify.brightstageai.com
- **PHP version:** ea-php82
- **Size:** ~51GB
- **SSL Status:** Pending — AutoSSL DCV failed because domains don't point to server yet
- **Action needed:** Switch nameservers at registrar, then run autossl_check

---

## Fund Collective (fundcollective.co)

- **SaaS application** hosted on the same server
- **Database:** MySQL (not PostgreSQL — decision made to stay with MySQL given existing cPanel/phpMyAdmin/schema)
- **Known code bugs:**
  - `AdminController.php:563` — SQL parameter mismatch crashing `/admin/queue`
  - Dashboard `Unknown column 'r.client_id'` error
- **ModSecurity fix:** `fc_journey_path` cookie was triggering rule 942100 (SQL injection false positive) due to raw JSON with double quotes — fixed by base64-encoding values in `fc-tracker.js`

---

1. **cPanel PHP-FPM won't appear** until ea-phpXX-php-fpm service is both installed AND enabled; php.conf may need direct sed edits
2. **Queue workers must run under the correct cPanel user**, not root — running as root caused zombie process incident
3. **MailPoet uses self-referential HTTP requests** — during migrations, if DNS doesn't resolve to the new server, these time out silently. Fix: add domain to /etc/hosts temporarily
4. **CrowdSec whitelist.yaml corruption** — duplicate IP entries cause service failure; delete the file and restart
5. **After lockout by CrowdSec/Fail2Ban** — recover via Hetzner Rescue System: activate rescue, hardware reset, mount RAID arrays (/dev/md3 = root, /dev/md2 = /boot), chroot, clear bans
6. **RAID layout:** md0 (EFI), md2 (/boot), md3 (root) — across nvme0n1/nvme1n1
7. **SecRuleUpdateTargetById in .htaccess or modsec2.user.conf** — CRASHES APACHE (all sites go down). The directive requires rules to be loaded at parse time. Fix false positives at code level instead.
8. **ModSecurity rule 942100** (libinjection SQL injection) is triggered by cookies containing raw JSON with double quotes. Fix by base64-encoding cookie values (e.g., Fund Collective's fc_journey_path cookie in fc-tracker.js).
9. **cPanel Git "Deploy HEAD Commit"** requires: valid `.cpanel.yml` with rsync task, clean working tree (use `.gitignore`), correct file ownership (not root).
10. **ImunifyAV false positives** on Python venv .so files — whitelist with `imunify360-agent malware ignore add <paths>`

---

## Firewall Ports Open (TCP)

```
20, 21, 25, 2222, 53, 80, 110, 143, 443, 465, 587, 853, 993, 995,
2077, 2078, 2079, 2080, 2082, 2083, 2086, 2087, 2095, 2096, 8443
```

---

## Pending Action Items

1. ⬜ Order additional IP for email (~€1.70/mo from Hetzner)
2. ⬜ Order Hetzner Storage Box 5TB (~€12/mo) for backups
3. ⬜ Switch DNS for all domains → ns1/ns2.hostfluid.com at registrars
4. ⬜ Switch convertpods.com + brightstageai.com nameservers → run AutoSSL
5. ⬜ Remove old server IPs from SPF records after Hostwinds cancellation
6. ⬜ Monitor 48 hours after DNS cutover
7. ⬜ Cancel Hostwinds subscription
8. ⬜ Sign up for UptimeRobot (free external monitoring)
9. ⬜ Fix Geophii AdminController.php and Dashboard bugs

---

## How to Migrate a New Account to This Server

### From another cPanel server:

1. WHM → **Transfers → Transfer Tool**
2. Enter remote server IP, root password, SSH port
3. Select account(s), enable Live Transfer + Update DNS Zone
4. Click Copy — wait for completion
5. Post-migration:
   ```bash
   # Add DMARC
   whmapi1 addzonerecord domain=DOMAIN.COM name=_dmarc.DOMAIN.COM. type=TXT txtdata="v=DMARC1; p=none; rua=mailto:edwardhadome@gmail.com"

   # Add ns2
   whmapi1 addzonerecord domain=DOMAIN.COM name=DOMAIN.COM. type=NS nsdname=ns2.hostfluid.com.

   # Rebuild PHP-FPM if 503 errors
   /scripts/php_fpm_config --rebuild
   systemctl restart ea-php82-php-fpm

   # Run AutoSSL (domain must point to 77.42.1.80 first)
   /usr/local/cpanel/bin/autossl_check --user=USERNAME
   ```
6. Disable shell: WHM → Manage Shell Access → set to noshell
7. Switch nameservers at registrar → ns1/ns2.hostfluid.com

---

## Preferences

- Prefers commands as **copy-paste blocks** rather than manual editor steps
- Prefers options presented as **one-by-one or bundled** command sets
- Values thorough testing and fallback availability before committing
- Kept old server live during entire migration as fallback
