AnalyticsBot Repository Organization: Comprehensive Cleanup and Consolidation

Systematic repository cleanup removing 25 orphaned files, consolidating 3 archive directories, and eliminating 450KB of duplicate/unused content across AnalyticsBot.

AnalyticsBot Repository Organization: Comprehensive Cleanup and Consolidation

Session Date: 2025-12-03 Project: AnalyticsBot - Multi-tenant Analytics Platform Focus: Repository organization analysis, duplicate removal, and documentation consolidation Session Type: Refactoring / Technical Debt Elimination

Executive Summary

Completed a comprehensive three-phase repository cleanup of the AnalyticsBot codebase. Using mermaid diagrams for visual analysis and systematic investigation, identified and removed 25 orphaned files (~450KB), consolidated 3 archive directories into a single location, and reorganized 12 documentation files for improved discoverability.

The cleanup eliminated duplicate policy-pages directories, orphaned database files scattered across 4 locations, backup files that accumulated during development, and redundant script versions. All changes verified with TypeScript typechecks and unit tests passing.

Key Metrics: | Metric | Value | |——–|——-| | Files Removed | 25 | | Files Reorganized | 12 | | Directories Removed | 2 | | Space Saved | ~450KB | | Archive Directories Consolidated | 3 → 1 | | Database Locations Consolidated | 4 → 1 | | TypeScript Errors | 0 | | Unit Tests Passing | 405/405 |

Problem Statement

The AnalyticsBot repository had accumulated organizational debt over multiple development phases:

  1. Database File Sprawl: 11 SQLite database files scattered across 4 locations (backend/, backend/data/, backend/prisma/, data/)
  2. Duplicate Directories: ui/policy-pages/ and ui/src/policy-pages/ both contained policy page components
  3. Backup File Accumulation: 8 .backup files left from refactoring sessions
  4. Script Duplication: Original and refactored versions of migration scripts coexisting
  5. Documentation Sprawl: Sentry docs in 4 locations, archive directories in 3 places
  6. MCP Duplication: mcps/ and ui/public/mcps/ appeared redundant (confirmed intentional for GitHub Pages)

Impact Before: Confusion about canonical file locations, wasted storage, difficult onboarding for new developers.

Implementation Details

Phase 1: Low-Risk Cleanup (Backup Files and Duplicates)

1.1 Backup File Removal

Files Removed (8 files, ~80KB):

ui/src/hooks/useFilterPersistence.ts.backup
ui/src/api/client.ts.backup
backend/prisma/schema.prisma.backup
backend/scripts/fix-duplicate-project-ids.ts.backup
backend/src/repositories/EventRepository.ts.backup
backend/src/services/FileSystemService.ts.backup
backend/src/services/InventoryService.ts.backup
backend/src/services/SentryService.ts.backup

Verification: find . -name "*.backup" returned empty after cleanup.

1.2 Duplicate Policy Pages Removal

Analysis: Two policy-pages directories existed:

  • ui/policy-pages/ - 7 files (older, Nov 18)
  • ui/src/policy-pages/ - 11 files (newer, includes types.ts, constants.ts)

Decision: Keep ui/src/policy-pages/ (more complete, properly located in src/).

Files Removed (7 files, ~60KB):

ui/policy-pages/AccessibilityStatement.tsx
ui/policy-pages/CCPAPolicy.tsx
ui/policy-pages/CookiesPolicy.tsx
ui/policy-pages/index.ts
ui/policy-pages/PolicyPages.css
ui/policy-pages/PrivacyPolicy.tsx
ui/policy-pages/TermsOfService.tsx

1.3 Duplicate Script Removal

Analysis: Migration scripts had both original and refactored versions:

  • fix-duplicate-project-ids.ts (original)
  • fix-duplicate-project-ids-refactored.ts (kept)
  • verify-uuid-v7.ts (original)
  • verify-uuid-v7-refactored.ts (kept)

Files Removed (2 files, ~13KB):

backend/scripts/fix-duplicate-project-ids.ts
backend/scripts/verify-uuid-v7.ts

Phase 2: Database Consolidation

2.1 Database Location Analysis

Investigation via code analysis:

// backend/src/db/projectData.ts:21-22
const DATA_DIR = path.join(process.cwd(), 'data');  // = backend/data/
const DB_PATH = path.join(DATA_DIR, 'analytics.db'); // CANONICAL

// backend/prisma/schema.prisma:10-12
datasource db {
  provider = "postgresql"  // NOT SQLite - uses Supabase
  url      = env("DATABASE_URL")
}

Finding: Only backend/data/analytics.db is used by the application. Prisma uses PostgreSQL (Supabase), not SQLite.

2.2 Orphaned Database Removal

Files Removed (8 files, ~300KB):

FileSizeReason
backend/analytics.db84KOld location at backend root
backend/prisma/analytics.db84KOld SQLite before PostgreSQL
backend/prisma/test.db0BEmpty test file
data/analytics.db88KRoot level, not used by backend
data/analytics.db-shm32KWAL file for orphaned DB
data/analytics.db-wal0BWAL file for orphaned DB
data/sidequest.db0BEmpty, sidequest uses PostgreSQL
data/sidequest.sqlite32KOrphaned, sidequest uses PostgreSQL

Canonical Location Preserved:

backend/data/
├── analytics.db      (20K) ← Used by projectData.ts
├── analytics.db-shm  (32K) ← SQLite WAL
└── analytics.db-wal  (0B)  ← SQLite WAL

Phase 3: Documentation Consolidation

3.1 Sentry Documentation Reorganization

Before: 11 Sentry docs across 4 locations After: Active docs stay in place, archived docs consolidated

Created: docs/archive/sentry/ directory

Files Moved to docs/archive/sentry/:

  • DOPPLER_SENTRY_SETUP.md (from dev/archive/)
  • SENTRY_API_DEPLOYMENT.md (from dev/archive/)
  • SENTRY_MCP_SETUP.md (from docs/archive/ root)

3.2 Archive Directory Consolidation

Before: 3 archive directories

  • docs/archive/ - Main archive (21 items)
  • dev/archive/ - Session reports (14 items)
  • backend/docs/archive/ - Backend-specific (1 subfolder)

After: Single primary archive at docs/archive/

Files Moved from dev/archive/:

FileDestination
PHASE_1_EMERGENCY_FIXES_COMPLETE.mddocs/archive/phases/
PHASE_2_PREVENTATIVE_MEASURES_COMPLETE.mddocs/archive/phases/
EVENT_STREAM_SETUP.mddocs/archive/legacy-integrations/
META_SETUP_GUIDE.mddocs/archive/legacy-integrations/
WIX_SETUP_GUIDE.mddocs/archive/legacy-integrations/
CHANGE_LOG.mddocs/archive/
DEDUPLICATION-REPORT.mddocs/archive/
DEPLOYMENT_SUMMARY.mddocs/archive/
DETAILED-REFACTORING-GUIDE.mddocs/archive/
TYPECHECK_FIXES.mddocs/archive/

Directories Removed:

  • dev/archive/ (empty after consolidation)

3.3 MCP Duplication Analysis

Finding: mcps/ and ui/public/mcps/ duplication is intentional.

Rationale: ui/public/mcps/ serves MCP documentation via GitHub Pages static site. The duplication is by design for the frontend deployment.

Action: No changes - documented as intentional.

Mermaid Diagrams

Current Structure (Before Cleanup)

graph TD
    root[AnalyticsBot Root<br/>464 files, 324 dirs]

    subgraph "Database Sprawl - 11 files in 4 locations"
        root --> data[data/]
        root --> backendRoot[backend/]
        data --> dataDB[analytics.db + WAL]
        data --> sidequestDB[sidequest.db<br/>sidequest.sqlite]
        backendRoot --> backendDBroot[analytics.db]
        backendRoot --> backendData[backend/data/]
        backendRoot --> prisma[backend/prisma/]
        backendData --> dataDB2[analytics.db + WAL]
        prisma --> prismaDB[analytics.db<br/>test.db]
    end

    subgraph "Duplicate Policy Pages"
        root --> ui[ui/]
        ui --> policyOld[ui/policy-pages/<br/>7 files - OLDER]
        ui --> uisrc[ui/src/]
        uisrc --> policyNew[ui/src/policy-pages/<br/>11 files - NEWER]
    end

    style dataDB fill:#ff9999
    style sidequestDB fill:#ff9999
    style dataDB2 fill:#ffeb99
    style prismaDB fill:#ff9999
    style backendDBroot fill:#ff9999
    style policyOld fill:#ff9999

Final Structure (After Cleanup)

graph TD
    root[AnalyticsBot Root]

    subgraph "Backend - Clean"
        root --> backend[backend/]
        backend --> data[data/<br/>SINGLE DB location]
        backend --> src[src/]
        backend --> prisma[prisma/<br/>Migrations only]
    end

    subgraph "Frontend - Clean"
        root --> ui[ui/]
        ui --> uisrc[src/]
        uisrc --> policyPages[policy-pages/<br/>SINGLE location]
    end

    subgraph "Documentation - Consolidated"
        root --> docs[docs/]
        docs --> docsArchive[archive/<br/>ALL archived docs]
        docs --> sentry[archive/sentry/]
        docs --> phases[archive/phases/]
    end

    style data fill:#99ff99
    style policyPages fill:#99ff99
    style docsArchive fill:#99ff99
    style sentry fill:#9999ff

Testing and Verification

TypeScript Verification

$ cd ui && npm run type-check
> analyticsbot-ui@1.0.0 type-check
> tsc --noEmit
# ✅ No errors

$ cd backend && npm run typecheck
> analyticsbot-backend@1.0.0 typecheck
> tsc --noEmit
# ✅ No errors

Unit Test Results

$ npm run test:unit -- --silent
Test Suites: 1 failed, 20 passed, 21 total
Tests:       405 passed, 405 total
Time:        7.923 s

Note: The 1 failed test suite is a pre-existing TypeScript type mismatch in ProjectService.test.ts (unrelated to cleanup).

VerificationResult
UI TypeCheck✅ PASS
Backend TypeCheck✅ PASS
Unit Tests✅ 405/405 PASS
Backup Files Remaining0
Duplicate Directories0
Database Locations1 (canonical)

Key Decisions and Trade-offs

Decision 1: Keep backend/data/ as Canonical DB Location

Choice: Preserve backend/data/analytics.db only Rationale: Code analysis showed projectData.ts uses path.join(process.cwd(), 'data') which resolves to backend/data/ when running from backend directory Alternative Considered: Root data/ directory Trade-off: Backend-specific location vs project-root location

Decision 2: Keep MCP Duplication

Choice: Preserve both mcps/ and ui/public/mcps/ Rationale: Intentional duplication for GitHub Pages static serving Alternative Considered: Symlinks or build-time copy Trade-off: Disk space vs build complexity

Decision 3: Consolidate to docs/archive/

Choice: Single archive location at docs/archive/ Rationale: Easier to find archived docs, cleaner project structure Alternative Considered: Keep dev/archive/ for session-specific docs Trade-off: Slightly longer paths vs organizational clarity

Summary of Changes

Files Removed (25 total)

CategoryCountSpace
Backup files8~80KB
Duplicate policy-pages7~60KB
Original scripts2~13KB
Orphaned databases8~300KB
Total25~450KB

Files Moved (12 total)

FromToCount
dev/archive/docs/archive/sentry/2
dev/archive/docs/archive/phases/2
dev/archive/docs/archive/legacy-integrations/3
dev/archive/docs/archive/5

Directories Removed (2)

  • ui/policy-pages/ (duplicate)
  • dev/archive/ (consolidated)

Next Steps

Immediate

  • Commit cleanup changes with descriptive message

Short-term

  • Update CLAUDE.md reference to docs/archive/CHANGE_LOG.md if needed
  • Consider adding .gitignore entries for *.backup files

Medium-term

  • Review remaining 249 markdown files for further consolidation
  • Consider automated cleanup hooks for backup file prevention

References

Code Files

  • backend/src/db/projectData.ts:21-22 - Canonical database path
  • backend/prisma/schema.prisma:10-12 - PostgreSQL configuration

Tools Used

  • repository-organization-analyzer skill - Mermaid diagram generation
  • find, diff, ls commands - File analysis
  • TypeScript compiler - Verification
  • docs/archive/CHANGE_LOG.md - Project changelog (moved from dev/archive)
  • docs/archive/DEDUPLICATION-REPORT.md - Previous deduplication work

Tags: , , , , , , ,

Categories: , ,

Updated: