README

License: GNU GPL, Version 2

Repository: https://gitlab.sgalinski.de/typo3/sg_logs

Please report bugs here: https://gitlab.sgalinski.de/typo3/sg_logs

About

This extension provides a TYPO3 backend module to browse and manage log files, typically located in var/log/. It is specifically designed to help administrators identify recurring issues by grouping similar log entries and providing a clear overview of the system's health.

Features

• Log File Overview: List all log files in the configured directory with size and modification date.
• Intelligent Grouping: Similar log entries (e.g., the same exception but different error codes or URLs) are grouped into a single row to reduce clutter.
• Variations View: Drill down into a grouped entry to see all individual occurrences and their details.
• Solved Status: Mark log entries as "Solved" to keep track of what has been addressed. The state is persisted in the database.
• Advanced Filtering: Filter entries by "Solved/Unsolved" status or use the search field with support for exclusion (e.g., !Status 200).
• File Management: Delete log files directly from the backend.
• Real-Time Plain Log View: Open a start/stop watcher per file and stream newly appended lines live (without parsing).
• Optimized for Performance: Efficiently reads even large log files (up to 100MB buffer).
• Integrated Pagination: Handles large volumes of log entries with ease.
• Prometheus Metrics Exporter: Optional /metrics endpoint that exports incremental TYPO3 error counters.
• Backend Metrics Preview: View metrics output directly in the backend module.

Installation

1. Install the extension via Composer:

   composer require sgalinski/sg-logs
   

2. Activate the extension in the TYPO3 Extension Manager or via CLI.
3. The module "/var/log Viewer" will appear in the "System" section of your TYPO3 Backend.

Configuration

Extension Configuration

You can configure the path to your log files in the Extension Configuration (Settings → Extension Configuration → sg_logs):

• Log File Path: Defaults to /var/log/. Ensure this path is relative to your project root.

Metrics Exporter Configuration

The metrics exporter is disabled by default. Configure it in Settings → Extension Configuration → sg_logs:

• metrics.enabled: Enable the exporter (required).
• metrics.path: HTTP path (default /sg-logs/metrics/).
• metrics.errors_path: Optional debug endpoint (default /sg-logs/errors/top/).
• metrics.auth.token: Required bearer token for authentication.
• metrics.auth.ip_allowlist: Optional CIDR allowlist (comma-separated).
• metrics.trust_proxy / metrics.trusted_proxies: Use X-Forwarded-For only for trusted proxies.
• labels.instance_id: Optional instance identifier (defaults to TYPO3 sitename, then global).
• labels.env: Environment label (default prod).
• logs.pattern: Log discovery pattern (defaults to typo3_*.log). Log directory uses the Log File Path setting.
• threshold.min_level: Minimum severity to count (default warning).
• export.top_fingerprints / export.top_exception_classes: Top-N exports.
• limits.max_new_bytes_per_scrape / limits.max_new_lines_per_scrape / limits.initial_tail_lines: Safety caps.

Example configuration values:

metrics.enabled = 1
metrics.path = /sg-logs/metrics/
metrics.auth.token = change-me
labels.instance_id = customer-prod-1
labels.env = prod
logs.dir = /var/log/
logs.pattern = typo3_*.log

TypoScript Configuration

The backend module uses TypoScript for its view configuration. You can override the template, partial, or layout paths if needed:

module.tx_sglogs {
    view {
        templateRootPaths {
            10 = EXT:your_extension/Resources/Private/Templates/
        }
    }
}

Usage

Viewing Log Entries

Click on a log file in the file list to view its contents. The entries are automatically parsed and grouped.

Real-Time Plain Log Viewer

In the log file list, click "Real-Time" for a file to open the plain-text live view.
Use Start to begin polling and Stop to pause it. Clear empties only the viewer output and does not touch the file.

Searching & Filtering

• Use the search field to find specific messages.
• Reverse Search: Prepend an exclamation mark ! to exclude terms (e.g., !PHP Warning).
• Solved Filter: By default, the list shows "Not Solved" entries. You can switch to "Solved" or "All" using the dropdown menu.

Marking as Solved

Click the "Not Solved" button on an entry to mark it as "Solved". This state is stored in the tx_sglogs_domain_model_solvedstatus table.

Deleting Files

In the file list view, you can delete old or large log files using the "Delete" button. A confirmation modal will appear.

Backend Metrics Preview

The log module shows a metrics info panel with a "Preview metrics output" button. This backend-only preview does not require the metrics token and is intended for quick verification. The preview does not update the metrics cursor.

Metrics Exporter Usage

Prometheus Scrape

scrape_configs:
  - job_name: 'typo3_logs'
    scrape_interval: 30s
    metrics_path: /sg-logs/metrics/
    static_configs:
      - targets: ['example.org']
    authorization:
      credentials: change-me

Top Errors Debug Endpoint

GET /sg-logs/errors/top/?window=3600&limit=20

Returns JSON with the most frequent error fingerprints in the time window. Example messages are normalized and never used as Prometheus labels.

Counters represent events observed since the metrics state was initialized. The exporter also exposes debug gauges for the active log (mtime/size/offset) and the last seen line timestamp to simplify troubleshooting.

Instance identifiers are normalized to a slug for metric labels. The display name is exported via sg_typo3_instance_info{name="..."} 1.

Additional scrape health gauges include sg_typo3_metrics_exporter_scrape_duration_seconds and sg_typo3_metrics_exporter_scrape_timestamp_seconds for external monitoring.

Counters represent events observed since the metrics state was initialized. The exporter also exposes debug gauges for the active log (mtime/size/offset) and the last seen line timestamp to simplify troubleshooting.

Testing

Unit tests for the metrics exporter live in vendor/sgalinski/sg-logs/Tests/Unit. Run them with:

vendor/bin/phpunit -c vendor/sgalinski/sg-logs/phpunit.xml.dist

Extension Architecture

The extension follows a clean and modular architecture:

Controllers

• LogController: The main entry point for the backend module. It handles routing and delegates tasks to services. It uses the ModuleTemplateFactory for rendering the TYPO3 backend interface.

Services

• LogService: Contains the core logic for reading, parsing, and grouping log entries. It also handles the persistence of "Solved" statuses.
• PaginationService: Provides helper methods for calculating pagination-related data.
• LogFilePathService: Centralizes safe log directory/file path resolution and filename sanitization.
• RequestPayloadService: Centralizes request parameter extraction and payload validation for backend actions.
• TextResponseFactory: Creates plain text PSR-7 responses used by controller and middleware code paths.

Normalization Logic

The grouping logic uses a normalization process in LogService::getGroupKey() to identify similar messages. It automatically masks:

• TYPO3 error codes (e.g., Code: 2026011619211738ec151c)
• Request IDs (e.g., Request: ea309b6021bc1)
• URLs (including query parameters)
• File paths
• Hexadecimal addresses and hashes
• Generic numbers (PIDs, line numbers, etc.)

This ensures that the same root cause is always grouped together, regardless of the incident-specific data.

Persistence

The "Solved" status of log entries is stored in a dedicated database table tx_sglogs_domain_model_solvedstatus. Each entry is identified by an SHA-256 hash of the normalized message and the filename.