255 lines
8.6 KiB
ReStructuredText
255 lines
8.6 KiB
ReStructuredText
================================
|
||
PyraLogs - A Centralized Logging Hub
|
||
================================
|
||
|
||
**PyraLogs** is a **public domain** application built on the `Pyramid <https://trypyramid.com>`_ framework. It offers a flexible, agent-friendly solution for centralized logging, allowing environments and agents to log events to a central hub. PyraLogs supports a multi-database design where each namespace maintains its own SQLite database for logs. The main database (``main.db``) stores system- and user-level data, while each namespace’s personal DB file handles their specific logs.
|
||
|
||
Key Features
|
||
============
|
||
|
||
- **Passwordless Login** using email verification codes.
|
||
- **Namespace-Based Logging** with per-namespace SQLite databases.
|
||
- **JWT Authentication for Agents**, enabling secure log submissions.
|
||
- **Token Versioning** for per-agent token revocation without affecting others.
|
||
- **Public/Private** namespace visibility controls.
|
||
- **Admin Tools** for managing namespaces, agents, and user access.
|
||
- **Agent-Friendly** architecture: easily scriptable endpoints for logging and management.
|
||
- **Configurable** via environment variables (including secret keys).
|
||
|
||
Git Repository
|
||
==============
|
||
|
||
The project is maintained at:
|
||
|
||
- `PyraLogs Git Repo <https://git.unturf.com/engineering/unturf/logs.unturf.com>`_
|
||
|
||
Since this is public domain, you can adapt and redistribute it freely.
|
||
|
||
Configuration via Environment
|
||
=============================
|
||
|
||
PyraLogs fetches settings from environment variables with sensible defaults:
|
||
|
||
- ``PYRALOGS_SECRET``
|
||
The secret key for cookie session signing.
|
||
If missing, PyraLogs automatically generates a **random 64-character** string at runtime.
|
||
Remove this file or change this value to log our all cookie sessions for all namespaces.
|
||
|
||
- ``PYRALOGS_JWT_SECRET``
|
||
The secret key for signing JSON Web Tokens (JWTs) for agents without a mailbox.
|
||
If missing, PyraLogs automatically generates a **random 64-character** string at runtime.
|
||
Remove this file or change this value to log out all JWT agents for all namespaces.
|
||
|
||
- ``PYRALOGS_DB_URL``
|
||
Connection string for the main database. Default: ``sqlite:///main.db``.
|
||
|
||
- ``PYRALOGS_HOST`` and ``PYRALOGS_PORT``
|
||
The host and port to serve on. Defaults: ``0.0.0.0`` (host), ``6544`` (port).
|
||
|
||
- ``PYRALOGS_SMTP_HOST`` and ``PYRALOGS_SMTP_PORT``
|
||
SMTP server details. Defaults: ``localhost:25``.
|
||
|
||
- Any other environment variables you wish to incorporate can be accessed in the code.
|
||
|
||
Local Setup
|
||
===========
|
||
|
||
1. **Clone the Project**
|
||
|
||
.. code-block:: bash
|
||
|
||
git clone https://git.unturf.com/engineering/unturf/logs.unturf.com.git
|
||
cd logs.unturf.com
|
||
|
||
2. **Create a Virtual Environment**
|
||
|
||
.. code-block:: bash
|
||
|
||
python -m venv venv
|
||
source venv/bin/activate # Linux/Mac
|
||
# or
|
||
venv\Scripts\activate.bat # Windows
|
||
|
||
3. **Install Dependencies**
|
||
|
||
.. code-block:: bash
|
||
|
||
pip install -r requirements.txt
|
||
|
||
4. **Initialize the Database**
|
||
|
||
Create the main database before running the application.
|
||
|
||
.. code-block:: bash
|
||
|
||
python initialize_db.py
|
||
|
||
5. **Run**
|
||
|
||
.. code-block:: bash
|
||
|
||
# Optionally set PYRALOGS_SECRET if you want a custom secret.
|
||
export PYRALOGS_SECRET="YOUR_OWN_LONG_RANDOM_STRING"
|
||
python app.py
|
||
|
||
If ``PYRALOGS_SECRET`` is **not** set, the app automatically generates a 64-character secret at runtime.
|
||
|
||
6. **Access**
|
||
|
||
Point your browser to `http://localhost:6544` or `http://<HOST>:<PORT>` according to your environment variables.
|
||
|
||
Example: Agent Workflow Script
|
||
==============================
|
||
|
||
Below is a sample Bash script showing how an **agent** might:
|
||
|
||
1. Obtain a JWT for logging.
|
||
2. Send logs to the PyraLogs server using the JWT.
|
||
|
||
.. code-block:: bash
|
||
|
||
#!/usr/bin/env bash
|
||
#
|
||
# Usage:
|
||
# ./test_agent.sh "Your log message"
|
||
#
|
||
# Description:
|
||
# Send a log message to the PyraLogs server using a JWT.
|
||
#
|
||
# Notes:
|
||
# - Replace <YOUR_NAMESPACE_JWT> with your actual JWT token.
|
||
# - Replace <NAMESPACE_SHORT_ID> with your namespace's short ID.
|
||
# - Adjust BASE_URL if the server is running elsewhere.
|
||
|
||
BASE_URL="${BASE_URL:-http://localhost:6544}"
|
||
JWT_TOKEN="<YOUR_NAMESPACE_JWT>"
|
||
LOG_MESSAGE="$1"
|
||
|
||
if [[ -z "$LOG_MESSAGE" ]]; then
|
||
echo "Usage: $0 \"Your log message\""
|
||
exit 1
|
||
fi
|
||
|
||
echo "Sending log message..."
|
||
RESPONSE=$(curl -s -X POST "$BASE_URL/webhook" \
|
||
-H "Content-Type: application/json" \
|
||
-H "Authorization: Bearer $JWT_TOKEN" \
|
||
-d "{
|
||
\"message\": \"$LOG_MESSAGE\",
|
||
\"level\": \"INFO\",
|
||
\"metadata\": {\"w\": 1}
|
||
}")
|
||
|
||
echo "Server Response: $RESPONSE"
|
||
|
||
Dockerfile with Caddy + uWSGI
|
||
=============================
|
||
|
||
Below is an example Dockerfile that:
|
||
|
||
- Uses **uWSGI** to run PyraLogs.
|
||
- Uses **Caddy** as a reverse proxy (and optionally HTTPS if configured).
|
||
- Defines a **volume** for databases (so they are stored on the host).
|
||
|
||
.. code-block:: dockerfile
|
||
|
||
###########################################################
|
||
# Stage 1: Build the Python environment
|
||
###########################################################
|
||
FROM python:3.9-slim AS builder
|
||
|
||
WORKDIR /app
|
||
COPY . /app
|
||
|
||
# Install Python dependencies
|
||
RUN pip install --no-cache-dir -r requirements.txt
|
||
|
||
###########################################################
|
||
# Stage 2: Final image with Caddy + uWSGI + Python
|
||
###########################################################
|
||
FROM caddy:2-alpine
|
||
|
||
# Install Python, pip, and uWSGI from Alpine
|
||
RUN apk add --no-cache python3 py3-pip uwsgi-python3
|
||
|
||
WORKDIR /app
|
||
|
||
# Copy the app from the builder
|
||
COPY --from=builder /app /app
|
||
|
||
# Copy Caddyfile for the reverse proxy
|
||
# (Assuming you have a Caddyfile in your repo root)
|
||
COPY Caddyfile /etc/caddy/Caddyfile
|
||
|
||
# Copy uWSGI config if desired
|
||
# For example (assume you created uwsgi.ini in your repo):
|
||
# [uwsgi]
|
||
# module = app:app
|
||
# master = true
|
||
# processes = 4
|
||
# socket = 127.0.0.1:8080
|
||
# vacuum = true
|
||
# die-on-term = true
|
||
COPY uwsgi.ini /app/uwsgi.ini
|
||
|
||
# Environment variables (optional overrides).
|
||
ENV PYRALOGS_DB_URL="sqlite:///data/main.db"
|
||
ENV PYRALOGS_HOST="0.0.0.0"
|
||
ENV PYRALOGS_PORT="6544"
|
||
|
||
# Expose HTTP and HTTPS
|
||
EXPOSE 80
|
||
EXPOSE 443
|
||
|
||
# Define volume so host can persist databases outside the container
|
||
# We'll store database files in /data
|
||
VOLUME ["/data"]
|
||
|
||
# Command starts uWSGI and then starts Caddy
|
||
CMD ["/bin/sh", "-c", "\
|
||
uwsgi --ini /app/uwsgi.ini & \
|
||
caddy run --config /etc/caddy/Caddyfile \
|
||
"]
|
||
|
||
.. note::
|
||
|
||
- We use ``/data`` as the volume. By default, the environment variable ``PYRALOGS_DB_URL`` is set to ``sqlite:///data/main.db``, so the main DB (and any namespace DB files) go inside ``/data``.
|
||
- For namespace DB files, your app can also interpret an environment variable (like ``PYRALOGS_DB_BASEPATH=/data``) if you want to make that path configurable in the code.
|
||
- Make sure to **mount** a volume at ``/data`` when you run the container, e.g.:
|
||
|
||
.. code-block:: bash
|
||
|
||
docker run -d \
|
||
-p 80:80 -p 443:443 \
|
||
-v /my/local/dbfolder:/data \
|
||
--name pyralogs \
|
||
pyralogs-image:latest
|
||
|
||
- With that, any namespace database is stored in ``/my/local/dbfolder`` on the host.
|
||
|
||
Tips
|
||
====
|
||
|
||
- For **production**, you likely want to set up a real SMTP server or third-party service (e.g., Mailgun, Improvmx) and configure it via environment variables (``PYRALOGS_SMTP_HOST``, etc.).
|
||
- Ensure you mount a volume for persistent SQLite files if you want to avoid data loss when containers are removed or replaced.
|
||
- Because the project is **public domain**, you can adapt it without restriction, removing features, adding custom logic, etc.
|
||
- **Security Reminder**: Always keep your ``PYRALOGS_SECRET`` and ``PYRALOGS_JWT_SECRET`` secure. Do not expose them in your code repositories or logs.
|
||
|
||
License & Public Domain
|
||
=========================
|
||
|
||
This project is in the public domain. You are free to use, adapt, and redistribute it without attribution or additional licensing.
|
||
|
||
If you find PyraLogs helpful, feel free to contribute back or share your enhancements!
|
||
|
||
Support and Contact
|
||
===================
|
||
|
||
If you register a gitlab account, let me know, and I will grant you developer access to contribute.
|
||
|
||
- **Issues**: Please open tickets at the `PyraLogs project page <https://git.unturf.com/engineering/unturf/logs.unturf.com>`_.
|
||
- For general inquiries, you can reach out to the maintainers directly.
|
||
|
||
We hope PyraLogs helps you set up a centralized logging solution quickly and efficiently!
|
||
|
||
Enjoy and happy logging!
|