Skip to content
Ljómi Systems
Ljómi

LjómiTechnicalfact sheet

/ˈljouːmɪ/ · Icelandic, “light”

A real-time system that analyses aurora video frame by frame and emits MIDI notes, chords and continuous controller streams derived from auroral colour, brightness structure, area coverage, centroid motion and optical-flow wave features.

Status
Working instrument
Implementation
Python, 24 modules
Core stack
OpenCV · NumPy · rtmidi
Acceleration
OpenCL / CUDA / CPU
Input
Video file or live camera
Output
MIDI 1.0, multi-port
Licence
MIT
Author
Ljomi Systems Ltd

Aurora as dataset and as performer

Ljómi treats each video frame as a field of measurable visual activity rather than an image to be recognised. It segments the frame into auroral colour classes, extracts photometric and kinematic features per class, and maps those features onto pitch, harmony, velocity and continuous control data. Every mapping is declared in YAML, and you can re-tune it mid performance without restarting the process.

The output is MIDI, not synthesised audio. That is deliberate. A symbolic control stream keeps the interpretive step out in the open, where you can read it: the aurora supplies behaviour, contour and constraint, and the mapping layer makes the musical decisions. Nothing here claims the aurora “sounds like” the result.

Per-frame processing pipeline

  1. 01

    Acquire

    File or camera capture, decimated by frame_skip, resized to a fixed processing resolution (default 854×480), converted BGR→HSV.

  2. 02

    Segment

    Per colour class, an HSV threshold produces a binary mask, refined by morphological opening then closing with a 5×5 elliptical kernel.

  3. 03

    Constrain

    Optional interactive ROI rectangle and a freehand painted exclusion mask are ANDed into every colour mask, suppressing foreground light pollution.

  4. 04

    Extract

    Ten features per colour class: intensity, area, mean brightness, brightness regions, Sobel gradients, centroid motion, and four optical-flow wave descriptors.

  5. 05

    Map & emit

    Features are quantised to scale degrees, chord types and CC values, then routed per colour to MIDI channels and devices with explicit note-lifecycle tracking.

Three auroral colour classes, independently voiced

Colour classes correspond to the dominant auroral emission features as they appear in consumer-camera footage: atomic oxygen at 557.7 nm, the oxygen red line near 630.0 nm, and the N₂⁺ bands toward 427.8 nm. The system segments, analyses and routes each class independently, so a colour behaves as its own musical voice. You can enable or disable classes at runtime, and disabling one flushes its held notes.

  • Green

    Ch 1

    557.7 nm · atomic oxygen

    Hue
    45–76
    Sat
    30–255
    Val
    30–255
    Regions
    max 4, ≥0.1%

  • Pink

    Ch 2

    630.0 nm · oxygen red line

    Hue
    140–180
    Sat
    0–255
    Val
    0–255
    Regions
    max 3, ≥0.01%

  • Purple

    Ch 3

    427.8 nm · N₂⁺ bands

    Hue
    115–155
    Sat
    10–255
    Val
    10–255
    Regions
    max 3, ≥0.01%

Hue is in OpenCV’s 0–180 convention (half-degrees). Saturation and value floors are deliberately permissive to retain faint pink and purple structure. A parallel cloud_ranges set substitutes three low-saturation luminance bands, so the same pipeline can sonify daytime cloud without code changes. It doubles as a control condition.

Feature extraction and its methods

The system extracts ten features per colour class, per analysed frame. Six are photometric or kinematic measures of the mask itself; four describe the wave structure of dense optical flow across it.

The ten features, in full

Method, output range, and default MIDI CC assignment for each.Open

  • intensity

    CC 74 · every frame

    Mean HSV value channel within mask, normalised by 255. Range 0–1.

  • area

    CC 75 · every frame

    Non-zero mask pixels ÷ total frame pixels. Range 0–1.

  • mean brightness

    CC 87

    Masked mean of the value channel. Range 0–1.

  • brightness regions

    → notes

    Value threshold → 8-connected components with stats; regions below a minimum area fraction discarded, survivors ranked by brightness × area. Range 0–4 regions.

  • brightness gradient

    CC 85 / 86

    Sobel ∂x and ∂y over the masked value channel, L1 mean per axis, first-order IIR smoothing (α = 0.8). Range 0–1 per axis.

  • centroid motion

    CC 71 / 72

    Image-moment centroid displacement between frames, amplified (×4.6), IIR-smoothed (α = 0.9), clamped and offset to a bipolar CC. Range ±1 → 0–127.

  • wave coherence

    CC 24

    Magnitude of the first circular moment of dense flow angles, |⟨e^iθ⟩|; gates all wave output below 0.05. Range 0–1.

  • wave intensity

    CC 22

    Temporal brightness variation over the frame window: 0.4·mean normalised inter-frame difference + 0.6·√(max local variance), exponentially weighted over 15 samples. Range 0–1.

  • wave direction

    CC 20

    Magnitude-weighted mean flow angle over significantly-moving pixels, rate-limited to 5–40° per step, blended 0.7 / 0.3 with the circular mean of a 5-frame history. Range 0–360°.

  • wave speed

    CC 21

    Flow magnitude weighted by local (20×20) directional coherence, pooled as 0.6·mean + 0.2·max + 0.2·σ, compressed by s/(s+5), momentum 0.7, γ = 1.2. Range 0–1.

Method notes

Optical-flow parameters, temporal windowing, direction convention, and why every feature is smoothed.Open

  • Dense optical flow

    Wave descriptors derive from Farnebäck dense flow computed on the masked value channel, with parameters tuned for the aurora's long, low-contrast structures rather than rigid-body motion: pyr_scale 0.4, levels 5, winsize 25, iterations 3, poly_n 7, poly_sigma 1.5. Flow runs at half the processing resolution. A DIS flow backend is selectable for A/B comparison, with a magnitude-calibration multiplier to keep mappings comparable across algorithms.

  • Temporal windowing

    Wave features require a filled 12-frame buffer per colour, so they carry a real latency floor: at frame_skip 4 on 30 fps footage the window spans roughly 1.6 s of source time. Per-frame features (intensity, area, brightness, gradients) are single-frame and immediate. Buffers are flushed whenever the ROI changes frame geometry, so stale-resolution frames are never compared.

  • Direction convention

    Flow angles are computed in image coordinates (y increasing downward) and rotated +90° into a clock convention before CC scaling, so 12 o'clock, meaning upward motion in the frame, is CC 0. Overlay arrows are drawn at a further +180°, because Farnebäck reports where pixels came from rather than where they are going.

  • Stability strategy

    Every feature is stabilised before it reaches MIDI: IIR smoothing on gradients and motion, exponential weighting on wave intensity, momentum on speed, and rate-limited circular smoothing on direction. Unsmoothed frame-to-frame estimates produce audible jitter, so the smoothing is a musical requirement. It also means the extracted features are a low-pass view of the scene, and short events are attenuated.

From scalar features to notes, harmony and control data

Two trigger modes

Notes are triggered either by brightness, where each discrete bright region within a colour mask triggers material, or by area, where the colour’s frame coverage alone selects it. The two are mutually exclusive for note triggering. Controller streams are emitted in both.

Two note modes

In single notes, each qualifying region yields one pitch, giving a pointillistic texture whose density tracks spatial fragmentation. In chords, only the brightest region is voiced, as a chord. Mode is set per colour class, so classes can differ.

The mapping layer

Nine stages from pitch space through note lifecycle to device routing.Open

  • Pitch space

    A scale is generated across 3 octaves from a configurable root pitch class and base octave. 37 scale types are implemented: diatonic modes, pentatonics, blues, jazz and symmetric sets, and world and synthetic scales. All output is constrained to this set.

  • Degree selection

    Each colour declares an ordered list of value bands; the band containing the current brightness or area supplies a chord_offset, an index into the generated scale. Bands are per colour, so the three classes trace different harmonic paths over the same scene.

  • Harmony

    A second, independent band table maps the same value onto chord type and inversion: 8 brightness bands or 6 area bands selecting among triad, seventh, ninth, eleventh and thirteenth. Inversions are applied by octave-displacing the lowest voice. Brighter or larger therefore means harmonically denser, not merely higher.

  • Velocity

    velocity = value × 127 × velocity_scale, clamped to 0–127, then raised to a per-colour min_velocity floor so faint detections stay audible rather than collapsing to silence.

  • Note lifecycle

    The system diffs a per-colour active-note set every frame and sends explicit note-offs for departing notes, including when a colour falls below threshold, disappears, or is disabled at runtime. This is what stops a continuously-varying scene producing stuck notes.

  • Controller streams

    CC values are de-duplicated against their last sent value. Intensity and area are the exceptions: they go out every frame, so downstream automation sees continuous motion rather than a stepped envelope.

  • Layering

    When a colour's area exceeds a threshold with hysteresis, its notes and CCs are duplicated to an additional MIDI channel. This orchestration layer engages only when that colour is prominent.

  • Bass voice

    An optional dedicated channel tracks either the pitch class of the lowest active green note or the configured musical root, revoiced to a chosen octave, with a hold-time debounce and optional sustain through silence.

  • Routing

    Per-colour and per-layer MIDI device assignment across multiple output ports, plus an MMC transport-play message on start for DAW synchronisation.

Operating as a live instrument

Live reconfiguration

  • YAML config watched on disk; observers refresh affected components in place
  • Hardware MIDI controllers mapped to configuration parameters, across multiple input ports
  • Gyroscopic controller input mapped to MIDI
  • Named YAML presets applied to the running system
  • On-screen overlay of live controller mappings and values

Compute

  • OpenCL (via OpenCV UMat), CUDA, and CPU backends with auto-detection and CPU fallback
  • Pre-allocated per-colour mask and gradient buffers; cached HSV bounds; cached inter-frame difference terms across the rolling window
  • Frame decimation and independent processing / display resolutions
  • Included GPU benchmark harness across backends and resolutions

Visual feedback

  • Composited mask overlays (solid or per-colour hatching), flow arrows, area coverage, measured processing rate
  • Debug readout of matched brightness and area bands
  • DPI-aware overlay scaling for projection

Provenance

  • Presented as a live electronic performance with projected footage
  • Aurora footage shot in Alaska by geophysics researcher Vincent Ledvina
  • 23 test modules covering music theory, wave direction and speed mapping, cloud mode, live reconfiguration and performance regressions

Scope and known limitations

  • Not photometrically calibrated.

    Detection operates on consumer-camera HSV values, which white balance, gain, gamma and codec compression all affect. Colour classes approximate auroral emission features. They are not radiometric measurements, and brightness values are not proportional to intensity in rayleighs.

  • Colour classes are configured, not learned.

    I hand-tune the HSV windows per footage source. There is no automatic exposure normalisation or per-scene adaptation, and the pink and purple windows overlap in hue, so a pixel can register in more than one class.

  • Features are deliberately smoothed.

    The stabilisation that musical output needs low-passes the extracted signals, and wave descriptors are windowed over a further 12 analysed frames. Anything faster than that window will not survive to the MIDI output.

  • No ground-truth validation.

    Extracted features have not been compared against all-sky camera data, magnetometer records or keogram-derived measures. Whether they track physically meaningful auroral dynamics is still an open question, and the one I would most like to work on with someone.

Source and recorded outputavailable on request

Ljómi is an independent project, MIT licensed, written in Python across 24 modules. Get in touch if you want the configuration examples, recorded output, or a conversation about validating the extracted features. Aurora footage credit: Vincent Ledvina.

Get in Touch Back to Ljómi