Technical fact sheet
Ljómi
/ˈljouːmɪ/ · Icelandic, “light”
A real-time system that analyses aurora video frame by frame and emits MIDI notes, chords and continuous controller streams derived from auroral colour, brightness structure, area coverage, centroid motion and optical-flow wave features.
- Status
- Working instrument
- Implementation
- Python, 24 modules
- Core stack
- OpenCV · NumPy · rtmidi
- Acceleration
- OpenCL / CUDA / CPU
- Input
- Video file or live camera
- Output
- MIDI 1.0, multi-port
- Licence
- MIT
- Author
- Ljomi Systems Ltd
Overview
Aurora as dataset and as performer
Ljómi treats each video frame as a field of measurable visual activity rather than an image to be recognised. It segments the frame into auroral colour classes, extracts per-class photometric and kinematic features, and maps those features onto pitch, harmony, velocity and continuous control data. Every mapping is declared in YAML and can be re-tuned live, during a performance, without restarting the process.
The design commitment is that MIDI — not audio synthesis — is the output. The system produces a symbolic control stream, which keeps the interpretive step explicit and inspectable: the aurora supplies behaviour, contour and constraint, while the mapping layer supplies the musical decisions. It makes no claim that the aurora “sounds like” the result.
Signal path
Per-frame processing pipeline
Stage 01
Acquire
File or camera capture, decimated by frame_skip, resized to a fixed processing resolution (default 854×480), converted BGR→HSV.
Stage 02
Segment
Per colour class, an HSV threshold produces a binary mask, refined by morphological opening then closing with a 5×5 elliptical kernel.
Stage 03
Constrain
Optional interactive ROI rectangle and a freehand painted exclusion mask are ANDed into every colour mask, suppressing foreground light pollution.
Stage 04
Extract
Ten features per colour class: intensity, area, mean brightness, brightness regions, Sobel gradients, centroid motion, and four optical-flow wave descriptors.
Stage 05
Map & emit
Features are quantised to scale degrees, chord types and CC values, then routed per colour to MIDI channels and devices with explicit note-lifecycle tracking.
Detection · what
Three auroral colour classes, independently voiced
Colour classes are chosen to correspond to the dominant auroral emission features — atomic oxygen at 557.7 nm, the oxygen red line near 630.0 nm, and the N₂⁺ bands toward 427.8 nm — as they appear in consumer-camera footage. Each class is segmented, analysed and routed independently, so a colour behaves as its own musical voice. Classes can be enabled or disabled at runtime; disabling one flushes its held notes.
Green
Ch 1557.7 nm — atomic oxygen
- Hue
- 45–76
- Sat
- 30–255
- Val
- 30–255
- Regions
- max 4, ≥0.1%
Pink
Ch 2630.0 nm — oxygen red line
- Hue
- 140–180
- Sat
- 0–255
- Val
- 0–255
- Regions
- max 3, ≥0.01%
Purple
Ch 3427.8 nm — N₂⁺ bands
- Hue
- 115–155
- Sat
- 10–255
- Val
- 10–255
- Regions
- max 3, ≥0.01%
Hue is in OpenCV’s 0–180 convention (half-degrees). Saturation and value floors are deliberately permissive to retain faint pink and purple structure. A parallel cloud_ranges set substitutes three low-saturation luminance bands, letting the same pipeline sonify daytime cloud without code changes — useful as a control condition.
Detection · how
Feature extraction and its methods
Ten features are extracted per colour class, per analysed frame. Six are photometric or kinematic measures of the mask itself; four describe the wave structure of dense optical flow across it.
intensity
CC 74every frameMean HSV value channel within mask, normalised by 255
Range 0–1
area
CC 75every frameNon-zero mask pixels ÷ total frame pixels
Range 0–1
mean brightness
CC 87Masked mean of the value channel
Range 0–1
brightness regions
→ notesValue threshold → 8-connected components with stats; regions below a minimum area fraction discarded, survivors ranked by brightness × area
Range 0–4 regions
brightness gradient
CC 85 / 86Sobel ∂x and ∂y over the masked value channel, L1 mean per axis, first-order IIR smoothing (α = 0.8)
Range 0–1 per axis
centroid motion
CC 71 / 72Image-moment centroid displacement between frames, amplified (×4.6), IIR-smoothed (α = 0.9), clamped and offset to a bipolar CC
Range ±1 → 0–127
wave coherence
CC 24Magnitude of the first circular moment of dense flow angles, |⟨e^iθ⟩|; gates all wave output below 0.05
Range 0–1
wave intensity
CC 22Temporal brightness variation over the frame window: 0.4·mean normalised inter-frame difference + 0.6·√(max local variance), exponentially weighted over 15 samples
Range 0–1
wave direction
CC 20Magnitude-weighted mean flow angle over significantly-moving pixels, rate-limited to 5–40° per step, blended 0.7 / 0.3 with the circular mean of a 5-frame history
Range 0–360°
wave speed
CC 21Flow magnitude weighted by local (20×20) directional coherence, pooled as 0.6·mean + 0.2·max + 0.2·σ, compressed by s/(s+5), momentum 0.7, γ = 1.2
Range 0–1
Dense optical flow
Wave descriptors derive from Farnebäck dense flow computed on the masked value channel, with parameters tuned for the aurora's long, low-contrast structures rather than rigid-body motion: pyr_scale 0.4, levels 5, winsize 25, iterations 3, poly_n 7, poly_sigma 1.5. Flow runs at half the processing resolution. A DIS flow backend is selectable for A/B comparison, with a magnitude-calibration multiplier to keep mappings comparable across algorithms.
Temporal windowing
Wave features require a filled 12-frame buffer per colour, so they carry a real latency floor: at frame_skip 4 on 30 fps footage the window spans roughly 1.6 s of source time. Per-frame features (intensity, area, brightness, gradients) are single-frame and immediate. Buffers are flushed whenever the ROI changes frame geometry, so stale-resolution frames are never compared.
Direction convention
Flow angles are computed in image coordinates (y increasing downward) and rotated +90° into a clock convention before CC scaling, so 12 o'clock — upward motion in the frame — is CC 0. Overlay arrows are additionally drawn at +180°, because Farnebäck reports where pixels came from rather than where they are going.
Stability strategy
Every feature is stabilised before it reaches MIDI: IIR smoothing on gradients and motion, exponential weighting on wave intensity, momentum on speed, and rate-limited circular smoothing on direction. This is a musical requirement — unsmoothed frame-to-frame estimates produce audible jitter — but it also makes the extracted features a deliberately low-pass view of the scene.
MIDI conversion
From scalar features to notes, harmony and control data
Two trigger modes
Note generation is driven either by brightness — discrete bright regions within a colour mask each trigger material — or by area, where the colour’s frame coverage alone selects material. The modes are mutually exclusive for note triggering; controller streams are emitted in both.
Two note modes
In single notes, each qualifying region yields one pitch, giving a pointillistic texture whose density tracks spatial fragmentation. In chords, only the brightest region is voiced, as a chord. Mode is set per colour class, so classes can differ.
Pitch space
A scale is generated across 3 octaves from a configurable root pitch class and base octave. 37 scale types are implemented — diatonic modes, pentatonics, blues, jazz and symmetric sets, and world and synthetic scales. All output is constrained to this set.
Degree selection
Each colour declares an ordered list of value bands; the band containing the current brightness or area supplies a chord_offset, an index into the generated scale. Bands are per colour, so the three classes trace different harmonic paths over the same scene.
Harmony
A second, independent band table maps the same value onto chord type and inversion: 8 brightness bands or 6 area bands selecting among triad, seventh, ninth, eleventh and thirteenth. Inversions are applied by octave-displacing the lowest voice. Brighter or larger therefore means harmonically denser, not merely higher.
Velocity
velocity = value × 127 × velocity_scale, clamped to 0–127, then raised to a per-colour min_velocity floor so faint detections stay audible rather than collapsing to silence.
Note lifecycle
A per-colour active-note set is diffed every frame; departing notes receive explicit note-offs, including when a colour falls below threshold, disappears entirely, or is disabled at runtime. This is what keeps a continuously-varying scene from producing stuck notes.
Controller streams
CC values are de-duplicated against their last sent value, except intensity and area, which are force-sent every frame so downstream automation sees continuous motion rather than a stepped envelope.
Layering
When a colour's area exceeds a threshold with hysteresis, its notes and CCs are duplicated to an additional MIDI channel — an orchestration layer that engages only when that colour is prominent.
Bass voice
An optional dedicated channel tracks either the pitch class of the lowest active green note or the configured musical root, revoiced to a chosen octave, with a hold-time debounce and optional sustain through silence.
Routing
Per-colour and per-layer MIDI device assignment across multiple output ports, plus an MMC transport-play message on start for DAW synchronisation.
System
Operating as a live instrument
Live reconfiguration
- YAML config watched on disk; observers refresh affected components in place
- Hardware MIDI controllers mapped to configuration parameters, across multiple input ports
- Gyroscopic controller input mapped to MIDI
- Named YAML presets applied to the running system
- On-screen overlay of live controller mappings and values
Compute
- OpenCL (via OpenCV UMat), CUDA, and CPU backends with auto-detection and CPU fallback
- Pre-allocated per-colour mask and gradient buffers; cached HSV bounds; cached inter-frame difference terms across the rolling window
- Frame decimation and independent processing / display resolutions
- Included GPU benchmark harness across backends and resolutions
Visual feedback
- Composited mask overlays (solid or per-colour hatching), flow arrows, area coverage, measured processing rate
- Debug readout of matched brightness and area bands
- DPI-aware overlay scaling for projection
Provenance
- Presented as a live electronic performance with projected footage
- Aurora footage shot in Alaska by geophysics researcher Vincent Ledvina
- 23 test modules covering music theory, wave direction and speed mapping, cloud mode, live reconfiguration and performance regressions
Scope and known limitations
Not photometrically calibrated. Detection operates on consumer-camera HSV values, which are subject to white balance, gain, gamma and codec compression. Colour classes approximate auroral emission features; they are not radiometric measurements, and brightness values are not proportional to intensity in rayleighs.
Colour classes are configured, not learned. HSV windows are hand-tuned per footage source. There is no automatic exposure normalisation or per-scene adaptation, and the pink and purple windows overlap in hue, so a pixel may register in more than one class.
Features are deliberately smoothed. The stabilisation needed for musical output low-passes the extracted signals, and wave descriptors are additionally windowed over 12 analysed frames.
No ground-truth validation. Extracted features have not been compared against all-sky camera data, magnetometer records or keogram-derived measures. Establishing whether they track physically meaningful auroral dynamics is an open question — and the most interesting one for collaboration.
Contact
Source and recorded output available on request
Ljómi is an independent project, MIT licensed, implemented in Python across 24 modules. If you want the configuration examples, recorded output, or a conversation about validating the extracted features, get in touch. Aurora footage credit: Vincent Ledvina.