the augmented theatre

The audience's phones, cued from the console.

Augmented Theatre sends text, images, video, sound, colour and vibration to the audience's phones during a performance. An operator cues them from a console on the venue's own Wi-Fi.

Download on the App Store Get it on Google Play
AUGMENTED THEATRE SHOW · 2016 · PHOTO — LEONID SELEMENEV
AudienceAugmented Theatre App

The audience installs the app before the performance. There is no account to create and nothing to configure. During the show every phone acts on the cues the operator sends, at the same time.

A phone held in the audience showing the stage through the app's rain
            effect, the actors dissolving behind running water
Suddenly a distant sound is heard as if from the sky,
the sound of a string breaking, dying away The Cherry Orchard
TextSurtitlesA line of text appears on every phone when the operator sends it. The size is set on the cue. Text is drawn over whatever is already on screen — a film, a camera look, a shader — rather than replacing it.
Images and videoMedia playbackVideo and still images play on every phone, started by one cue and kept in step.
LightColour and torchEvery screen fills with a single colour, or the torch on the back of every iPhone turns on. Android devices have no torch here and show a white screen instead.
VibrationHapticsA cue sets how long the phone vibrates, how long after the cue it begins, how strong it is, and whether it repeats until stopped.
CameraCamera looksThe camera picture is drawn with an effect applied: rain on glass, heat haze, scratched film, or the front camera.
AR setsAugmented realityThe audience points the phone at a printed marker image. A 3D set is drawn onto it, with video playing on its surfaces.
Spatial audioPositioned soundWith headphones, a sound source can be placed at a position around the listener and moved while it plays.
Audio-reactiveShaders driven by the microphoneA GLSL fragment shader whose output changes with the sound level from the phone's own microphone, so the image follows the sound in the venue.
MIDIThe ensembleEach phone sounds one note of a chord played at the console. Which note a phone takes is decided by the phone itself, so the chord is distributed across the audience rather than sounded in full by every phone.
DeliveryMedia sent before the showThe show's files are copied to each phone over the venue's Wi-Fi while the audience takes its seats. No internet connection is needed, and nothing appears on screen until a cue asks for it.
In the backgroundThe phone keeps listeningOn Android, leaving the app does not take the phone out of the performance: a notification says the show is running without you, and returns you to it. On iPhone the app keeps running while show audio is playing.

Free on the App Store and Google Play. It carries no advertising and asks for no account, and all of its traffic stays on the venue's own network.

Augmented reality on the audience's phone

The consoleAugmented Theatre Dashboard for Mac
Download on the Mac App Store

The operator's console. It holds the show's surtitles and sends them a line at a time, copies the show's media to the audience's phones before the performance and reports how much each one holds, and keeps a running order in which media and effects are the same kind of row. It runs on the venue's own Wi-Fi and needs no internet connection.

The Augmented Theatre Dashboard: the script pad holding the Macbeth soliloquy with the two Send buttons that walk down or back up it, the effect-chip card with the whole cue vocabulary, the show timeline — the same clip as flat video and as a 360° sphere, a colour-screen instance, rain, a positioned 3D voice — each row carrying its own settings and a live count of the phones holding it, and at the foot the second dashboard on the network, offered the show
SHIFT-RETURNScript padHolds the show's surtitles. Each press sends the current line to every phone and moves to the next; the line the audience is reading stays marked. ⌘-Return moves back one line.
BEFORE THE PERFORMANCEDeliveryImages, video and sound are copied to each phone before the performance. A phone does not fetch a file it already holds.
THE LISTRunning orderMedia and effects in one list, dragged into order and duplicated with ⌘D. Each row carries its own settings, so the same clip can play flat in one row and as a 360° sphere in another.
CUE STRIPEffectsRain on glass, heat haze, scratched film, rain in three dimensions, the back and front cameras, a colour screen, vibration, the torch, a system sound, an audio-reactive shader and a MIDI row played from a keyboard. Each chip adds a row to the show.
KEYBOARDPlay the audienceA MIDI row makes every phone sound one note of a chord, played from the keyboard on screen or from one plugged into the Mac. Learn takes the channel and the span of keys from two keys played on it.
3DPositioned soundBinaural audio placed at a point around the listener and moved while it plays, using a position pad.
GLSLShaders are checked before sendingA fragment shader is compiled against the same GLES rules the phones enforce, and every uniform it declares is checked against the ones the app supplies. A shader that would render black on the handsets is rejected rather than sent. Each phone then drives it from its own microphone.
DROP A FOLDERAR setsA folder holding the anchor image, the model and its projections is checked, bound together and sent as one item, then raised and struck by cue.
START · PAUSE · STOPTransportStart plays a clip, shows an image, runs an effect or raises an AR set, according to the row. Stop removes it.
UPDATE ASSETSDelivery tallyHow many phones hold each file, and the percentage completed on those still fetching it.
0 – 100%Part of the audienceA cue can be addressed to a fraction of the phones rather than all of them.
RIGHT-CLICKCopy OSC AddressPuts the exact cue a row sends on the clipboard, as one line. A flat row and a 360° row of the same clip produce different cues. Long-press does the same on iPad.

Free to download. A demo show is staged on first launch with one of each kind of item: a still, an animated GIF, a flat clip, a 360° clip, a sound, a positioned voice, an AR set and an audio-reactive shader. Text and the basic cues are free indefinitely. Sending files, addressing part of the audience and the delivery tally run free for five days and are then part of the full version, a one-time purchase. Requires macOS 11 or later.

ManualThe Dashboard manual
What this app does

The Dashboard is the operator's console. It sends cues over the venue's own Wi-Fi to the free Augmented Theatre app running on the audience's phones: lines of text, and the images, video and sound a production needs. There is no setup on the phones and no pairing — a phone that receives a cue acts on it.

Shows
The File menu: making, opening and revealing a show
The File menu: making, opening and revealing a show

File ▸ New Show… asks where your shows are kept, then makes one. A show is an ordinary folder: a document with the show's name, and a media folder holding everything sent to the audience for it. Inside that media folder each AR set has a folder of its own, named after the set, holding its drawing, its model, its videos and its sound — so one row in the list is one folder on disk, and the anchor to print is where you would look for it. Everything else lies flat beside them. A show made before this is reorganised into that shape the next time it is opened. File ▸ Open Show… reopens one, and Reveal Show in Finder opens it in Finder. To remove a production after its run, delete that folder.

Text cues
The script pad, the two Send buttons, the text-size slider and Pulse
The script pad, the two Send buttons, the text-size slider and Pulse

Type in the large field and press Return to put that line on every phone. The slider under it sets the size the text is drawn at. Pulse the text size makes the line change size repeatedly — press it again to stop. A line is drawn over whatever the phones are already showing — a film, a camera look, a shader — rather than replacing it, so a translation can run through a scene. Cued on its own it appears on black. Sending an empty line removes the text; it does not blank the screen, which is what Stop and the Clear chip are for.

Sending files
The cue strip; the five media chips are ringed
The cue strip; the five media chips are ringed

A chip for each kind: Video, Image, Audio, AR Set and Audio-reactive. Each one opens a panel offering only what the phones can present as that kind, so a file they could not show cannot be staged. Files are copied into the show's media folder and announced to the phones, which fetch them in the background — while the audience takes its seats, so that a cue during the performance is immediate. A phone never fetches a file it already holds. An AR set is a folder rather than a file, and can also be dragged straight onto the list.

The cue strip
The cue strip: every chip adds a row to the show
The cue strip: every chip adds a row to the show

The row of chips above the delivery list adds instances — rows with nothing to deliver behind them. An instance is a cue the phones already know how to act on, so nothing is staged and nothing is fetched: it is ready as soon as you add it. Each carries its own setting on its row — a colour, a sound, a line of text, a level — and Start and Stop send it exactly as they send a delivered file. Instances belong to the show and are still there when you open it again. One thing they all share: a chip addresses every phone. The share slider does not reach them, which is why their rows never offer one.

Colour, vibration and torch
The cue strip; Colour Screen, Vibrate and Torch are ringed
The cue strip; Colour Screen, Vibrate and Torch are ringed

Colour Screen fills every screen with a single colour, chosen on the row. Vibrate vibrates every device, and its row sets how: how long one buzz lasts and how long after the cue it begins — both in milliseconds, up to a minute — how strong it is, as a percentage of full strength, and whether it repeats until the instance is stopped. On the wire that is /vibrate/1 length start loop strength, in that order. A phone with a haptic engine applies all four; an older one vibrates for approximately that length at its only strength. Torch turns on the torch on the back of every iPhone; Android has none here and shows a white screen instead. All three take effect on every phone at once.

The debug switch

At the far right of the cue strip, past the looks and set apart from them, is a switch rather than a chip: it adds no row to the show and delivers nothing. While it is on, every phone draws the cues it receives along the foot of its own screen, over whatever it is showing, and goes on acting on them as usual. It answers the question that costs the most rehearsal time — did the cue reach the phones at all — because a phone that is silent because it never received the cue is indistinguishable from one that received it and did something else.

Each line is the phone's own clock, the address and the arguments as they arrived; the first line is that phone's IP and port, so one phone can be told from another. A cue that repeats is collapsed into a single line that counts up, so a running MIDI row does not push everything else off the screen. On the wire it is /debug/1 and /debug/0, which any sender can use.

A phone that joins after the switch was pressed has not received it and shows nothing; press it again. Leave it off with an audience present.

Title and volume
The cue strip; Title and Volume are ringed
The cue strip; Title and Volume are ringed

Title writes a line into the top bar of every phone — the act, the scene, the name of the piece — typed on the row. iPhones show it; Android ignores it. Volume sets how loud clips and sounds play, and is a level rather than a cue: nothing waits on its having been sent, and it applies to whatever is playing now as well as to whatever plays next, so it can be changed under a running clip. A spatialized 3D voice is the one exception: it does not respond to it.

The House chip
The cue strip; the House chip is ringed
The cue strip; the House chip is ringed

House sends /start/1 and /stop/1 to every phone. Start carries the show's name and, optionally, a title line; Stop takes no arguments.

On Start a phone stops whatever clip is playing, switches to that show's files, displays Poster.jpg — the delivered one if the show staged it, the app's own otherwise — and marks itself as being in the show. While it is, the idle screen's own controls are suppressed: the six-tap diagnostic menu will not open, and on Android the AR button and the Wi-Fi readout are hidden.

On Stop a phone stops whatever clip is playing, displays the foyer logo, clears the title line and any surtitle still on screen, and marks itself as out of the show; the idle controls return. Because Stop puts a picture on the screen it also replaces a colour fill, a camera look or a shader, which is why it is used to clear every phone at the interval or at the end.

System sounds
The cue strip; the Sound chip is ringed
The cue strip; the Sound chip is ringed

A sound instance plays one of the sounds already on the phone. Nothing is staged and nothing is fetched, so it is the one cue that works on a phone holding no files at all — useful when a sound is needed immediately and there was no time to deliver one. The bell on the row selects which. iPhones only: Android ignores it. It plays at the phone's own ringer volume, which the audience sets and the operator does not, and a phone switched to silent may not play it at all, so a system sound should never be a cue the production depends on.

The sounds the menu offers

These are the six on the bell menu, each with the number it sends. The Dashboard offers a short list deliberately: these are the ones that remain audible across a venue.

Other ids an iPhone answers

The cue itself takes any id iOS knows, so a patch written by hand — see Copy OSC Address — can reach the rest. Those from 1020 to 1036 are the alert tones iOS lists under Settings ▸ Sounds. An id iOS does not know plays nothing at all, silence rather than an error, and the set varies a little by iOS version and device, so test one on the phones you will actually use before an audience is present.

MIDI
The cue strip; the MIDI chip is ringed
The cue strip; the MIDI chip is ringed

A MIDI row makes every phone sound one note of a chord. Which note a phone takes is decided by the phone from its own install identifier, so a triad is distributed across the audience rather than sounded in full by every phone. A chord carries up to sixteen notes. The sound is generated on the phone: a struck tone that decays.

Start does not sound anything. It sends /midi/1, which makes every phone allocate its audio and begin listening. Starting an audio engine on a few hundred phones takes long enough that a chord sent with it would sound late, so arming and playing are separate cues. Stop silences every phone and ends the row.

No note event is ever sent. The keys held down at the console are the state, and the Dashboard sends that state again about three times a second for as long as the row is running. A chord is not a message that can be missed: it is sent again 0.3 s later. Releasing a key is not a message either — it produces a different chord, repeated like any other. A phone that stops receiving falls silent by itself when the ttl expires. Releasing every key is a rest: the phones go quiet without the row being stopped.

How a MIDI row plays

The keyboard button on the row opens its settings. Density is how often each phone attacks its note again, in attacks per minute; at 0 it is attacked once and left to decay, and above 0 it repeats. Each phone offsets its own re-attacks by up to a quarter of that interval either way, so that a hundred phones do not attack in lockstep; the offset is derived from the phone's identifier, so two performances sound the same. Gain is the level, 0 to 127. Share is the fraction of phones in the ensemble, and it is the one share in the Dashboard that is not re-rolled for each cue: a phone decides once, from its own identifier, and stays in or out for the whole passage.

Try sounds a chord at the settings currently shown, without starting the row.

Playing it
The keyboard window
The keyboard window

Window ▸ Keyboard opens the keyboard on screen; starting a MIDI row opens it as well, already armed. Play it with the mouse or type: the letter layout is GarageBand's Musical Typing, and the second button under the keys draws the letters on the keys themselves. z and x change octave.

A MIDI keyboard connected to the Mac is used without configuration: USB, Bluetooth, or another application sending to the IAC bus, which is how a cue list in Ableton or Isadora drives the phones with no instrument present. The line under the keys names the MIDI source in use, or states that there is none; read it first when a key produces no sound. Keys played on the instrument light up on the drawn keyboard, so one window shows what is being held however it was played. A keyboard disconnected mid-chord releases every key it was holding.

The keyboard window, showing which letters play which keys
The second button draws the letters on the keys they play
Channel, key span, and Learn

Two settings on the row select which keys drive it. Any channel is the default and is correct for an operator with one keyboard. The key span is the range of keys this row answers, which is how one keyboard drives two rows from two hands: each row answers its own range.

Learn sets both without a number being entered. Press it, play the lowest key this row should answer, then the highest; the channel is taken from the same keys. Either order gives the same span, and the same key twice gives a row that answers one key. Nothing sounds while it is listening, and anything being held is released first. It is disabled when the console can hear no keyboard, is enabled as soon as one is connected, and stops and says so if the keyboard it was listening to is disconnected.

Camera looks
The cue strip; the camera looks are ringed
The cue strip; the camera looks are ringed

Five chips put each phone's own camera on its own screen, so a phone shows what its camera sees rather than anything delivered to it. Rain runs water over the glass, or, selected on the row, falls in a sphere around the viewer. Heat is a shimmer. Dziga is scratched film, after Dziga Vertov. Camera is the back camera unmodified. Mirror is the front camera. Each phone renders its own, so nothing is delivered and no two are identical. Clear ends whatever look is running and returns the phone to the show — one row that closes any of them, including an audio-reactive shader.

Audio-reactive shaders
The cue strip; the Audio-reactive chip is ringed
The cue strip; the Audio-reactive chip is ringed

Audio-reactive adds a shader — a small program that draws a look, handed to the Dashboard as a .frag file. It is driven by each phone's own microphone, so one shader renders differently on every phone, each following the sound level its own microphone reports. Nothing is synchronised and nothing crosses the network while it runs.

The Dashboard checks a shader before it will send it and reports what is wrong with one it refuses: it has to compile, and it may only read values the phones supply. A phone requests the microphone the first time a shader runs, never at launch, and stops listening when the look ends. A phone cannot measure the sound in a venue it is playing sound into, so starting a shader stops whatever is playing, and starting anything with sound or video ends the shader.

The demo show staged on first launch includes one, Rainbow Spinning Circle. It is commented throughout and names the four values it is given: the elapsed time, the sound level, the viewport size, and the live camera if it declares one.

AR stage sets
The cue strip; the AR Set chip is ringed
The cue strip; the AR Set chip is ringed

An AR set is scenery the audience raises on their phones over something real. The phone recognises an anchor — a printed drawing in their hands, an object, or the stage itself — and the set is drawn standing on it, with video playing on its surfaces. Drop the AR set's folder onto the delivery table to import it: the Dashboard checks it, delivers it to the phones like any other file, and the set appears as one row. Start raises it on the phones; Stop strikes it.

Creating an AR set

The set that ships with the Dashboard is a worked example, and both scripts that generated it travel inside it: Help ▸ the website carries demo-triadic-ballet.zip.

The anchor is the part to understand first. A phone cannot place a set until it recognises something real to place it on, and the anchor is that thing — a printed picture the audience holds, or one on a wall. The phone matches the camera image against it, computes where it is and how large it appears, and draws the set on top. That is why the anchor's printed width is declared in ar-set.json: every dimension in the model is measured against it.

Draw an anchor the way a tracker reads one. It finds a picture by its corners and edges, so a few large flat shapes give it almost nothing; break the design up until there is detail everywhere and no two corners look alike, and keep it asymmetric — a design with a mirror line can be recognised the wrong way round. At least 640 pixels across.

Then the set itself. Build in real metres and in Blender's own Z-up; the figure in the demo is 0.21 m tall, which is its height in the venue. Name every surface that carries video and UV-unwrap it — video is applied through texture coordinates, and a surface without them shows nothing — then give each one a custom property “video” naming its file, so the binding travels inside the model. Export glTF Binary as one file, with “+Y Up” unticked and Include ▸ Custom Properties ticked.

If the set is animated, make every bone that moves point the same way as its parent. A limb pointing down from a parent pointing up rests at exactly 180°, and a rotation about that angle makes the exporter write the same orientation with opposite signs on consecutive keys; glTF interpolates rotations linearly, so the limb rotates the long way round. A full revolution crosses the same angle once per turn.

A set can carry its own sound: put an audio file in the folder, or name one with the “audio” key. It loops for as long as the set stands and is independent of the screens, so a set with no video surface at all can still have sound.

The folder then holds the anchor, the .glb, any videos, any audio and ar-set.json. Drop it on the delivery table; the import sheet checks every piece and reports what is missing before anything is sent.

Building an AR set, step by step

1 · The anchor. The AR set is raised over something real, and the anchor image is how the phone recognises it. It can be a drawing printed and handed to the audience, a photograph of an object placed in the venue, or a photograph of the stage itself, taken from where the audience will stand. Any image works, but give it detail and contrast: at least 640 pixels across, and visually distinct from every other set's anchor in the show — two similar anchors make the phones guess which AR set to raise. Measure the real thing now: the width the anchor has in the venue (a printed A4 sheet is 0.21 m — the default) is what sets the scale of everything.

2 · The model. Build the AR set in Blender (or any tool that exports glTF), in real metres, in Blender's own Z-up axes. Imagine the anchor lying flat at the origin: the set stands on and around it, facing −Y — towards the viewer. An AR set anchored to a sheet in the audience's hands is small by design; one anchored to the stage is built at stage size. Name every surface that will carry video (screen-left, backdrop…) and UV-unwrap those surfaces — video lands through texture coordinates, and a mesh without them shows nothing.

3 · The projections. For each video surface, tell the AR set which file goes there. The cleanest way is in the model itself: add a custom property named video to the surface's object in Blender, with the video's filename as its value. Or skip properties and simply name each video file after its surface (screen-left.mp4). Or say it in ar-set.json — see below.

4 · Export. glTF Binary (.glb), one file — a .gltf with separate textures is refused. Two export settings matter: untick “+Y Up” (the phones expect the file Z-up, exactly as Blender has it, and do the turn themselves), and tick Include ▸ Custom Properties if the video bindings live in the model.

5 · The folder. One folder, named after the AR set: the anchor image, the .glb, the videos, and optionally a ar-set.json. Drop it on the Dashboard's delivery table. The import sheet checks everything — the anchor's size, the model's surfaces, every binding — and reports what is missing or ambiguous; nothing is sent until the errors are gone. Then Send, and the AR set is delivered to the phones like any other file. Start raises it; Stop strikes it.

⇩ starter-set.zip — a complete working AR set with a README
The demo show's AR set: anchor, metre-scale .glb with two named screens, two videos and a fully spelled-out ar-set.json. Take it apart and replace every piece with your own.

Preparing an AR set folder

One folder holds one AR set. It needs the anchor image — a drawing to be printed and handed out, or a photograph of the object or stage the set will stand on (JPEG or PNG, at least 640 pixels across; detailed, high-contrast images track best) — and the set itself as a single .glb, authored in metres (a .gltf with separate textures is refused). Videos beside them (.mp4, .mov, .m4v) are projected onto the model's surfaces: state which goes where in the model's own node properties (a custom property named “video” in Blender), in an ar-set.json beside it, or by naming the video after the surface. An optional ar-set.json can also name the anchor and model among other files and give two numbers: “widthMeters”, the real width of the anchor as the audience sees it (0.21 — an A4 sheet — if unstated), and “scale”, a correction for models not authored in metres (1 if unstated). Both can also be set in the import sheet. Keep an AR set well under 60 MB: every phone downloads all of it while the audience takes its seats.

The ar-set.json companion file

A small JSON file beside the model, for what the files alone cannot say. Every key is optional — a folder with one obvious drawing, one .glb and self-naming videos needs no ar-set.json at all. The folder's name becomes the AR set's id and title.

{
  "marker":      "three-sisters.jpg",
  "model":       "stage.glb",
  "widthMeters": 0.297,
  "scale":       1,
  "projections": {
    "screen-left":  "clouds-left.mp4",
    "screen-right": "clouds-right.mp4"
  }
}

marker — which image is the anchor, when the folder holds more than one. Without it, the first image alphabetically is used, with a warning.

model — which .glb is the AR set, when there are several.

widthMeters — the width the anchor really has in the venue, in metres: 0.21 (the default) for a printed A4 sheet, 0.297 for A3, or the measured width of the object or stage view photographed. The whole AR set scales from this one number — the same image printed at two sizes is two different anchors, so measure the real thing.

scale — a correction for models not authored in metres: the multiplier that brings one model unit to one metre. 1 (the default) for a correctly authored file.

projections — surface name → video filename, for models whose bindings are not in the file itself. Bindings written in the model's own custom properties win over ar-set.json; a video named after its surface is the fallback when neither says.

A template folder to copy

A folder holding the smallest set that works, drawn to the rules above. Drag it onto the delivery table as it stands and a set imports, with no errors and no warnings, and sends; then copy the folder, give it the name your set should have, and replace the pieces one at a time. Three files and a folder of documentation:

drawing.png — the anchor, 1654 × 2339 pixels, which is A4 at 200 dpi. High contrast, irregular speckle everywhere for the tracker to hold on to, large unlike shapes in the corners so a phone can tell which way up the sheet is, and no line of symmetry. A print-ready PDF of the same drawing lies beside it.

figure.glb — the model, in metres, with two surfaces whose names are the ones you will see in the import sheet: screen, a vertical 30 × 17 cm plane standing over the sheet, and base, a 30 × 30 cm pad lying flat on it. Drop a video beside them and there is somewhere obvious to put it.

ar-set.json — every key spelled out, including the two that are usually left to their defaults, so there is something to edit rather than something to write.

README/ — the instructions, and the sheet to print. They are in a subfolder on purpose, and it is the one convention worth carrying into your own sets: the Dashboard warns about every file that is not a drawing, a model, a video or a sound, and skips subfolders entirely. Notes, Blender sources, drafts and anything else that is not sent to the phones belongs in one, where it raises nothing.

⇩ ar-set-template.zip — the template, 67 kB
The bare minimum, to start from. For a set with everything in it — videos, projections, a score — take starter-set.zip above instead.

Printing the anchor, and measuring it

Print it at full size, with scaling off — “Fit to page” quietly shrinks a sheet by a few per cent, and the anchor's width is the one number the whole set hangs off. Then measure the printed sheet with a ruler and put that figure in widthMeters: 0.21 for the short edge of A4, 0.297 for A3, whatever the ruler says for anything else. Anything from a centimetre to three metres is accepted; outside that the Dashboard refuses to send the set, because no default is safe when the same drawing printed at two sizes is two different anchors. A number that turns out wrong can be corrected in the import sheet without touching the file. Print on matte paper: gloss catches the stage light, and a phone cannot track a reflection.

What travels to the phones

The folder is read at import, not sent. What the phones receive is the files themselves — the anchor, the model, the videos, the score — and one small manifest the Dashboard writes, the .atcset, naming nothing by filename: each part is a sha256 digest of its own contents, with the anchor's width and the model's scale beside them. A phone keeps files by digest and never learns their names, which is why the manifest is written that way, and why your ar-set.json does not travel: it is read here, and the choices it settles — which image is the anchor, which .glb is the set, which video is on which surface — are frozen into the manifest as digests. The phones need no filenames to know which image is the anchor, because by then it is a digest and not an image among others.

Two things follow. Editing an .atcset by hand is pointless: a digest that does not match is a part no phone can find, and a set is raised only once a phone holds every part of it — an anchor that tracks but has no complete set behind it is worse than one that is never recognised. And dropping a folder of the same name a second time replaces that set rather than adding a second one, because the folder's name is the set's id as well as its title.

Delivered to the audience
The delivery list, with the Delivered column
The delivery list, with the Delivered column

Delivered counts the phones that hold the whole file; while any are still fetching it, a bar beside the count shows how far they have got on average. A dot beside a name means the audience is looking at that file now. Phones report as they download; one that arrived with the file already cached from an earlier session never reports, so this table is a lower bound rather than a count of the audience.

Copy OSC Address
Right-clicking a row in the delivery list
Right-clicking a row in the delivery list

Right-click a row in the delivery list — long-press on iPad — and Copy OSC Address puts that row's cue on the clipboard exactly as it goes on the wire, as one line: the address, and the file's name where the cue names a file. It is meant for wiring a patch in Isadora, or any other OSC sender, against this show. It copies the name, which you could have typed yourself, and falls back to the digest only where two files in the show share a name. A flat clip and a 360° clip give different cues, because they are different cues. The share, the loop flag and the show are left out on purpose: those are whatever the Dashboard's own controls read at the moment you copy, not part of the row. Beside it, Copy OSC Stop Address gives the cue that ends the same row — /dziga/0 for a kino-eye, /atc/stop and the name for a clip — and is absent on the rows that have no stop, such as a bell or a title.

The MIDI row copies two lines rather than one, because its start is two cues: /midi/1 arms the row and /midi/state carries the chord. Arming alone is silent, and the chord has to keep arriving — about three times a second — or the phones let it expire and stop sounding. The numbers in the second line are an example rather than the sliders' current reading, and every one of them is a whole number deliberately: a decimal between 0 and 1 anywhere in that run is read as the share.

Play, Pause, Stop, Show, Hide
Start, Pause and Stop
Start, Pause and Stop

Select a delivered file and use these to put it on the audience's screens. Play, Pause and Stop are for video and sound; Show and Hide are for stills. Stop returns to the start and leaves the screen as it was before. None of these send more than one small packet, so they are safe to use mid-scene.

Part of the audience
The audience-share slider
The audience-share slider

The slider beside the transport sets how many of the phones a cue addresses. Every phone decides for itself, so at 40% roughly two in five act on it — not an exact count, and a different two in five each time the cue is sent. At the top of the slider the cue addresses every phone.

The playhead

While video or sound is playing, the bar shows approximately how far into the file the phones are. The phones do not report their position, so this is the Dashboard's own clock, started when the cue went out. The ± figure is the spread across the phones: one broadcast starts every phone at once, and what remains is the few hundred milliseconds of network and player start, widening slowly as phone clocks drift apart.

Two consoles, and how to be the only one
The other console on the network, and Hand Over
The other console on the network, and Hand Over

Two Dashboards on one network find each other and agree which of them is driving: the other says ON STANDBY and stays quiet, so a show is never cued by two consoles at once. The show can be handed over mid-performance from the File menu, and a console that is handed a show it does not hold fetches it first. Settings ▸ Hand the show between consoles turns the whole arrangement on; it is off when a Dashboard is first installed, and both consoles need it on. A Dashboard with it off — which is how one arrives — is the only console on the network by declaration: it announces nothing, so no other console stands down because of it; it ignores every request to hand over or take the show; and it drives from the moment it opens. That is what you want on a laptop that shares a network with other people's Dashboards — a classroom, a shared studio — where otherwise the nearest console can put yours on standby during your own show. It takes effect at once, and can be turned back on the same way.

UDP or TCP
Settings: how cues reach each phone
Settings: how cues reach each phone

Every cue is broadcast on the cue port, as it always has been, and anything listening there hears it — an Isadora patch, a lighting desk, a phone that has not reported in yet. What Settings ▸ Cues reach each phone by decides is the second copy, the one aimed at each handset this console has actually heard from. UDP sends that copy as a datagram. It is what every venue so far has run on, and it is already a real improvement on broadcast alone: an access point sends a broadcast frame once and never retries it, while a unicast frame is retried until the handset acknowledges it. TCP opens a connection to each phone instead and keeps it open. The cue is acknowledged, cues arrive in the order they were sent, and when a connection breaks this console finds out rather than cueing into silence. It costs a connection per handset, and it reaches only phones that have reported in — which is every phone that has been delivered to. Either setting can be changed mid-show: the streams open or close on the next cue, and the broadcast never stops.

The full version

Text and the show cues are free, always. Sending files, addressing part of the audience and the delivery table work for five days, and afterwards need the full version — a one-off purchase, available at any time from the application menu. A lapsed trial never interrupts a performance already running: the basic cues keep working.

The network
The status line: what is being served, and from where
The status line: what is being served, and from where

The Mac and the phones must be on the same Wi-Fi. Cues are broadcast on UDP port 3000, phones report on 3001, and files are served over HTTP on port 8090 — all three changeable in Settings…. Only the cue port must also be changed on the phones — in the iOS Settings app, or on Android by tapping the network line on the idle screen seven times; the other two are announced to the phones in the cues. Phones learn where the Mac is from the cue itself, so nothing else needs configuring — but a guest network that blocks broadcast between devices will stop cues reaching the phones.

ReferenceEvery cue, and what can send it

The phones listen for OSC, so any OSC sender can drive them: the Augmented Theatre Dashboard for Mac and iPad, or a patch written in Isadora, for which the demo version is sufficient. The vocabulary is the same either way; what differs is what has to stand behind the cue. Delivering a file requires the Dashboard's content-addressed store and file server, which an OSC sender on its own does not have. Presenting a file that has already been delivered needs nothing extra: since protocol v1.8 a cue may name a file by its filename rather than its digest, so /atc/play/360 tarkovsky-rain.mp4 can be typed by hand. Text, colour, vibration, the torch, system sounds and the camera looks have always been a single broadcast packet that any sender can produce.

Every row an OSC sender can drive has a photograph of Isadora's OSC Multi Transmit actor configured for that row. Click a photograph to see it full size.

Three things apply to all of them. The destination is the broadcast address 255.255.255.255 on port 3000, which reaches every phone rather than one. Isadora sends every number as a float, which the phones accept wherever a number is expected. A value port holds numbers only: a filename, a show name or a line of text is supplied by a Text actor connected to that port.

The share is the argument most often entered wrongly. A value field accepts a decimal, but only with the separator the machine is set to use: where that is a comma, 0,5 is one half and 0.5 is read as 5. A phone looks for the share only among floating-point arguments, so a share sent as text is not read as a share at all and the cue reaches every phone. Produce it from an actor that outputs a fraction — a Calculator set to divide, as in the /atc/effect photograph, or a slider on the control panel.

One row needs more than a single trigger. /midi/state is a state, not an event: it has to keep arriving about three times a second or the phones let it expire and stop sounding. In a patch its transmit trigger belongs on a Pulse Generator, not on a key.

Two behaviours of that row are not faults. At a density of 0 each phone attacks its note once per arming and lets the envelope finish; repeating the same chord does not attack it again, because the attack is re-armed only when the set of pitches changes or /midi/1 arrives. And the broadcast leaves the sending machine by its default route, so a VPN running on that machine takes it: the patch keeps sending, a listener on that same machine still sees every packet, and no phone receives anything. Switch the VPN off, or send to the network's own directed broadcast — 192.168.1.255 on a /24 — instead of 255.255.255.255.

/midi/1 is sent once and is not repeated, and a broadcast frame on Wi-Fi is not acknowledged, so a dropped arm used to leave the phones silent with nothing to indicate it. From app version 4.51 a phone that did not receive /midi/1 arms on the first /midi/state it receives, so the arm does not need repeating. Do not drive the arming transmitter from a Pulse Generator: /midi/1 clears the chord a phone is holding, so every repeat makes each phone attack its note again on the next broadcast. Send it once, from the scene's enter trigger. Changing the density of a chord the phones are already holding has no effect on its own, because the attack is re-armed only when the set of pitches changes or /midi/1 arrives; send a new density together with a new chord.

OSC addressWhat it does Augmented Theatre
Dashboard
Isadora
(or any OSC sender)
Text
/text/1/text/1 "Tomorrow, and tomorrow" 40A string, and optionally a size: the surtitle every phone shows. The Dashboard sends one line at a time from the script pad. The line is drawn over whatever the phone is already showing — a film, a camera look, a shader — rather than replacing it; with nothing else playing it appears on black. An empty string clears it.✓✓/text/1 “Tomorrow, and tomorrow” 40 — OSC Multi Transmitpatch
/fontsize/1/fontsize/1 40A number: the size the text is drawn at.✓✓/fontsize/1 40 — OSC Multi Transmitpatch
The device itself
/color/1/color/1 212 92 36Three numbers, 0–255: fills every screen with one colour.✓✓/color/1 212 92 36 — OSC Multi Transmitpatch
/vibrate/1/vibrate/1 [length] [start] [loop] [strength]
/vibrate/1 2000 250 1 40
a two-second buzz, a quarter-second after the cue, repeating at 40% until stopped
Vibrates every device. Four optional numbers, in this order: length, how long one buzz lasts (default 500); start, how long after the cue it begins (default 0); loop, 1 to repeat until /vibrate/0, 0 for a single buzz (default 0); and strength, a percentage from 1 to 100 (default 100). Both times are in milliseconds and both are capped at 60000, so a mistyped cue cannot leave a device vibrating through a scene. Write them as whole numbers or as floats — Isadora sends every number as a float — and they are read the same either way. The four are positional, because numbers of the same type cannot be told apart by type. The optional share of the audience is still found among them by type, and is told from the four by its value: a float above 0 and below 1 is the share, and any other number is one of the four. That is why strength is a percentage and not a fraction. A device with a haptic engine applies all four. A device without one — an iPad, an older iPhone — vibrates for approximately the requested length at its only strength, and records that in its log. A length or strength of zero falls back to the default rather than producing silence, because a cue that does nothing is indistinguishable from one that never arrived.
✓✓/vibrate/1 2000 250 1 40 — OSC Multi Transmitpatch
/vibrate/0/vibrate/0Stops a buzz on every device, including one left repeating by /vibrate/1 with loop set. Takes no arguments, and ignores a share: a stop always clears every phone, because a buzz left running on part of the audience cannot be stopped afterwards.✓✓/vibrate/0 — OSC Multi Transmitpatch
/torch/1 · /torch/0Turns the torch on the back of every iPhone on and off. Android has no torch here and shows a white screen instead.✓✓/torch/1 — OSC Multi Transmitpatch
/systemsound/1/systemsound/1 1000A sound id: one of the sounds built into iOS, played at the device's ringer volume. iPhone only; Android ignores it.✓✓/systemsound/1 1000 — OSC Multi Transmitpatch
Camera looks
/atc/effect/atc/effect rain 1 0.5A look by name — rain, heat, dziga, rain3d, camera, mirror — on or off, with an optional share of the audience. One address for all six.✓✓/atc/effect rain 1 0.5 — OSC Multi Transmitpatch
/rain/1 · /heat/1 · /dziga/1 · /rain3d/1The same four looks, one address each: rain on glass, heat haze, scratched film, and rain falling in a sphere around the viewer. Kept for patches written before the address above.✓✓/rain/1 — OSC Multi Transmitpatch
/camera/1 · /camera/2 · /augment/1The back camera drawn to the screen, the front camera as a mirror, and look-around mode.✓✓/camera/1 — OSC Multi Transmitpatch
/rain/0 · /rain3d/0 · /heat/0 · /dziga/0 · /color/0 · /camera/0 · /augment/0Ends whatever look is running and returns the phone to the show. Every look that can be started by name can be stopped under that same name, so a patch does not have to know that one address ends all of them: a kino-eye started with /dziga/1 ends with /dziga/0, a colour fill with /color/0, a rain sphere with /rain3d/0. They do the same thing; the names are there so the stop is as easy to find as the start. Both cameras stop on /camera/0.✓✓/dziga/0 — OSC Multi Transmitpatch
Whole-audience state
/title/1/title/1 "Act One, Scene Two"A string for the top bar of every phone: the act, the scene, or the name of the piece. iPhone shows it; Android ignores it. The Dashboard sends it from a Title row, which holds its own text.✓✓/title/1 “Act One, Scene Two” — OSC Multi Transmitpatch
/volume/1/volume/1 0.35A float from 0 to 1: how loud clips and sounds play on every device. It is a level rather than a cue — nothing depends on its having been sent, and it applies to whatever is playing now as well as to whatever plays next, so it can be changed under a running clip. The Dashboard sends it from a Volume row. A spatialized 3D voice does not respond to it.✓✓/volume/1 0.35 — OSC Multi Transmitpatch
/start/1 · /stop/1/start/1 "cherry orchard"Start takes the show's name and an optional title line: the phone stops whatever is playing, switches to that show's files, displays Poster.jpg and marks itself as in the show, which suppresses the idle screen's own controls. Stop takes no arguments: the phone stops whatever is playing, displays the foyer logo, clears the title line and any surtitle, and marks itself as out of the show. A third argument to Start, the sender's address, is read on iPhone and ignored on Android. The Dashboard sends both from a House row.✓✓/start/1 “cherry orchard” — OSC Multi Transmitpatch
Delivered files
/atc/manifest/atc/manifest 8090 "cherry orchard" 3001Tells the phones the show has changed: fetch the manifest from the sender and pre-stage anything not already held. Carries the file server's port, the show name, and the port to report on.✓
/atc/asset/atc/asset 8ccfabc0…5954 4096000 8090 video/mp4 rain.mp4One file by digest, with its size, MIME type, name, show, layout and 3D-audio flag: fetch it and present it.✓
/atc/play · /atc/play/flat · /atc/play/360 · /atc/play/displace · /atc/pause · /atc/stop/atc/play/360 tarkovsky-rain.mp4 1Transport for a delivered clip or sound, named by digest or by filename; play carries the loop flag and the share. The three suffixed forms state the mode directly — flat, a 360° sphere, or the displacement look over the live camera — so playing a 360° clip is a single cue rather than a layout change that the next cue depends on.✓✓/atc/play/360 tarkovsky-rain.mp4 1 — OSC Multi Transmitpatch
/atc/show · /atc/hide/atc/show poster.jpgThe same two operations for a still image.✓✓/atc/show poster.jpg — OSC Multi Transmitpatch
/atc/layout/atc/layout tarkovsky-rain.mp4 360Changes the type of a held clip without presenting it. To play a clip in a given mode, use the /atc/play/… address above, which is a single cue.✓✓/atc/layout tarkovsky-rain.mp4 360 — OSC Multi Transmitpatch
/atc/loop/atc/loop tarkovsky-rain.mp4 1Turns looping on or off during playback, for a file or a whole AR set, without restarting it.✓✓/atc/loop tarkovsky-rain.mp4 1 — OSC Multi Transmitpatch
/atc/hrtfpos/atc/hrtfpos voice.ogg 1.5 0 -2Moves a spatialized voice around the listener, in metres right, up and forward, during playback.✓✓/atc/hrtfpos voice.ogg 1.5 0 -2 — OSC Multi Transmitpatch
/atc/raise · /atc/strike/atc/raise three-sistersRaises an AR set on its anchor, and strikes it.✓✓/atc/raise three-sisters — OSC Multi Transmitpatch
/atc/audioreact/atc/audioreact 1 breathing-room.fragA shader by digest or filename, on or off. It is driven by the device's own microphone rather than by anything sent over the network, so the same shader renders differently on each device. The shader receives the elapsed time, that device's sound level, the viewport size, and the live camera if it declared one. Supported on iPhone and Android. Starting one stops whatever is playing, and any cue carrying sound or video ends it, because a device cannot measure a room it is playing into.✓✓/atc/audioreact 1 breathing-room.frag — OSC Multi Transmitpatch
/midi/1 · /midi/0/midi/1Arms and disarms the ensemble. Arming makes every phone allocate its audio and begin listening; it sounds nothing by itself, because starting an audio engine on a few hundred phones takes long enough that a chord sent with it would be late. /midi/0 silences every phone and is sent three times, since the repeats that would correct a loss have just stopped.✓✓/midi/1 — OSC Multi Transmitpatch
/midi/state/midi/state 7 4000 110 0 60 64 67
revision 7, good for 4 seconds, gain 110, attacked once, a C major triad, every phone
The chord being held, repeated about three times a second for as long as the row is running. There are no note-on and note-off messages: the state is the message, so a lost packet costs a late chord rather than a phone sounding indefinitely. The arguments are the revision, which counts up and lets a phone ignore anything older than it holds; the ttl in milliseconds, after which a phone that has stopped receiving falls silent by itself; the gain, 0 to 127; the density, re-attacks per minute per phone, 0 for one attack and no repeat; then the MIDI note numbers of the chord, up to sixteen. The share is found among them by type as usual, but is not re-rolled per message: a phone decides once, from its own identifier, and keeps that answer for the passage. Append it only when it is below 1: a share is a decimal strictly between 0 and 1, so a trailing 1.0 is not read as every phone but as one more note, the lowest the protocol has. Every phone is the absence of the argument. Each phone sounds one note of the chord, also chosen from its own identifier. No pitches at all is a rest, not a fault.
✓✓/midi/state 7 4000 110 0 60 64 67 — OSC Multi Transmitpatch
/atc/inventory/atc/inventory "cherry orchard"Asks every device what it holds for this show; devices answer on the telemetry port. The Dashboard's Refresh sends this and the manifest together, so a device that is missing files starts fetching them as it is counted.✓
/atc/discard/atc/discard poster.jpgDeletes one file, wherever it is stored.✓✓/atc/discard poster.jpg — OSC Multi Transmitpatch
/atc/clearshow/atc/clearshow "cherry orchard"Deletes everything held for one production, for use when a run ends.✓✓/atc/clearshow “cherry orchard” — OSC Multi Transmitpatch
Diagnostics
/debug/1 · /debug/0/debug/1Puts every cue a phone accepts on that phone's own screen, over whatever it is already showing — a film, a look, a shader, the foyer. The line reads the phone's clock, the address and the arguments as they arrived, and the first line is the phone's own IP and port, so one phone can be told from another. A cue that repeats, like /midi/state three times a second, is collapsed into one line that counts up rather than filling the screen. Twelve cues are kept; the oldest falls off the top. /debug/0 takes it away. Neither takes an argument, neither reads a share, and neither puts a phone into the show or out of it: asking a phone what it is receiving must not change what it is doing. For technical setup and rehearsal; with an audience present, leave it off.✓✓
Older addresses the apps still accept
/heartbeat/1Keeps late arrivals and restarted apps in show mode.✓/heartbeat/1 — OSC Multi Transmitpatch
/speaker/1 · /headphones/1 · /stream/1Audio routing and streaming. iPhones only.✓/speaker/1 — OSC Multi Transmitpatch
/cloudvideo/1 · /cloudaudio/1 · /cloudpicture/1 · /cloudanimatedgif/1 · /cloudalphacam/1From the Firebase delivery mechanism: play a file from the app's documents folder by name. Both apps still parse these, but the background sync that filled that folder no longer runs, so in a current build there is nothing for them to play. Superseded by /atc/manifest and /atc/asset, which deliver over the venue's own Wi-Fi and need no internet connection.

Since protocol v1.8 these cues accept a filename as readily as a digest, so /atc/play/360 tarkovsky-rain.mp4 can be written by hand. To copy one instead of typing it, right-click a row in the Dashboard's list — long-press on iPad — and choose Copy OSC Address. It copies the filename, falling back to the digest only when the show holds two files with the same name. The copied cue omits the share, the loop flag and the show name: those come from the Dashboard's own controls at the moment of copying rather than from the row.

The two MIDI rows need more than a table row holds: why the chord has to keep arriving, what a note-off is and why this protocol has none, and what the revision decides when two senders are sending at once. That is under MIDI in the bar at the top.

Cues are broadcast on UDP port 3000, and the phones take the sender's address from the packet itself. The last group is accepted by the apps but sent by no Dashboard control, so it is reachable only from a patch written by hand. The cloud addresses in that group have nothing behind them in a current build.

For a version of the app branded for your production, write to or the Facebook page.

MIDIOne chord, divided across the audience's phones

The ensemble cues are the only ones that carry a state rather than an event. The console broadcasts the set of pitches sounding now, about three times a second; each phone selects one pitch from that set and sounds it. No cue tells a phone to stop a note. Everything below follows from that, including why a patch has to keep sending the same chord.

One note, from key press to release A timeline in three lanes: Isadora, the network, and one phone. Isadora UDP 3000 each phone /midi/state · 3/s /midi/1 console stops sending 0 1 s 2 s +4 s 1 2 3 4 5 6
One note, from key press to release. Time runs left to right; the dashed section is not to scale.
  1. 1The operator presses transmit on the Isadora actor that sends /midi/1. One UDP packet is sent. It is not repeated.
  2. 2The packet is a UDP broadcast to port 3000, so every phone on the network receives it at once. There is no acknowledgement, and a phone that does not receive it reports nothing. From version 4.51 a phone that did not receive /midi/1 arms on the first /midi/state it receives instead, about 0.3 s later.
  3. 3On /midi/1 each phone starts its audio engine and discards any chord it is holding. Arming produces no sound. Starting the audio engine takes long enough that a chord sent in the same packet would sound late, so arming and playing are separate cues.
  4. 4The first /midi/state arrives about 0.3 s later. The phone holds no chord, so the set of pitches counts as changed and the phone attacks its pitch once. At a density of 0 it does not attack again. Each phone derives which pitch it takes from its own install identifier, so the pitches of a chord are distributed across the audience, and a chord of one pitch is sounded by every phone.
  5. 5The console repeats /midi/state about three times a second with the same revision number. A repeat does not cause an attack. It resets the expiry timer, and it is the only indication that the console is still sending.
  6. 6When the console stops sending, no cue announces it. Each phone releases its note when the ttl expires — four seconds in this example. A lost packet therefore cannot leave a note sounding.

Driving the ensemble from the Dashboard

The Dashboard's keyboard window with the house armed: four octaves of keys, the MIDI source line under them, and sliders for density, gain and share
The keyboard window, with a MIDI row started. The line under the keys names the MIDI source the console is receiving from — here, a patch sending to the Mac's IAC bus.

The ensemble is driven either by the Dashboard or by an OSC sender such as Isadora. To use the Dashboard: add a MIDI row from the cue strip and press Start. Start sends /midi/1, which starts the audio engine on every phone, and opens the keyboard window.

Play the keys with the mouse or the computer keyboard; the letter layout is GarageBand's Musical Typing, and z and x change octave. A MIDI keyboard connected to the Mac over USB or Bluetooth is used without configuration, as is any application sending to the IAC bus. The line under the keys names the MIDI source in use, or states that there is none; read it first when a key produces no sound.

Three sliders set three fields of /midi/state: density, the number of re-attacks per minute per phone, which at its left limit is 0 and gives a single attack; gain, 0 to 127; and share, the fraction of phones that participate. The line above the sliders shows the chord being broadcast and its revision number.

From version 4.51 density, gain and share can each be bound to a knob or fader on the connected MIDI keyboard. Press the dial beside the number, then move the control; the binding is stored with the show. A bound control does not change the value until the control reaches the value currently set, so moving a fader that rests at its minimum does not drop the value to its minimum. A binding records which instrument it was learned from and reports a mismatch when a different one is connected.

The Dashboard performs the whole protocol: it repeats /midi/state about three times a second while the row is running, increments the revision when the chord changes, and sends /midi/0 on Stop. The sections below are needed only to drive the ensemble from an OSC sender.

Driving the ensemble from an Isadora patch

The timeline above is one note. This patch sends eight: the first line of Twinkle Twinkle Little Star, from Isadora alone, with no Dashboard on the network.

Ten actors. A Pulse Generator at four slots a second sets the slot rate. A Counter wrapping 1 to 16 selects the slot. A Value Select holds the value each slot sends: the note number, then 128 for the rest that follows it. A second Counter supplies the revision, incremented on every slot. A Javascript actor converts the note number to a name for display. Each actor carries a comment stating what it does and what to change.

A rest is sent as pitch 128. 128 is outside the MIDI range 0–127, so the phone reads an empty chord. That is what makes a repeated note attack a second time: the set of pitches differs between the two notes. Sending /midi/1 before each note produces the same attack, but /midi/1 is an event, sent once with no repeat, and on the network this was measured on more than half of them were lost.

The second transmitter in the patch sends /midi/1 every twenty seconds. From app version 4.51 it is not needed: a phone that did not receive the first /midi/1 arms on the first /midi/state it receives. It also has a cost. /midi/1 clears the chord a phone is holding, so every repeat makes each phone attack its pitch again on the next slot. In this patch that is not audible, because the note and the rest change the chord on every slot in any case; under a sustained chord it produces an attack at the repeat rate. Stop Pulse Generator 2 and send /midi/1 once, at the start. Do not delete the actor: the single arm starts the audio engines before the first note, and it is also what removes the foyer logo. A phone that armed itself from a state cue sounds the chord with the logo still on screen.

Below is three bars of the line, recorded from one phone.

⇩ twinkle-midi.izz — the Isadora patch, 20 kB
Isadora 4; the demo version runs it. It broadcasts to 255.255.255.255 on port 3000, so the phones have to be on the same network as the Mac.


Three bars as one handset plays them. Rendered with the app’s own synthesis — a sine per note, the 2.2-second decay and the eight-voice bank from EnsembleVoices.cpp — not recorded in a room.

Every note number

A chord is sent as MIDI note numbers, not note names. The grid below lists all of them: read down for the note, across for the octave. 60 is middle C, marked. The apps and the Dashboard keyboard call it C4, following scientific pitch notation; Logic and GarageBand call the same key C3. The number is what is transmitted, so where a DAW and this grid disagree on the name, the number is correct.

Valid numbers are 0 to 127. Any value outside that range is read as no note, which is how a rest is sent: the patch sends 128 and the phone reads an empty chord. Because the set of pitches has changed, the note after the rest attacks again even when it is the same note. A chord carries up to sixteen numbers, and each phone sounds one of them, selected from its own install identifier.

-10123456789
C01224364860728496108120
C#11325374961738597109121
D21426385062748698110122
D#31527395163758799111123
E416284052647688100112124
F517294153657789101113125
F#618304254667890102114126
G719314355677991103115127
G#820324456688092104116
A921334557698193105117
A#1022344658708294106118
B1123354759718395107119

Why the chord repeats

In MIDI a note is two messages. Note-on carries the note number and the velocity and is sent when the key is pressed; note-off is sent when it is released. Neither carries a duration: the duration is the interval between them. On a cable both arrive. On a wireless network either can be lost, and the two losses are not equivalent — a lost note-on means one note does not sound, while a lost note-off leaves a phone sounding a note with no message remaining that would end it.

This app sends neither. The console does not send start this note and later stop it. About three times a second it sends the set of pitches sounding now, and each phone generates from that. The repeat is not a retransmission of something that may have been lost; the repeat is the message.

A phone left sounding a note cannot be stopped until the app is force-quit, and the app runs on venue networks that cannot be tested before the performance. Correctness therefore must not depend on packet delivery. Every look the app can start has a stop addressed by the name that started it, and the ensemble avoids the stuck note entirely by not sending note-off at all.

A lost packet costs a late chord change: never a wrong note, and never a note that will not stop. The next broadcast corrects it about 0.3 s later. A phone that joins late, recovers from a stall or returns from the background converges on the current state by itself, with no retransmission, no acknowledgement and no session to re-establish. Rhythm is generated on the phone and is unaffected by the network; only agreement between phones is affected, and only until the next broadcast.

The repeats are also the evidence that the console is still sending, which is what the ttl depends on. A phone that stops receiving releases its note when the ttl expires, rather than holding the last chord indefinitely. A repeat carrying a revision the phone already holds is treated as a keep-alive and resets the expiry. The first implementation rejected such repeats, and a held chord expired after its ttl while the Dashboard was still broadcasting it three times a second.

/midi/1 and /midi/0 are the two ensemble cues that are still events, sent once rather than repeated. Until version 4.51 a lost /midi/1 left a phone discarding three state cues a second for the rest of the piece. From 4.51 the state cue arms the phone instead. Two conditions stop that from undoing a stop: a phone that has received /midi/0 ignores state cues for 1.5 s afterwards, which is longer than any packet still in flight when the stop was sent; and a phone that has not yet established its identity cannot arm, because it could not select a pitch. /midi/0 is still an event, which is why it is sent three times: the repeats that would correct a loss have just stopped.

What the revision is for

The first argument of /midi/state is a counter. It states only which of two states is newer. A patch has no counter of its own: whatever stands in value 1 is sent unchanged, three times a second, for as long as the patch runs. The Dashboard does count: its Keyboard window starts at 0 and adds one each time the chord changes, never on a repeat.

A phone compares the incoming revision with the one it holds. Higher is a new state: accept it, and attack if the set of pitches has changed. Equal is the same chord sent again: do not attack, reset the expiry. Lower is older and is ignored, unless the held state has already expired, which is how a console that restarted and began counting from 0 is accepted again. This also makes reordered and duplicated UDP harmless: a packet that arrives late carries a lower number and is dropped. The protocol has no sequencing and no acknowledgement.

The phones follow whichever sender holds the highest revision, and two senders cannot share them. A Dashboard counting in the hundreds overrides a patch sending 8 on every packet. If that Dashboard holds no pitches — a rest — the phones are silenced rather than played over, and the patch cannot regain them. Stop one sender before using the other. Within a patch: increment the revision when the chord changes, and leave it unchanged while it does not.

The kitBuild a show and run it without the Dashboard

A public repository holding the scripts behind the demo show: one that generates the show’s media, one that builds the AR set, and a single-file control program that delivers the show to the phones and sends the cues from a terminal. MIT licensed, with no dependencies beyond Python 3. None of the media is sampled from anywhere — every file is produced by a script in the repository, which is what makes it publishable and rebuildable by anyone.

↗ github.com/eighteight/augmented-theatre-kit

Running it

You need the audience app on a phone, the phone and the computer on the same Wi-Fi network, and Python 3. The first command requires ffmpeg. Nothing else is needed, because the AR set is committed to the repository already built.

git clone https://github.com/eighteight/augmented-theatre-kit
cd augmented-theatre-kit
python3 demo-show/make-demo-show.py                   # the media, and the AR set
python3 drive/atc-drive.py demo-show/show --deliver   # walk-in: onto the phones
python3 drive/atc-drive.py demo-show/show --run       # curtain: run the show

Delivery and running are separate commands because they happen at different times: the show is copied onto the phones while the audience takes its seats, over several minutes, and the cues are sent during the performance. Either can be run again on its own — deliver again for a phone that arrived late, run again for a second performance. With neither flag, the script does one and then the other, which is useful at a desk.

The delivery wait is deliberate, and the script reports it as it goes. Each phone waits a random interval of up to 30 seconds before requesting the manifest, and up to 2 seconds more before each file, so that a few hundred phones do not request everything at once. No cue is sent until every phone that answered reports holding the whole show: a set raised before its files have arrived does nothing, and does so silently, which is indistinguishable from a broken cue.

When the set is raised, the drawing it stands on — this anchor, the same one the Dashboard’s demo uses — is opened on your own screen, so a phone can find it with nothing printed. The set is scaled against the width of that drawing: measure the picture on the screen and say what you measured — --marker-cm=14.5 — or print it at 21 cm and leave it alone.

Generating the show’s media

demo-show/make-demo-show.py writes a still image, an animated image, a flat film, an equirectangular 360 film, a chime and a spoken line — one of each kind the app can present, so that every cue path is exercised — and copies the demo AR set in beside them. The PNG is written byte by byte in pure Python; the rest uses ffmpeg. It takes a few seconds.

None of it is sampled from existing material. Media of unknown origin cannot be published with a repository, and generating it is cheaper than clearing the rights to it.

A control program in one file

drive/atc-drive.py does what the console does, in about 700 lines and with no libraries. It serves the show over HTTP with Range resume, sends /atc/manifest, listens on the telemetry port and prints what each handset holds and is still fetching, waits for the phones, then sends every cue in order. It announces each cue on the phones as a title line, so the demo can be followed from a handset rather than from the terminal.

None of it is simulated. It sends the same OSC and serves the same HTTP as the Dashboard, including the .atcset document naming which drawing, which model and which film on which surface, so a handset cannot distinguish the two. It is useful for learning the protocol, for testing a build, and for a small production that does not need a console.

python3 drive/atc-drive.py demo-show/show --deliver   walk-in
python3 drive/atc-drive.py demo-show/show --run       curtain
python3 drive/atc-drive.py demo-show/show             both, in one go
python3 drive/atc-drive.py demo-show/show --listen    watch the phones
python3 drive/atc-drive.py demo-show/show --raise     the AR set, alone

--deliver reports what it actually achieved: a phone that never answered has not been delivered to, and the output says so. --run asks the phones what they hold, prints the answer, and then sends the cues without waiting.

The AR set, and the scripts that build it

ar-set/ holds the demo set itself — the anchor, the rigged figure, two surface films, the eight-bit score and ar-set.json — with the two Blender scripts that build it and the prompt that rebuilds the whole thing from a description. Creating an AR set in the manual says what each piece is for and which four rules decide whether a set works on a phone.

It uses the same drawing as the Dashboard’s demo — this one, printed 21 cm square — so one sheet serves both. This matters because a phone looks for the image named in the set it was given. A set built with a different drawing will never be found on the sheet in your hands, and it fails silently: the cue is sent, the camera opens, and nothing appears. make-ar-set.py --new-anchor draws a fresh anchor for a set of your own; print that one instead.

Curtain callFirst production

The app was first used in production at the Meyerhold Theatre Center, Moscow, on 4 September 2016. Director Victor Ryzhakov · music director Tatyana Pykhonina · video artist and interactive media programmer Vladimir Gusev.

Vladimir Gusev

Theatre and art videographer and media director, based in New York. His interactive productions in the US, Europe and Russia combine video, sound, light, generative graphics and audience interaction.

Meyerhold Theatre Center

A theatre laboratory in Moscow working in production, experiment and education.

The apps

iOS, Android and the Dashboard for Mac and iPad, rebuilt in 2026 for current devices. The app is not tied to one production: any show can brand it and drive it.

MiscellaneousTools for testing and troubleshooting

Not used to run a performance; used to build and check one.

Testing the app without a Dashboard

Six taps on the idle logo, each within two seconds of the last, open a hidden menu of buttons that run the same handlers the cues above run — a surtitle, a colour fill, a vibration, the torch, each camera look, and entering and leaving the show — with no Dashboard on the network. Each button calls the real handler rather than a stand-in, so a shader, a capture or a network call happens exactly as it would from a cue. It is present in every build, including alpha and TestFlight, because that is where it is useful: checking a build on a device with nothing else to talk to. Close returns to the idle screen and re-arms the gesture.

Firing cues from Isadora, step by step
Put the device (running Augmented Theatre) and the computer on the same Wi-Fi network.
Download and install a demo version of Isadora from TroikaTronix (Windows & macOS).
Download the demo patch and its companion text file, then open the patch in Isadora:
⇩ loadjs.izz — Isadora patch ⇩ waitingforgodot.txt
Open the scenes left to right; the comments in each list the keyboard actions that fire cues. Press them and watch the device answer.

The broadcast port can be changed in the app's settings, which allows technicians to use a port separate from the audience's.

What the phones report back (UDP port 3001)
/atc/progressDevice id, digest, bytes received, bytes total. Sent while a file is downloading and once more when it completes. This is what fills the Dashboard's delivery counts.
/atc/holdingDevice id, show, chunk number, and the digests held. The answer to /atc/inventory. A device holding nothing still answers, which is how an empty phone is distinguished from an absent one.
/atc/shader/errorDevice id, the shader's name, and the compiler's error text. Sent when an audio-reactive shader fails to compile on that device. The Dashboard checks every shader against a GLSL compiler before sending, so this means one handset disagreed. The device is left as it was and the look does not appear there; without this message the operator would not know which phone was affected.
/location/1About once per second while Bluetooth is on: the nearest proximity beacon's name (or NONE), a device identifier, and the device's IP address.
/devicestate/1On app state change: the state (startup, foreground, background or shutdown), a device identifier, and the device's IP address.