the augmented theatre

The audience is the stage.

Augmented Theatre turns every phone in the house into part of the stage design. Text, sound, video, light, vibration — cued live from the show's media server, in perfect sync with the performance.

Download on the App Store Get it on Google Play
AUGMENTED THEATRE SHOW · 2016 · PHOTO — LEONID SELEMENEV
AudienceAugmented Theatre App

The audience downloads it before the doors open, and then forgets it: there is no account to make, nothing to configure, and nothing to press. From the moment the show begins, every phone in the house answers the booth at once — a hundred screens, a hundred speakers and a hundred small lights that the director can play like any other instrument on stage.

A phone held in the audience showing the stage through the app's rain
            effect, the actors dissolving behind running water
Suddenly a distant sound is heard as if from the sky,
the sound of a string breaking, dying away The Cherry Orchard
WordsSurtitles, in any languageA line appears in every hand at the moment it is spoken — the translation, the letter being read aloud, the thought nobody says. As large or as quiet as the scene needs.
PicturesFilm and stills, togetherVideo and photographs play on every phone at once, started by a single cue and running in step across the room.
LightThe house becomes the lighting rigA hundred screens fill with one colour, or a hundred torches come up together — an effect no lantern in the grid can make, because it happens inside the audience rather than above it.
TouchSomething felt, not watchedThe show reaches into the audience's hands: a pulse timed to a heartbeat, a knock, a tremor under the seats.
The roomThe theatre through the cameraPhones raised, the venue itself is seen through rain running down the glass, through heat rising off the stage, through scratched film — or turned around, so the audience is looking at itself.
SceneryA set that stands on a drawingThe audience holds a printed image, points a phone at it, and the scenery rises off the paper in three dimensions with video playing on its surfaces.
VoicesSound that has a place in the roomWith headphones, a voice can be set down beside a listener and moved around them mid-sentence — behind the shoulder, across the room, close enough to flinch from.
Before curtainEverything arrives during the walk-inThe evening's media reaches every phone quietly while the audience finds its seats, over the venue's own Wi-Fi — no internet, no waiting, and nothing on screen until the show wants it.

Free on the App Store and Google Play. It carries no advertising, asks for no account, and works entirely on the venue's own network — a phone that walks out of the building goes back to being a phone.

The boothAugmented Theatre Dashboard for Mac
Download on the Mac App Store

The whole booth in one window. Paste the evening's script and walk it line by line; stage the show's media during walk-in and watch the audience load it; build a running order where effects and files are equals — the same clip flat in act one, wrapped around the viewer in act two. Built for the five minutes before curtain, on the venue's own Wi-Fi — no internet required.

The Augmented Theatre Dashboard: the script pad holding the Macbeth soliloquy with the two Send buttons that walk down or back up it, the effect-chip card with the whole cue vocabulary, the show timeline — the same clip as flat video and as a 360° sphere, a colour-screen instance, rain, a positioned 3D voice — each row carrying its own settings and a live count of the phones holding it, and at the foot the second dashboard on the network, offered the show
SHIFT-RETURNThe script padPaste the whole evening's surtitles. Each press sends one line to every phone and steps to the next; the line the house is reading stays marked. ⌘-Return steps back when the show retreats on you.
CUSTOMDelivery during walk-inPictures, video and sound reach every phone quietly before curtain; a phone never fetches what it already holds.
THE LISTA timeline of instancesMedia and effects side by side, dragged into running order, duplicated with ⌘D — and every instance carries its own settings: the same clip can play flat, wrap the viewer as a 360° sphere, or drive a camera effect.
TWELVE CHIPSThe effect vocabularyRain on the glass, heat shimmer, scratched film, rain falling in a sphere, the live camera and the mirror, a colour screen, a vibration, the torch, a system sound — each chip adds an instance to the show.
3DA voice around the listenerBinaural audio placed anywhere in the room and moved live with a position pad — even mid-line.
DROP A FOLDERAR sets, deliveredThe designer's folder — anchor, model, projections — validated, bound and sent to the house as one thing, raised and struck on cue.
START · PAUSE · STOPOne word eachStart plays a clip, shows a picture, fires an effect, raises an AR set — whatever the row is. Stop takes it away.
UPDATE ASSETSThe delivery tallyHow many phones hold each file, and how far the rest have got — live, with the in-flight percentage.
0 – 100%Part of the houseAny cue can address a fraction of the audience rather than all of it.

Free to download, with a demo show staged on first launch. Text and the basic cues are free forever; sending files, audience targeting and the delivery tally run free for five days and are included in the full version — a one-time purchase. Requires macOS 11 or later.

The prompt bookThe Dashboard manual

The same help that ships in the app — every section of it. Also available in Russian, German, Spanish and French from the app's Help menu.

What this app does

The Dashboard is the operator's console. From the booth it sends cues over the venue's own Wi-Fi to the free Augmented Theatre app running on the audience's phones: lines of text, and the images, video and sound a production needs. There is no setup on the phones and no pairing — a device that hears a cue obeys it.

Shows

File ▸ New Show… asks where your shows live, then makes one. A show is an ordinary folder: a document with the show's name, and a media folder holding everything sent to the audience for it. File ▸ Open Show… reopens one, and Reveal Show in Finder puts it in front of you. Clearing a production once its run has ended is deleting that folder.

Text cues

Type in the large field and press Return to put that line on every phone in the house. The slider under it sets the size the text is drawn at. Pulse the text size makes the line breathe rather than sit still — press it again to stop.

Sending files

Send Files… stages images, video or audio for the show that is open. Only file types the phones can present are offered. Files are copied into the show's media folder and announced to the house, and the phones fetch them quietly in the background — during walk-in, ideally, so that a cue at curtain is instant. A phone never fetches a file it already holds.

AR stage sets

An AR set is a piece of scenery the audience raises in their phones over something real. The phone recognises an anchor — a printed drawing in their hands, an object, the stage itself — and the set stands on it, with video playing on its surfaces. Drop the AR set's folder onto the delivery table to import it — the Dashboard checks it, delivers it to the house like any other file, and the set appears as one row. Start raises it on the phones; Stop strikes it.

Building an AR set, step by step

1 · The anchor. The AR set is raised over something real, and the anchor image is how the phone recognises it. It can be a drawing printed and handed to the audience, a photograph of an object placed in the venue, or a photograph of the stage itself, taken from where the audience will stand. Any image works, but give it detail and contrast: at least 640 pixels across, and visually distinct from every other set's anchor in the show — two similar anchors make the phones guess which AR set to raise. Measure the real thing now: the width the anchor has in the venue (a printed A4 sheet is 0.21 m — the default) is what sets the scale of everything.

2 · The model. Build the AR set in Blender (or any tool that exports glTF), in real metres, in Blender's own Z-up axes. Imagine the anchor lying flat at the origin: the set stands on and around it, facing −Y — towards the viewer. An AR set anchored to a sheet in the audience's hands is small by design; one anchored to the stage is built at stage size. Name every surface that will carry video (screen-left, backdrop…) and UV-unwrap those surfaces — video lands through texture coordinates, and a mesh without them shows nothing.

3 · The projections. For each video surface, tell the AR set which file goes there. The cleanest way is in the model itself: add a custom property named video to the surface's object in Blender, with the video's filename as its value. Or skip properties and simply name each video file after its surface (screen-left.mp4). Or say it in ar-set.json — see below.

4 · Export. glTF Binary (.glb), one file — a .gltf with separate textures is refused. Two export settings matter: untick “+Y Up” (the phones expect the file Z-up, exactly as Blender has it, and do the turn themselves), and tick Include ▸ Custom Properties if the video bindings live in the model.

5 · The folder. One folder, named after the AR set: the anchor image, the .glb, the videos, and optionally a ar-set.json. Drop it on the Dashboard's delivery table. The import sheet checks everything — the anchor's size, the model's surfaces, every binding — and says plainly what is missing or ambiguous; nothing is sent until the errors are gone. Then Send, and the AR set travels to the house like any other file. Start raises it; Stop strikes it.

⇩ starter-set.zip — a complete working AR set with a README
The demo show's AR set: anchor, metre-scale .glb with two named screens, two videos and a fully spelled-out ar-set.json. Take it apart and replace every piece with your own.

Preparing an AR set folder

One folder holds one AR set. It needs the anchor image — a drawing to be printed and handed out, or a photograph of the object or stage the set will stand on (JPEG or PNG, at least 640 pixels across; detailed, high-contrast views track best) — and the set itself as a single .glb, authored in metres (a .gltf with separate textures will be refused). Videos beside them (.mp4, .mov, .m4v) are projected onto the model's surfaces: say which goes where in the model's own node properties (a custom property named “video” in Blender), in a ar-set.json beside it, or simply by naming the video after the surface. An optional ar-set.json can also name the anchor and model among other files and give two numbers: “widthMeters”, the real-world width of the anchor as the audience sees it (0.21 — an A4 sheet — if unsaid), and “scale”, a correction for models not authored in metres (1 if unsaid). Both can also be adjusted in the import sheet. Keep an AR set well under 60 MB — every phone pulls all of it during walk-in.

The ar-set.json companion file

A small JSON file beside the model, for what the files alone cannot say. Every key is optional — a folder with one obvious drawing, one .glb and self-naming videos needs no ar-set.json at all. The folder's name becomes the AR set's id and title.

{
  "marker":      "three-sisters.jpg",
  "model":       "stage.glb",
  "widthMeters": 0.297,
  "scale":       1,
  "projections": {
    "screen-left":  "clouds-left.mp4",
    "screen-right": "clouds-right.mp4"
  }
}

marker — which image is the anchor, when the folder holds more than one. Without it, the first image alphabetically is used, with a warning.

model — which .glb is the AR set, when there are several.

widthMeters — the width the anchor really has in the venue, in metres: 0.21 (the default) for a printed A4 sheet, 0.297 for A3, or the measured width of the object or stage view photographed. The whole AR set scales from this one number — the same image printed at two sizes is two different anchors, so measure the real thing.

scale — a correction for models not authored in metres: the multiplier that brings one model unit to one metre. 1 (the default) for a correctly authored file.

projections — surface name → video filename, for models whose bindings are not in the file itself. Bindings written in the model's own custom properties win over ar-set.json; a video named after its surface is the fallback when neither says.

Delivered to the audience

Loaded counts the phones that hold the whole file. In flight counts those still fetching it, with how far they have got on average. A dot beside a name means the audience is looking at that file right now. Phones report as they download; one that arrived with the file already cached from an earlier session simply never reports, so this table is a floor, not a headcount.

Play, Pause, Stop, Show, Hide

Select a delivered file and use these to put it on the audience's screens. Play, Pause and Stop are for video and sound; Show and Hide are for stills. Stop returns to the start and leaves the screen as it was before. None of these touch the network beyond one small packet, so they are safe to fire mid-scene.

Part of the audience

The slider beside the transport decides how much of the house a cue addresses. Every phone rolls for itself, so at 40% roughly two in five obey — not an exact count, and a different two in five each time you fire. At the top of the slider the cue is for everyone.

The playhead

While video or sound is playing, the bar shows about how far in the audience is. The phones do not report their position, so this is the Dashboard's own clock, started when the cue went out. The ± figure is the spread across the house: one broadcast starts every phone at once, and what remains is the few hundred milliseconds of network and player start, widening very slowly as handset clocks drift apart.

The full version

Text and the show cues are free, always. Sending files, addressing part of the audience and the delivery table work for five days, and afterwards need the full version — a one-off purchase, available at any time from the application menu. A lapsed trial never interrupts a performance already running: the basic cues keep working.

The network

The Mac and the phones must be on the same Wi-Fi. Cues are broadcast on UDP port 3000, phones report on 3001, and files are served over HTTP on port 8090 — all three changeable in the Dashboard's Settings ▸ Ports…. Only the cue port must also be changed on the phones (iOS Settings; on Android, tap the network line on the idle screen seven times — the same guard Android uses for its own developer options); the other two are announced to the house in the cues. Phones learn where the Mac is from the cue itself, so nothing else needs configuring — but a guest network that blocks broadcast between devices will stop cues reaching the house.

ReferenceEvery cue, and who can send it

The phones answer OSC, so anything that speaks OSC can drive them — the Augmented Theatre Dashboard for Mac and iPad, or a patch you wire yourself in Isadora, for which a demo version is enough. The difference is not the vocabulary but what stands behind it: the cues that name a file are answered from the Dashboard’s own content-addressed store and file server, and an OSC sender with no store behind it has nothing to name and nothing to serve. Everything else — text, colour, vibration, the torch, sounds, and the six camera looks — is one broadcast packet, and any sender can fire it.

OSC addressWhat it does Augmented Theatre
Dashboard
Isadora
(or any OSC sender)
Words on every screen
/text/1A string, and optionally a size — the surtitle every phone shows. The Dashboard sends it a line at a time from the script pad.
/fontsize/1A number — how large that text is drawn.
The audience's own device
/color/1Three numbers, 0–255 — fill every screen with one colour.
/vibrate/1Buzz every device. Android reads an optional length in milliseconds; iPhones give their standard tap. Android also answers /vibrate/0.
/torch/1 · /torch/0The torch on the back of every iPhone, on and off. Android has no torch here and shows a white screen instead.
/systemsound/1A sound id — one of the system sounds. iPhones only; Android does nothing.
Looks over the phone's own camera
/atc/effectA look by name — rain, heat, dziga, rain3d, camera, mirror — on or off, with an optional share of the house. One verb for all six.
/rain/1 · /heat/1 · /dziga/1 · /rain3d/1The same four looks, one address each: rain on the glass, heat shimmer, scratched film, and rain falling in a sphere around the viewer. Kept for patches written before the verb above.
/camera/1 · /camera/2 · /augment/1The back camera straight to the screen, the front camera as a mirror, and look-around mode.
/rain/0 · /heat/0 · /camera/0 · /augment/0The way out: whatever look is running ends and the phone is back to the show.
The house itself
/title/1A string for the top bar of every phone — the act, the scene, the name of the piece. iPhones show it; Android ignores it. The Dashboard sends it from a Title instance, which carries its own words.
/volume/1A float, 0 to 1 — how loud the next clip or sound plays on every device. A setting sent ahead of a cue rather than a fader ridden under one; the Dashboard sends it from a Volume instance.
/start/1 · /stop/1The house into the show, and back to the foyer. Start takes the show’s name and puts the poster up; Stop returns every phone to its idle screen. Both stop whatever is playing. The Dashboard sends them from a House instance — Start and Stop of one row.
Files delivered to the house
/atc/manifestThe show changed — fetch the manifest from the sender and quietly pre-stage anything not already held. Carries the file server’s port, the show, and where to report.
/atc/assetOne file by digest, with its size, MIME type, name, show, layout and 3D-audio flag — fetch it and present it.
/atc/play · /atc/pause · /atc/stopTransport for a delivered clip or sound, named by its digest; play carries the loop flag and the share.
/atc/show · /atc/hideThe same two moves for a still.
/atc/layoutHow the next play presents a clip: flat, a 360° sphere, or driving the displacement shader over the live camera.
/atc/loopFlips looping mid-play, for a file or a whole AR set, without restarting it.
/atc/hrtfposMoves a spatialized voice around the listener — metres right, up and forward — live, even mid-line.
/atc/raise · /atc/strikeAn AR set stands up on its anchor, and comes down again.
/atc/inventoryAsks every device what it holds for this show; devices answer on the telemetry port. The Dashboard’s Refresh sends this and the manifest together, so a device that is behind is mended as it is counted.
/atc/discardDeletes one file, wherever it is filed.
/atc/clearshowDeletes everything held for one production — the night a run ends.
Older addresses the apps still answer
/heartbeat/1Keeps late arrivals and restarted apps in show mode.
/speaker/1 · /headphones/1 · /stream/1Audio routing and streaming. iPhones only.
/cloudvideo/1 · /cloudaudio/1 · /cloudpicture/1 · /cloudanimatedgif/1 · /cloudalphacam/1The Firebase era: play a file from the app’s own documents folder by name. Both apps still parse these, but the background sync that used to fill that folder no longer runs, so in a current build there is nothing for them to play. Superseded by /atc/manifest and /atc/asset, which deliver over the venue’s own Wi-Fi and need no internet at all.

Cues are broadcast on UDP port 3000; the phones learn where the sender is from the packet itself. The last group is what remains unclaimed: answered by the apps but sent by no Dashboard control, reachable only by hand from a patch like the one below — and the cloud addresses among them have nothing behind them at all any more.

Firing cues from Isadora, step by step
Put the device (running Augmented Theatre) and the computer on the same Wi-Fi network.
Download and install a demo version of Isadora from TroikaTronix (Windows & macOS).
Download the demo patch and its companion text file, then open the patch in Isadora:
⇩ loadjs.izz — Isadora patch ⇩ waitingforgodot.txt
Open the scenes left to right; the comments in each list the keyboard actions that fire cues. Press them and watch the device answer.

The broadcast port is changeable in the app's settings — useful for giving technicians a channel separate from the audience's.

What the house says back (UDP port 3001)
/atc/progressdevice id, digest, bytes received, bytes total — sent while a file is downloading, and once more when it lands. This is what fills the Dashboard's delivery counts.
/atc/holdingdevice id, show, chunk number, and the digests it holds — the answer to /atc/inventory. A device holding nothing still answers, which is how the operator tells an empty phone from an absent one.
/location/1~once per second when Bluetooth is on: nearest proximity beacon name (or NONE), a unique device identifier, and the device IP.
/devicestate/1on app state change: state (startup / foreground / background / shutdown), device identifier, device IP.

To brand the app for your production, write to or the Facebook page.

Curtain callPremiered on a real stage

The app premiered in the first Russian augmented-theatre production at the Meyerhold Theatre Center, Moscow, September 4, 2016. Director Victor Ryzhakov · music director Tatyana Pykhonina · video artist and interactive media programmer Vladimir Gusev.

Vladimir Gusev

NYC-based, critically acclaimed theatre and art videographer and media director. His interactive productions in the US, Europe and Russia blend video, sound, light, generative graphics and audience interaction to extend the traditional notion of theatre.

Meyerhold Theatre Center

An innovative theatre laboratory and a leading Russian theatrical production, experiment and education organization.

The apps

iOS · Android · the Dashboard for Mac — rebuilt in 2026 for current devices. The app is show-agnostic: any production can brand it and drive it as its own.