Gaussian splats

A Gaussian splat capture stores a place as millions of small, soft, coloured ellipsoids rather than as surfaces. A trainer fits them to photographs until, seen from where the photographs were taken, they add up to the place. Each one is drawn as an ellipse the size it projects to, fading out towards its edges and blended over whatever’s behind it. That means they have to be drawn back to front, and sorted again whenever the camera moves.

OrblitScene.splats takes a list of OrblitSplats, one per cloud.

A capture

import 'dart:math' as math;

import 'package:orblit_filament/orblit_filament.dart';
import 'package:vector_math/vector_math_64.dart';

OrblitSplats garden(OrblitDeviceProfile? device) => OrblitSplats(
  key: 1,
  path: OrblitResources.nameFor('garden.osplat'),
  // Structure-from-motion leaves a lot of captures upside down.
  transform: Matrix4.rotationX(math.pi),
  harmonics: math.min(2, device?.harmonicDegree ?? 2),
  limit: device?.splatBudget,
  coarseOrder: device?.coarseSplatOrder ?? false,
);

OrblitScene place(OrblitCamera camera, OrblitDeviceProfile? device) =>
    OrblitScene(camera: camera, objects: const [], splats: [garden(device)]);

path is a file, or a name you’ve provided bytes for. Four formats read:

Format	What it is	View-dependent colour
.ply	What the reference trainer writes	Up to degree 3
.splat	Compact 32-byte records	None
.spz	Niantic’s compressed format, versions 2 and 3	Up to degree 3
.osplat	The cloud exactly as the renderer holds it	Whatever it was cooked with

A name that doesn’t end in .ply, .spz or .osplat is read as .splat records.

transform takes the capture’s coordinates to the world’s. A capture comes out of structure-from-motion facing whichever way the first photograph did, and a half turn about x is the usual fix. A .spz is turned into the same frame as a .ply as it’s read, so one transform does for either. Scaling the transform scales every splat’s size along with its position, so a capture scaled up is the same picture, only larger. opacity multiplies every splat’s own opacity, and brightness its colour.

Splats are drawn after the solid scene. They test against its depth but never write any, so a wall in front of a cloud hides it, and a cloud never hides a wall. They aren’t lit, because a capture’s lighting is already in its colours, and they neither cast nor receive shadows.

The device

OrblitDeviceProfile puts each device in a tier, and three answers for splats hang off it:

Tier	splatBudget	harmonicDegree	coarseSplatOrder
Low	250,000	0	true
Medium	1,000,000	2	false
High	3,000,000	3	false

A device is low if its featureLevel is below 3, it can’t hold a 4096 texture, it has two threads or fewer, or it has under 3 GB of memory. High needs 8192 textures, eight threads and 8 GB, and a device that won’t say how much memory it has can’t be high. Everything else is medium. These are starting points, and no phone has measured them yet.

The view knows which device it’s on:

import 'package:flutter/widgets.dart';
import 'package:orblit_filament/orblit_filament.dart';

class Place extends StatefulWidget {
  const Place({super.key, required this.scene});

  final OrblitScene Function(OrblitDeviceProfile? device) scene;

  @override
  State<Place> createState() => _PlaceState();
}

class _PlaceState extends State<Place> {
  OrblitDeviceProfile? device;

  // Null until the renderer has started, which in a browser is a frame or two
  // after the view is laid out, so this asks again until there's an answer.
  Future<void> learn(int viewport) async {
    for (var attempt = 0; attempt < 50 && mounted; attempt++) {
      final profile = await OrblitView.profileOf(viewport);
      if (profile != null) {
        if (mounted) setState(() => device = profile);
        return;
      }
      await Future<void>.delayed(const Duration(milliseconds: 100));
    }
  }

  @override
  Widget build(BuildContext context) =>
      OrblitView(scene: widget.scene(device), onViewport: learn);
}

onViewport hands you the view’s number once the renderer has one, and OrblitView.profileOf asks what that viewport’s device can do. The answer is measured once, when the renderer starts, so ask until you get it and then keep it. The gallery does exactly this, every 100 ms for up to five seconds.

A capture sent before the answer arrives is read at degree 2 with no limit, and then read again once the answer does, because a new degree or limit reads the file again. On a small device, leave the capture out of the scene until you know what the device is.

Made in Dart

A cloud can come from memory instead, in the same 32-byte records a .splat file holds:

import 'dart:math' as math;
import 'dart:typed_data';

import 'package:orblit_filament/orblit_filament.dart';

// A ring of 10,000 flat, orange discs, 2 m across.
Uint8List ring() {
  const count = 10000;
  final positions = Float32List(count * 3);
  final scales = Float32List(count * 3);
  final colours = Float32List(count * 4);
  for (var i = 0; i < count; i++) {
    final angle = i / count * 2 * math.pi;
    positions[i * 3] = math.cos(angle);
    positions[i * 3 + 2] = math.sin(angle);
    scales.setAll(i * 3, [0.03, 0.001, 0.03]);
    colours.setAll(i * 4, [0.9, 0.5, 0.2, 0.6]);
  }
  return OrblitSplats.pack(
    positions: positions,
    scales: scales,
    colours: colours,
  );
}

OrblitSplats ringSplats(Uint8List records, int revision) =>
    OrblitSplats(key: 2, data: records, revision: revision);

Scales are standard deviations in metres, not the logarithms a .ply stores. Colours are red, green and blue from 0 to 1, with alpha as the splat’s peak opacity. rotations is optional, as quaternions in (w, x, y, z) order. Colour and rotation are kept to a byte a channel, which is what a .splat does.

Build the records once and keep them. They only travel to the renderer when revision changes, so bump it when you’ve written new ones and leave it alone otherwise. A few hundred thousand splats is megabytes, and a cloud that’s only being looked at shouldn’t send any of it. There’s no room in the records for view-dependent colour, so a cloud made this way is flat whatever harmonics says.

View-dependent colour

A trainer fits each splat’s colour as spherical harmonics, so a surface can be one colour from here and another from over there. That’s what a polished floor, a window or wet tarmac looks like. harmonics says how many degrees of that to read, from 0 to 3, and each one costs 16 bytes a splat. At a million splats that’s 16 MB for degree 1, 32 MB for degree 2 and 48 MB for degree 3, on top of the 48 MB the splats themselves take. The default is 2, which is what most captures are trained to.

A file trained to a lower degree than you ask for is read as far as it goes. One trained higher is read to your degree, and the scene notes say what was left out.

Limits

limit is the most splats to draw, up to 16,777,215. As the cloud is read, every splat is ranked by how opaque it is and how much of the screen it can cover, and the limit keeps the ones at the top. So what a smaller budget takes away is the faint, small splats a capture is thickest with, rather than a random share of everything, and the rest never reach the GPU. The scene notes say when a limit dropped any.

A limit is applied as the cloud is read, so a new one reads a file again. A cloud from memory takes a new limit with its next revision.

Sorting

Whenever the camera moves, the cloud is sorted again, natively on a thread of its own, or on a Web Worker in a browser. A browser doesn’t get threads because they’d need the page to be cross-origin isolated and a second build of Filament. A cloud of 16,384 splats or fewer is sorted on the thread that asks, which is quicker than handing it over.

A page that won’t start a worker has its clouds sorted on the page’s own thread, with a line in the console saying so. A worker that hasn’t answered in half a second is sorted for on the page too, since a frame spent sorting is better than a cloud left in the wrong order.

Before sorting, splats behind the camera or more than a quarter of a screen past its edge are dropped, and only the ones kept are uploaded and drawn.

coarseOrder sorts on 16 bits of depth instead of 32, which is half the passes. Splats closer together than a 65,536th of the depth the visible ones span can come out in either order. It’s there for a device where a full sort can’t keep up with a turning camera. sorted: false draws every splat in the order given, and only exists to measure what sorting is worth.

Cooking a capture

A .ply spends its load working out an exponential, a quaternion and a covariance for every splat. An .osplat holds the results, so opening one is a read and four copies. native/headless/build.sh in orblit_filament builds orblit_splat_cook, and that script is written for a Mac:

orblit_splat_cook garden.ply garden.osplat --harmonics 2 --limit 1000000

It reads anything the renderer reads, including an .osplat cooked at a higher degree. --harmonics defaults to 3, and --limit ranks splats the same way the renderer does, so a cooked file can be the small one a phone loads. Afterwards it reads both files again and prints how long each took.

For one of Niantic’s .spz samples, reading went from 340 ms to 17 ms, and the cooked file drew the same frame to the byte. A 300,000-splat .ply at degree 3 went from 67 ms to 6 ms.

What it’s worth

• Niantic’s own .spz samples, of 786,233 and 932,560 splats at degree 3, read and draw.
• From a viewpoint inside the 786,233-splat capture, culling kept 519,095 of them.
• On an M4 Pro, a real capture of 786,000 splats costs 6.6 ms of GPU a frame. A generated ring of a million splats costs about 45 ms at 1600×1200, both before culling was added and after. Every splat in that ring is a large, half-transparent disc, and all of it’s on screen, so there’s nothing to cull and every pixel is drawn many times over.
• With the clock held, native frames of the ring were byte-identical before and after culling and the new sorter went in, at 300,000 splats and at a million.
• The browser’s sorting worker was seen answering in 18.6 ms. Headless Chrome’s virtual clock usually doesn’t get round to the worker before the screenshot, so the browser check mostly exercises the half-second fallback instead.

In a scene file

A splats component has an asset, a budget and harmonics. The stage passes the budget as limit and clamps harmonics to 0–3, or uses 2 if the file doesn’t say. It doesn’t ask the device, so a component with no budget draws every splat. See Scene files.

Where it’s been seen

The Gaussian splats example generates its cloud, so there’s nothing to download: a ring of 100,000, 300,000 or a million flat, half-transparent discs, striped so its near and far sides are different colours. Turn the sort off and you can see exactly what goes wrong where the two sides overlap. A solid pillar stands in the ring to show the depth rule, and the cloud is held to the device’s budget, degree and coarse sort.

Splats have been seen on macOS and in Chrome. They haven’t been checked on Android or iOS hardware, on Windows or Linux, or in any browser but Chrome.