GATS selects 25% of tokens using LiDAR and cuts L2 trajectory error to 0.520 m

The authors of an arXiv preprint have proposed GATS (Geometry-Aware Token Selection), a method that uses LiDAR and radar to discard unneeded image patches in advance in camera-based autonomous driving models. At a 25% token budget, the L2 trajectory error (the mean deviation of the predicted path from the real one at 3 seconds) on the nuScenes dataset was 0.520 m, versus 0.573 m for the dense model that processes the whole frame. On the NAVSIM benchmark, the aggregate EPDMS score (the official driving-quality score) rose from 85.9 to 88.9, and total compute was lower. The sparse model plans better than the dense one that sees everything.

Why the dense model loses to itself

Six surround cameras processed by DINOv2-Small produce about 3500 tokens per frame and 185 GFLOPs for encoding alone. Most of those tokens describe sky, buildings, or road far from the route. Meanwhile, a neighboring car occupies just 3 tokens (the median across all cars in the validation set), while 6913 tokens end up in the planner's cross-attention. A planner with limited attention drowns in thousands of useless patches.

The authors tested the hypothesis with a simple experiment. Randomly dropping 75% of tokens lowers the L2 error from 0.573 m to 0.550 m. A trained LiDAR feature-fusion branch (a four-layer MLP with 2.4M parameters) did not improve planning. The researchers' conclusion: the bottleneck is not what the model sees, but how much irrelevant content competes for the planner's attention.

How GATS chooses where to look

The selector works only with measured 3D geometry and does not need image features. As a result, the vision ViT encodes only the selected patches, and the discarded ones cost no compute at all. This is impossible for methods that judge token importance from the network's own outputs: they must encode the whole frame first.

Multi-sweep LiDAR points and radar detections are projected onto each camera's grid. For each patch, the system builds a 16-dimensional vector: height above ground, motion residuals between sweeps, radial velocity from radar, and LiDAR reflection intensity. The budget is split into three streams with non-overlapping candidate pools:

• 55% goes to above-ground patches, scored by obstacle features, motion, and proximity to the vehicle's path;
• 30% goes to the road surface near the driving corridor;
• 15% is spread across all patches and catches objects such as suspended traffic lights, which have no ground-level LiDAR points.

The 55/30/15% split and the fixed scoring coefficients were tuned on validation, while a small two-layer network is fine-tuned together with the world model. Each selected token gets a 3D anchor, the median position of its LiDAR points. The anchors let the system link patches across different cameras and align frame history using odometry. A self-supervised latent world model predicts future features and 3D offsets without box annotations.

Numbers and limitations

At a 10% budget, the full pipeline costs 24 GFLOPs, compared with 185 GFLOPs for the dense model's encoding alone. L2 on nuScenes is 0.537 m and EPDMS on NAVSIM is 87.9, which is also better than the dense baseline (0.573 m and 85.9).

On 739 selected scenarios where the intentions of other road users are critical, GATS covers 95.8% of dangerous vehicles. For random selection this figure is 81.5%, and for a learned selector it is 70.0%. The collision rate was 0.09% versus 0.34% for the dense model, which is 73.5% lower. The authors attribute the robustness to rare situations to the fact that measured geometry does not depend on how often objects appear in the training set: any vehicle that reflects LiDAR above ground level will be selected.

There are caveats. The method relies on LiDAR and radar, so it does not suit camera-only systems. The scoring coefficients and stream shares were tuned by hand. A direct comparison with published end-to-end systems is difficult: they use different encoders, training data, and evaluation protocols. On NAVSIM, where a standardized scoring function makes comparison fair, GATS beats the dense baseline by 3 EPDMS points. The practical takeaway: if a vehicle already has LiDAR, it is more useful as a compute router for the vision network than as one more source of features.