AI4Fire: SQL Tool Lifts LLM Accuracy on Wildfire Databases From 16% to 88%

Six base language models, five wildfire management tasks, two run modes. That is the setup of AI4Fire, a new benchmark on arXiv; an extended run adds 29 more models. The headline result: connecting a read-only SQL tool raised database accuracy for each of the six base models from at most 16% to at least 88%. On forecasting staffing needs, the picture is the opposite: no model beat simply repeating today's figure.

How the comparison works

The authors start from the premise that inflated scores are costly in this field, because property and lives are at stake. So every task is run in two variants. In the "bare" mode, the model receives only the task's input data. In the "grounded" mode, one task-specific element is added to that input. For smoke detection, it is a smoke-free frame from the same camera. For database work, it is a SQL tool. All runs are zero-shot, with no fine-tuning and no examples.

The paired comparison gives a clean estimate of what each addition contributes. According to the authors, they found 138 papers in the literature on fire-related tasks, and none combines this range of models, breadth of tasks and paired runs. The prompts, responses, scores, code and a record of the literature review are published.

• Base models: 6, plus 29 in the extended run
• Wildfire tasks: 5
• Mode: zero-shot, bare and grounded
• Papers in the literature review: 138
• SQL task: from ≤16% to ≥88% for every base model

Where the addition works and where it doesn't

The authors' first finding: grounding helps most where the added element carries the answer itself. A model with read-only access to the database, via SQL queries, pulls what it needs from the tables instead of recalling or guessing. The gap between 16% and 88% suggests that without the tool, the task effectively cannot be solved.

The second finding is sobering. In staffing forecasts, simple rules proved hard to beat: no base model outperformed repeating today's value. Two open-weight models mostly copied the median of similar past fire days. That is a weaker forecast than the naive rule. The model looks as if it is reasoning, but in practice it reproduces an averaged template that is worse than yesterday's figure.

Traps in public data

The third finding concerns the datasets themselves. In one public release, the fire danger index column perfectly separates the held-out set. That is leakage: a high model score on such a set says almost nothing about the model's capabilities. In another dataset, 67 aerial fire images have "smoldering" or "no fire" labels derived from a clipped maximum temperature. If the sensor hit the ceiling of its scale, the label reflects the instrument's limit, not reality.

Each of these errors inflates results, and the authors call exactly such scores dangerous for the field.

What teams deploying LLMs in fire services should do

Before any pilot, set a simple baseline rule, such as repeating the current staffing level, and require the model to beat it. If it doesn't, it is too early to connect it to planning. Where the answer lies in the data, give the model an access tool rather than relying on its memory: read-only SQL delivers a large gain with minimal risk of corrupting data. Before using public datasets, check them for feature leakage and for where the labels came from. It is enough to ask whether a single column can predict the target variable and whether the label was derived from a saturated signal.

Caution is also needed. All runs are zero-shot, and the conclusions apply to this specific set of five tasks. Results for fine-tuned or specialized systems may differ. But the open responses and code let others recheck the numbers and add their own models.