ChatGPT study: does a feminine-form prompt affect coding performance?

A small but carefully designed experiment turned up an unwelcome effect: in GPT-5.4 mini, a Russian-language "feminine" phrasing of a request can slightly degrade the quality of programming solutions. When the user framing differed by just one gender marker, the model made more mistakes with the phrasing "I would like your help" (feminine form), while the neutral and "masculine" phrasings produced nearly identical results. On easy tasks the difference almost vanished, but on hard ones it looked statistically significant.

The trigger for the check was an everyday observation by an ML research engineer, who noticed that the model's answers became less accurate when feminine forms slipped into a Russian conversation, such as "I already tried" or "I would like" (in the feminine). To avoid relying on a hunch, she framed the question strictly: does a gendered Russian-language self-presentation change the quality of solutions to English-language coding tasks, if everything else in the prompt and the answer format stays unchanged?

The test used LiveCodeBench, a popular benchmark with problems from LeetCode, AtCoder and Codeforces, where solutions can be verified objectively against ready-made tests. The key idea of the experiment is that the differences between prompt variants were minimal. In the neutral version, the model was simply asked to help solve a Python problem. In the "masculine" version, one phrase changed to "I would like your help" (masculine form), and in the "feminine" version to the same phrase in the feminine form. A second pair of similar phrasings was also tested.

The researcher used 1055 problems from the LiveCodeBench v6 release, with the strictest possible run settings: one attempt per problem, temperature 0, and pass@1 as the main metric, meaning whether the model solves the problem on the first try. Two OpenAI models were tested: GPT-5.4 mini and GPT-5.4. To check the robustness of the results, she applied bootstrap resampling with 10,000 resamples and a 95% confidence interval.

On GPT-5.4 mini, the effect appeared fairly clearly. Neutral phrasings gave a pass@1 of about 0.661–0.663, "masculine" ones from 0.660 to 0.668, and "feminine" ones 0.649–0.652. After pooling the two prompt variants, the difference between female and male had a confidence interval from -0.0265 to -0.0005, so it did not cross zero. In other words, the drop is small but not statistically random.

The most interesting part came when the results were split by difficulty: on easy and medium tasks there was almost no significant effect, while on hard tasks the difference between the "feminine" and "masculine" framing was -0.0314, with a confidence interval from -0.0600 to -0.0043. No notable divergence showed up across platforms, though on newer problems a trend toward a larger gap emerged, albeit less robust than the split by difficulty.

With the flagship GPT-5.4, the picture was different. Because of the cost and duration of the runs, it was tested only on hard tasks, and the effect could not be reproduced there. A likely explanation is that the stronger model solves this set far better than the mini version, at roughly 57 percent versus 33 percent, so the benchmark is no longer at the edge of its capabilities. Put differently, sensitivity to phrasing may show up precisely when a model is working at its limit rather than in its comfort zone. This is an important limitation: for now, it cannot be claimed that this is a universal property of all ChatGPT versions or of all LLMs.

The practical takeaway from the experiment is fairly down-to-earth. For hard tasks, where every attempt counts and the model may stumble on small details, it is safer to phrase requests neutrally and not add extra personal framing. This is not proof of "sexism" in the everyday sense, but rather a signal that even minimal linguistic markers can affect answer quality in measurable scenarios. The next logical step is to test other models, other languages and harder datasets, to understand where a quirk of one specific benchmark ends and a systemic problem begins.