Exploring MLLMs for ultra-fine-grained agricultural classification

The proliferation of high-resolution imaging technology across modern agriculture has enabled the routine acquisition of large quantities of crop and animal imagery at field scale. Unlike general object recognition, species-level agricultural classification frequently demands discrimination at the subcategory level, a challenge formally characterised as Ultra Fine-Grained Visual Classification (Ultra-FGVC)^[1], where inter-class differences are confined to highly localised morphological features such as leaf venation patterns, surface texture, or reproductive organ morphology.

The economic stakes of misclassification at this granularity are substantial: pathogens and pests cause global yield reductions for major food crops estimated at 20%–30%^[2], and accurate subspecies identification is a prerequisite for targeted interventions, resistance monitoring, and precision agriculture workflows that minimise chemical inputs^[2]. Erroneous weed identification can result in ineffective herbicide application^[3,4], while failure to distinguish morphologically similar pest species may lead to inappropriate pesticide selection and consequent yield loss^[5,6]. Such identifications have historically required trained entomologists or plant taxonomists whose availability does not scale with the volume of field imagery now routinely captured by UAVs and IoT-connected sensors^[7,8].

The difficulty of automated Ultra-FGVC is twofold. First, discriminative features are highly localised and morphologically subtle: differences may manifest as minor variations in antennal segmentation, leaf margin serration, or seed coat patterning that occupy only a small fraction of image pixels, as illustrated in Fig. 1. Second, as classification granularity descends from species to cultivar level, the number of available labelled images per category decreases sharply^[9], creating a data scarcity problem that undermines standard supervised learning. Together, these constraints define a regime where both convolutional and Vision Transformer architectures plateau well below operationally useful accuracy thresholds, as demonstrated on early Ultra-FGVC benchmarks^[1].

Figure 1. 

Representative images from the Cotton80 cultivar classes (three images per class)^[1]. Despite belonging to distinct cultivar categories, samples share near-identical macro-morphology: a defining characteristic of Ultra-FGVC that limits even state-of-the-art supervised methods to below 60% accuracy at K = 50. The figure is original and created by Chenhui Wang. © 2026 Chenhui Wang.

Multimodal Large Language Models (MLLMs)^[10,11] offer a fundamentally new avenue for addressing these limitations. Trained on web-scale corpora, MLLMs encode substantial structured knowledge about biological entities including morphology, ecology, phenology, and diagnostic attributes that is entirely absent from image-label supervision. This parametric knowledge is available at inference time without task-specific retraining, making it particularly attractive under the data-scarce conditions characteristic of Ultra-FGVC.

Despite this potential, MLLM performance on Ultra-FGVC remains largely unexplored. Existing multimodal benchmarks (MMBench^[12], SEED-Bench^[13], and MMMU^[14]) evaluate visual reasoning at coarse to mid-level granularity, typically ImageNet-scale species-level discrimination. Agricultural vision benchmarks similarly operate at disease category or pest group level rather than cultivar or subspecies level. The few studies that do engage with standard FGVC granularity, such as discriminating aircraft variants^[15] or car models^[16] already document measurable accuracy shortfalls in MLLM performance relative to fine-tuned supervised models, even with careful prompt engineering^[17,18], pointing to a fundamental visual encoder limitation^[19] that the present work is the first to characterise at Ultra-FGVC scale. The transition from species to subspecies classification collapses inter-class feature distances and removes the coarse semantic anchors upon which MLLM visual encoders rely, yet no systematic study has characterised how this transition degrades model performance.

To address this gap, we present a systematic evaluation of state-of-the-art MLLMs: Kimi-K2.5^[20], GPT-5.4^[21], Gemini-3-Flash-Preview^[22], Qwen3.5-397B^[23] on Ultra-FGVC tasks at $ K \in \{3, 10\} $, where K denotes the number of candidate categories presented at inference time, corresponding to operationally meaningful deployment scenarios in precision agriculture. Each experimental component is designed to isolate a distinct hypothesis: whether MLLMs can serve as viable retraining-free alternatives to supervised methods at these scales; whether accuracy at this granularity is driven by visual exemplars or by parametric knowledge activated by category names; and whether prompt engineering can materially close the gap to supervised performance, or whether the bottleneck lies upstream in the visual encoder itself.

We evaluate few-shot classification^[10], where we provide a few reference images alongside structured prompts, as well as classification augmented with semantic category names, and a controlled ablation of six prompt design strategies including direct classification, chain-of-thought reasoning, and contrastive attribute prompting. A preliminary three-round evaluation at K = 50 was conducted for GPT-5.4; the remaining three models were subject to context-window and per-request image-count constraints at that scale, and systematic K = 50 results therefore remain an open empirical question. The main contributions are:

1. A systematic empirical evaluation of four state-of-the-art MLLMs on Ultra-FGVC benchmarks at $ K \in \{3, 10\} $, establishing the first controlled baselines for MLLM few-shot ICL performance at this granularity. MLLMs exceed human expert accuracy at K = 3 but fall 12%–34% below fine-tuned supervised methods at K = 10, delineating the practical boundary of current MLLM capability.

2. A controlled blind-vs-named label experiment across twelve model–K–dataset combinations, providing the first statistical examination of whether label semantics contribute to Ultra-FGVC accuracy beyond visual exemplars. No condition yields a significant gain from name disclosure after Bonferroni correction (m = 12), though the K = 9 conditions remain underpowered and results should be treated as inconclusive rather than as evidence of visual primacy.

3. A ten-round, six-variant prompt ablation study on two model backends, revealing strongly model-dependent prompt sensitivity with sign-reversed effects across all four tested design dimensions. These findings caution against adopting prompt strategies validated on a single architecture as transferable defaults.

4. A characterisation of three fundamental failure modes limiting MLLM deployment in Ultra-FGVC—the visual encoder resolution, output generation unreliability, and confidence miscalibration—together with concrete technical directions (richer visual backbones, constrained decoding, and post-hoc calibration) well positioned to advance MLLMs toward operational viability.

Since the introduction of CUB-200-2011^[24], FGVC has evolved into a well-defined research area. Early benchmarks, including FGVC-Aircraft^[15] and Stanford Cars^[16], established the task as discrimination among visually similar subcategories within a superordinate class (e.g., distinguishing a Boeing 737-700 from other 737 variants), driving progress towards modern ViT-based models^[25] such as SM-ViT^[26] and Cross-Layer Feature Fusion ViT^[27], both of which substantially outperform CNN baselines including ResNet^[28] and AlexNet^[29].

Ultra-FGVC is qualitatively harder: distinguishing traits are highly localised and morphologically subtle; the expert knowledge required for their reliable identification contributes to data scarcity; and labelled images per category decrease sharply as granularity descends from species to cultivar level^[9].

The first dedicated Ultra-FGVC dataset was established by Yu et al.^[1], who released a large-scale leaf dataset across cotton and soy with baseline CNN evaluations. Even state-of-the-art supervised methods struggle: the CNN-based architecture of Pan et al.^[30] and the ResNet-50/ViT hybrid of Yu et al. achieve top-1 accuracies of 62.08% and 60.42% on Cotton80, and 49.67% and 56.17% on SoyLocal. The more recent pure-Transformer CLE-ViT achieves similarly modest results^[31], suggesting that the bottleneck is representational rather than architectural.

Beyond static recognition, real-world agricultural deployment introduces a temporal dimension: new cultivars are continually registered, and crops must be recognised across successive reproductive stages. Zhang et al.^[32] formalise this as continual ultra-fine-grained visual recognition (C-UFG), decomposing the problem into class-incremental learning of new cultivars (VC-UFG) and domain-incremental recognition of the same cultivar across growth stages (HC-UFG). Their analysis of pre-trained model-based continual learning methods on large-scale soy benchmarks reveals that inter-task prompt interference and classifier recency bias emerge as the dominant failure modes in the continual regime, compounding the representational limitations already observed in static Ultra-FGVC. The core difficulty in this field thus lies not only in visual discrimination within a fixed category set, but in maintaining that discrimination robustly as the task distribution evolves.

2.2.   Multimodal large language models

MLLMs extend language models to operate jointly over images and text by coupling a CLIP-style vision encoder^[33] with a language model backbone via a lightweight cross-modal connector^[34], as shown in Fig. 2. Rather than using a fixed classification head, MLLMs treat recognition as instruction-following, enabling zero-shot and few-shot classification across open-ended label spaces within a single generative framework.

Figure 2. 

Base architecture of a multimodal large language model (MLLM) for few-shot in-context learning. Reference images ^[1] (K categories, n shots each) and an unlabelled query image are encoded by a CLIP-based vision encoder^[33], whose patch features are projected into the language model's token space by a lightweight cross-modal connector (Q-Former^[35] or MLP bridge^[11]). A text prompt encoding the task instruction and category identifiers is tokenised in parallel. Both token streams are consumed jointly by the LLM backbone, which generates a structured output containing the predicted category label. In this work, category identifiers are numeric only (Task 1) or real species names (Task 2); the backbone architecture is otherwise identical across conditions. The figure is original and created by Chenhui Wang. © 2026 Chenhui Wang.

Flamingo^[10] demonstrated few-shot visual learning from interleaved image-text sequences via gated cross-attention, achieving state-of-the-art performance across 16 benchmarks. BLIP-2^[35] introduced the Q-Former, achieving stronger zero-shot VQA performance than Flamingo-80B with 54X fewer trainable parameters. InstructBLIP^[36] extended this with instruction-aware query transformers across 26 datasets. At the proprietary frontier, GPT-4V^[37], Gemini 1.5^[38], and Claude 3^[39] serve as key reference points for zero-shot visual reasoning.

MLLMs exhibit strong classification ability when prompted effectively: LLaVA-1.5 achieves 85% zero-shot accuracy on MNIST and 79% on skin-lesion classification without fine-tuning^[17], and GPT-4 with generated textual descriptions rivals EVA-CLIP-ViT-E across 16 benchmarks^[18]. Domain-adapted variants amplify performance further: LLaVA-Med^[40] improves biomedical VQA by approximately 30 percentage points after instruction tuning, and Co-LLaVA^[41] exceeds 85% on remote sensing scene classification. Persistent limitations include insufficient visual discriminability in CLIP-based encoders for fine-grained recognition^[19], object hallucination^[42], and the fact that ImageNet accuracy still lags behind specialised contrastive models without careful prompt engineering^[17,18].

2.3.   Prompting strategies for visual classification

Prompt engineering (the systematic design of natural language instructions supplied at inference time without parameter modification) directly governs how an MLLM frames a recognition task, which features it attends to, and how it produces output, as shown in Fig. 3. Prompt design has been shown to produce accuracy swings of tens of percentage points on identical model weights and datasets^[44], making it a primary engineering variable in the MLLM classification pipeline.

Figure 3. 

Structural comparison of the three prompting paradigms reviewed in this section. (a) Few-shot ICL prepends $ K $ labelled reference image-label pairs $ ({\bf{x}}_k, y_k) $ as in-context demonstrations before the query, enabling task adaptation without gradient updates^[10,43]. (b) Describe-then-classify applies chain-of-thought decomposition: the model first articulates observable visual features, then compares them against candidate classes before committing to a label, reducing confirmation bias^[44,45]. (c) Contrastive prompting extends (b) by requiring explicit enumeration of matching and contradicting evidence per class and a justified elimination of the runner-up^[46,47]. Node shading: dashed grey = reference demonstrations; solid grey = query image; blue = chain-of-thought step; orange = contrastive reasoning step; green = predicted output. The figure is original and created by Chenhui Wang. © 2026 Chenhui Wang.

2.3.1.   Chain-of-thought and describe-then-classify

Chain-of-thought (CoT) prompting^[44] elicits step-by-step intermediate reasoning by including worked demonstration examples, decomposing complex decisions into traceable steps. Zhang et al.^[48] extended this to the multimodal setting, showing that a two-stage rationale-then-answer pipeline enabled a sub-1B-parameter model to surpass GPT-3.5 on multimodal reasoning benchmarks while substantially reducing hallucination.

The most impactful variant for visual classification is the describe-then-classify paradigm, in which the model first generates a verbal description of the observable visual evidence before committing to a category label. Menon & Vondrick^[45] formalised this by querying an LLM for discriminative per-class descriptors and aggregating CLIP similarity scores over those descriptors, yielding consistent gains on zero-shot benchmarks. Pratt et al.^[49] showed that automatically generating category-specific descriptive prompts via an LLM consistently outperforms hand-crafted CLIP templates across diverse tasks. FineR^[50] further instantiated this as a training-free framework that extracts part-level visual attributes via VQA, reasons over them with an LLM's world knowledge, and constructs multimodal classifiers surpassing state-of-the-art FGVC models on multiple benchmarks. Most directly relevant to the agricultural domain, ChatLeafDisease^[51] applied structured CoT to crop disease classification, with disease-description scoring rules boosting GPT-4o accuracy from 45.9% to 88.9%; removing those rules caused accuracy to collapse to 51.8%.

2.3.2.   Few-shot in-context learning

In-context learning (ICL) allows an MLLM to adapt to a new task at inference time by conditioning on interleaved image-label demonstrations without gradient updates. Flamingo^[10] established this paradigm, showing that a handful of reference image-text pairs sufficed to match task-specific fine-tuned models across wide-ranging benchmarks. Jiang et al.^[43] subsequently showed that classification accuracy scales log-linearly with demonstration count up to 2,000 shots for frontier models such as GPT-4o and Gemini 1.5 Pro, with this benefit absent in open-weight models.

Beyond shot count, demonstration selection and ordering are also consequential. Lu et al.^[52] showed that permuting the order of otherwise identical few-shot examples can shift accuracy from near state-of-the-art to near chance. Zhang et al.^[53] showed that semantically similar examples retrieved via nearest-neighbour search consistently outperform random sampling. Min et al.^[54] found that ground-truth labels within demonstrations contribute less to ICL performance than the exposed label space, input distribution, and prompt format, implying that reference examples function primarily as task-specification signals. Ferber et al.^[55] demonstrated that kNN-selected histopathology images brought GPT-4V to parity with specialist networks on cancer classification, a finding with direct implications for data-scarce agricultural classification.

2.3.3.   Contrastive prompting

Contrastive prompting instructs the model to reason about categories relative to one another, identifying distinguishing features between visually similar classes. Chia et al.^[46] proposed Contrastive CoT, augmenting demonstrations with paired invalid reasoning examples. Yao et al.^[56] showed that a lightweight contrastive instruction, prompting consideration of both a correct and a plausible incorrect answer, raised GPT-4 accuracy on GSM8K from 35.9% to 88.8%, integrating seamlessly with existing CoT pipelines.

In the visual domain, Jiang et al.'s Discrepancy-aware Text Prompt (DaTP)^[47] incorporates explicit subcategory-discriminative language into CLIP-based fine-grained prompts, highlighting the features that set each category apart from its closest neighbours. This reflects the view that fine-grained recognition is inherently a task of relative discrimination, and directly motivates the contrastive priming strategy adopted in the present work.

We evaluate on three Ultra-FGVC benchmarks spanning agricultural cultivar classification and ornithological species discrimination. All train/test splits are generated deterministically prior to inference. Following Yu et al.^[1], for each dataset K categories are sampled without replacement per round using seed $ s_r = s_{r-1} + 1 $ with $ s_0 = 42 $, making any round exactly reproducible from its index.

SoyLocal is a subset of the UFG image dataset^[1], containing 1,200 leaf images across 200 soybean cultivar categories with six images per category, acquired under controlled laboratory conditions with labels assigned from a genetic resource bank. The primary challenge is an extremely small inter-class variation paired with large intra-class variation: human breeding experts achieve only 63.89% top-1 accuracy on three-category evaluation and 12.84% on 50-category classification^[1]. For Cotton80 and SoyLocal, category labels are integer identifiers from a shared annotation file; reference and query images come from non-overlapping dataset splits.

Cotton80 is a second subset of the same UFG dataset^[1], containing 480 cotton leaf images across 80 cultivar categories. The dominant challenge is a severe self-overlapping problem arising from the palm-like morphology shared across categories, suppressing even the best supervised methods to below 60% accuracy on 50-way evaluation.

CUB-Crows is a curated Ultra-FGVC subset of CUB-200-2011 corresponding to the crow subset identified by Lu et al.^[57], comprising nine visually confusable categories spanning four core crow species and five morphologically similar species that trained models systematically confuse with crows. State-of-the-art supervised methods achieve substantially lower accuracy on this subset than on the full CUB benchmark^[57]. For CUB-Crows, where no pre-existing split is available, each category's images are sorted lexicographically and divided 50/50 into train and test pools; lexicographic ordering was validated to show no systematic correlation with acquisition metadata. Directory names beginning with $ \mathrm{like}\_$ or $ \mathrm{similar}\_\mathrm{to}\_$ are tagged as confuser categories for post-hoc stratified analysis.

CUB-Crows is included because its ecological confusion structure (visually near-identical corvid species that supervised models systematically confuse^[57]) closely mirrors cultivar-level confusion in the crop datasets, while its real species names (e.g. American Crow, Fish Crow) provide a controlled testbed for Task 2's label-semantics experiment. In agricultural deployment, the analogous question is whether naming a pest species (e.g. Helicoverpa armigera vs H. zea) helps an MLLM beyond visual exemplars alone. All agricultural inferences drawn from CUB-Crows results are made explicit where stated.

CUB-Cormorant is a three-category subset of CUB-200-2011 comprising three morphologically similar cormorant species (Brandt's Cormorant, Red-faced Cormorant, and Pelagic Cormorant). It is used exclusively in Task 2. The dataset is included because standard K = 3 CUB-Crows conditions, where most models achieve 84%–92% accuracy, are subject to accuracy ceiling effects that suppress any detectable semantic benefit regardless of whether one exists. CUB-Cormorant, where models achieve only 67%–74% under the blind condition, provides a K = 3 comparison free of ceiling effects. The same 50/50 lexicographic train/test split protocol applied to CUB-Crows is used here.

Split protocol. Two files are produced per configuration: $ \mathrm{splits}\_\mathrm{simple}\_\mathrm{K.txt}$, read directly by the inference pipeline, encoding role, path, and string category label per line (ground truth is therefore embedded in the file, requiring no separate annotation lookup); and $ \mathrm{splits}\_\mathrm{detailed}\_\mathrm{K.txt}$ documenting seed, category roles, and per-category image lists for auditability. We evaluated $ K \in \{3, 10\} $ for Cotton80 and SoyLocal, and $ K \in \{3, 9\} $ for CUB-Crows ($ K = 9 $ exhausts the available categories), and K = 3 for CUB-Cormorant (K = 3 exhausts the available categories). The values K = 3 and K = 10 were selected to match two of the evaluation conditions of the benchmark paper^[1], enabling direct comparison with its reported baselines at those levels.

Scope limitation. The K = 50 condition was not evaluated for Gemini-3-Flash-Preview, Kimi-K2.5, or Qwen3.5-397B, as supplying 150 reference images (50 × 3) exceeds their context-window or per-request image-count limits. For GPT-5.4, a preliminary evaluation comprising N = 3 rounds was feasible and is reported in Section 4.1. Results at $ K \in \{3, 10\} $ correspond to the deployment-relevant triage and screening scenarios described in Section 1. The K = 3 condition should be interpreted with caution given the 33.3% random baseline, while K = 10 is the more demanding and informative result.

3.2.   Models and experiment tasks

We selected four state-of-the-art MLLMs spanning different providers and architectural lineages based on the general vision capability analysis by arena.ai^[58], as summarised in Table 1. All models receive identical prompts and reference images with no model-specific calibration beyond API-mandated formatting differences. Exact model version strings as used in API calls are reported in Table 1; temperature was set to the provider default, and no model-specific system prompts beyond the task instruction were applied. Because these models are accessed via commercial APIs whose weights and training data are not publicly disclosed, results should be interpreted as characterising the deployed versions at the time of evaluation rather than fixed architectural configurations.

We formulate Ultra-FGVC as a few-shot in-context learning problem for MLLMs. Given K categories, each with n reference images, and a set of unlabelled query images, the model assigns each query to one of the K categories by visual comparison against the provided references, without gradient-based adaptation or fine-tuning. This setup directly mirrors precision agriculture deployment constraints, where labelled data is scarce, making retraining impractical.

The evaluation is structured as three complementary tasks that decompose MLLM Ultra-FGVC performance into orthogonal factors: Task 1 establishes a semantics-free baseline that isolates raw visual encoding capacity, providing the anchor against which the remaining tasks are interpreted; Task 2 quantifies whether label semantics contribute accuracy beyond what visual exemplars alone provide, with Task 1's blind protocol as the reference; and Task 3 measures prompt-design sensitivity with model, dataset, and reference images held fixed, so that any accuracy differences are attributable solely to prompt structure. This decomposition is necessary because the three factors are confounded in typical MLLM evaluations; disentangling them is essential to identify which failure modes require architectural remedies versus engineering interventions.

Task 1: Baseline MLLM Performance on Ultra-FGVC. We evaluate all four models on SoyLocal and Cotton80 under the unified few-shot ICL protocol. In each round, the model receives three reference images per category alongside a structured prompt and classifies held-out query images into one of the K categories. We target n = 20 rounds at K = 3 and n = 10 at K = 10. GPT-5.4 was additionally evaluated at K = 50 for n = 3 preliminary rounds; the remaining models could not be evaluated at K = 50 due to context-window and image-count constraints. All categories are identified by numeric labels only ($ 1, 2, \ldots, K $), conveying no semantic information about cultivar names and thereby isolating pure visual discrimination ability from any benefit of label semantics, which is investigated in Task 2.

Task 2: Effect of Category Name Disclosure. Task 2 investigates whether disclosing real species names (rather than the numeric identifiers of Task 1) improves accuracy, or whether MLLMs rely primarily on visual exemplars regardless of label semantics. We conduct a controlled two-pass evaluation on CUB-Crows and CUB-Cormorant, uploading reference and query images once per round and reusing them across both passes to eliminate API upload cost as a confound. The passes are structurally identical, differing only in label presentation. Within each round, the blind pass is administered before the named pass; pass order is fixed across rounds to allow cost-efficient image reuse, and any carryover effects within a round are assumed negligible given the stateless nature of the API inference calls.

● Blind pass. Categories are assigned anonymous numeric identifiers $ \{1, 2, \ldots, K\} $, matching the protocol of Task 1.

● Named pass. Categories are presented with their real string identifiers (e.g. American Crow, Fish Crow), allowing the model to draw on any world knowledge associated with those names.

Query images are anonymised and randomly ordered within each pass. The primary quantity of interest is the per-round accuracy delta:

aggregated over $ N = 20 $ rounds for $ K = 3 $ conditions and $ N = 10 $ for the $ K = 9 $ CUB-Crows condition; CUB-Cormorant ($ K = 3 $) was evaluated over $ N = 20 $ rounds for all four models. This matches the number of rounds used in Task 1. A consistently positive $ \Delta_r $ would indicate that label semantics provide a meaningful signal beyond visual similarity; values near zero would indicate visual-exemplar-driven behaviour.

Task 3: Prompt Ablation Study. Task 3 quantifies the contribution of prompt engineering to Ultra-FGVC performance by comparing six variants—Baseline, Direct, Instructions-Last, Contrastive, Interleaved, and Minimal—with model, dataset, category subset, and reference images held fixed, so any accuracy differences are attributable solely to prompt structure. All variants share the few-shot ICL scaffold (three reference images per category) following Flamingo^[10], and differ along three dimensions: (i) whether CoT decomposition is applied; (ii) whether contrastive elimination is required; and (iii) whether instructions are co-located with the images they govern. Table 2 summarises these choices.

Scope of Task 3. The ablation is conducted on SoyLocal only, using two of the four evaluated backends (Gemini-3-Flash-Preview and Kimi-K2.5). GPT-5.4 and Qwen3.5-397B are not included due to API cost constraints. The choice of dataset, SoyLocal, is due to the results from Task 1 as shown in Section 4.1. SoyLocal at $ K = 10 $ puts MLLMs in the 45%–52% range, while Cotton80's accuracy at $ K = 10 $ is 33%–39%, dangerously close to the 10% random-chance baseline. The study is most informative when there is room for the prompt effects to manifest in both directions.

The Baseline implements describe-then-classify^[45]: the model first articulates the query image's fine-grained visual features without reference to any category name, then compares those features against each category's references before deciding. This two-stage decomposition reduces confirmation bias by separating the descriptive and discriminative stages^[48], and serves as the primary reference point.

The Contrastive variant extends the Baseline by requiring the model to enumerate both matching and contradicting features for every candidate category before reaching a verdict, and to name a runner-up with a justification for its elimination. This operationalises contrastive priming^[47,46]: that fine-grained recognition is inherently a relative judgment, and surfacing contradictions reduces the rate at which plausible but incorrect categories escape elimination.

The Interleaved variant retains CoT structure but co-locates per-category discriminative cues with their corresponding reference image block rather than using a global preamble, testing whether local instruction placement improves feature binding.

The Instructions-Last variant presents all images before any task instruction, testing whether deferring instruction onset alters the model's initial attention allocation and produces different feature extraction behaviour compared to instruction-first designs.

The Direct variant removes CoT entirely, presenting a concise classification instruction with no intermediate reasoning steps, isolating the contribution of the two-stage decomposition by holding all other elements constant relative to the Baseline.

The Minimal variant reduces the prompt to its absolute minimum: a category list, reference images, and query images with no discriminative guidance, establishing the lower bound of prompt engineering contribution.

All six variants share an identical JSON output schema with six required fields, ensuring parsing differences cannot confound accuracy comparisons. Each variant is evaluated over $ N_{\mathrm{rounds}} = 10 $ independent rounds on SoyLocal for both model backends, with accuracy reported as mean $ \pm $ standard deviation across rounds. The actual prompts are shown in Fig. 4.

Figure 4. 

All six prompt variants used in Task 3, shown schematically (part 1 of 2: variants a–c). Image blocks are denoted $ \langle\,\cdot\,\rangle $; K categories with three reference shots each are assumed throughout. The shared JSON output schema is reproduced verbatim in every box so that differences in discriminative content are visible by direct inspection. See Fig. 4 for variants (d)–(f). (part 2 of 2: variants d–f.) Variants differ along three orthogonal dimensions: whether chain-of-thought (CoT) decomposition is applied ([a], [d], [e] vs [b], [c], [f]); whether explicit match/contradiction elimination is required ([d] only); and where task instructions appear relative to the images (preamble: [a], [b], [d]; after all images: [c]; co-located per category: [e]; inline: [f]). The Baseline (a) is the primary reference for all comparisons in Section 4.3; the Minimal (f) estab]lishes the lower bound of prompt engineering contribution. The figure is original and created by Chenhui Wang. © 2026 Chenhui Wang.

3.3.   Inference protocol

Prompt structure. The prompt presents three sequential components, illustrated in Fig. 5: (i) a task preamble naming the $ K $ categories and directing attention to subtle discriminative cues; (ii) per-category reference image blocks, each prefaced by a category header; and (iii) query image blocks followed by a three-step classification instruction and a mandatory JSON output schema. Category identifiers are numeric only in Tasks 1 and 3; real species names replace numeric identifiers in the named condition of Task 2.

Figure 5. 

Full baseline prompt used in Tasks 1–3. Italicised $ K $ and indexed placeholders are instantiated at runtime. The mandatory JSON schema enforces structured output and eliminates free-text parsing as a confound across all six Task 3 variants. In the named condition of Task 2, numeric category identifiers are replaced with real species names; all other elements are identical. The figure is original and created by Chenhui Wang. © 2026 Chenhui Wang.

Structured output. Each model returns a strictly formatted JSON array. Every record must contain $ \mathrm{filename}$, $ \mathrm{observed}\_\mathrm{features}$ three to five items), $ \mathrm{category}$, $ \mathrm{runner}\_\mathrm{up}$, $ \mathrm{confidence}$ (float $ \in [0, 1] $), and $ \mathrm{reasoning}$. The parser uses bracket-depth tracking and regex-based fence extraction to handle malformed outputs robustly, raising an exception if any required key is absent.

Filename resolution. After parsing, anonymised names are resolved to original filenames via the $ \mathrm{name}\_\mathrm{mapping}$ dictionary before accuracy evaluation.

Table 3 reports top-1 accuracy for all four MLLMs under the blind numeric-label protocol alongside the supervised baselines of Yu et al.^[1].

At $ K = 3 $, where the random-guessing baseline is 33.3%, cross-model mean accuracy reaches 82.4% on SoyLocal and 63.5% on Cotton80. With only three candidate categories, however, the discrimination task is substantially easier than the benchmark's primary evaluation condition, and even simple nearest-neighbour matching achieves high accuracy. The modest gap to supervised baselines at $ K = 3 $ therefore does not constitute a meaningful comparison of MLLM versus supervised performance.

Comparison with human experts. At $ K = 3 $, all four MLLMs substantially exceed human expert accuracy on SoyLocal (cross-model mean 82.4% vs 63.9%). On Cotton80, three of four models individually exceed human experts (Gemini 67.8%, GPT-5.4 71.7%, Kimi 67.2% vs 66.7%); the cross-model mean (63.5%) falls below the human expert level, pulled down by Qwen3.5-397B's substantially lower performance (47.2%). This reversal on SoyLocal is notable: domain experts whose labelling effort underlies the benchmark are outperformed by general-purpose MLLMs on the three-category task. The advantage does not persist at $ K = 10 $, however: cross-model MLLM means drop to 47.3% on SoyLocal and 35.3% on Cotton80, while human experts retain 32.5% and 63.3% respectively. On Cotton80 at $ K = 10 $, human experts substantially outperform all four MLLMs. This pattern is consistent with MLLMs benefiting from strong generic visual priors at very low category counts, while at higher category counts human experts' specialised morphological knowledge provides a decisive advantage.

The more informative condition is $ K = 10 $, where all four models fall substantially below fine-tuned supervised methods. The absolute gap widens sharply: DCL retains 63.3% and 73.3% at $ K = 10 $, while the best MLLM (GPT-5.4) reaches only 51.7% and 39.3%, deficits of 12 and 34 percentage points respectively. SoyLocal is consistently easier than Cotton80, a difference attributable to greater macro-morphological variation between soybean accessions compared to the subtler textural differences that distinguish cotton cultivars.

GPT-5.4 leads in three of four MLLM conditions; the sole exception is SoyLocal at K = 3, where Gemini achieves the highest top-1 accuracy (85.6% vs 81.7%). All models exhibit notable overconfidence, reporting mean confidence scores of 0.74–0.91 regardless of accuracy level.

Preliminary results at $ K = 50 $. A three-round preliminary evaluation of GPT-5.4 at $ K = 50 $ yields top-1 accuracy of 20.4% $ \pm $ 3.8% on SoyLocal and 12.4% $ \pm $ 5.1% on Cotton80. Both figures are well below all fine-tuned CNN baselines (AlexNet: 33.3% and 24.7%; ResNet-50: 62.0% and 52.7%) and, on Cotton80, fall below human expert performance (14.0%). On SoyLocal, GPT-5.4 at $ K = 50 $ slightly exceeds the human expert (12.8%), which may partly reflect the availability of semantic priors at higher category counts; however, three rounds are far too few to draw reliable conclusions. These preliminary results nonetheless support the visual encoder bottleneck hypothesis at benchmark scale: even the highest-performing MLLM degrades sharply from $ K = 10 $ to $ K = 50 $ in a manner consistent with CLIP-based encoder saturation rather than a failure of prompting or in-context learning.

Scaling behaviour. The degradation from $ K = 3 $ to $ K = 10 $ follows a consistent pattern across all four models and both datasets, with the gap to supervised methods widening at every step. The preliminary $ K = 50 $ data for GPT-5.4 are consistent with continued or accelerating degradation, though this must be confirmed with more rounds and additional models. At category counts representative of field-level cultivar triage ($ K \leq 10 $), MLLMs provide operationally useful accuracy, but they are not yet viable substitutes for fine-tuned supervised models in broader screening applications.

4.2.   Task 2: Effect of category name disclosure

Statistical significance was assessed using a one-sample paired t-test applied to the per-round accuracy difference $ \delta_r = A_{\mathrm{N}}^{(r)} - A_{\mathrm{B}}^{(r)} $, testing $ H_0{:}\;\mu_\Delta = 0 $ against a two-sided alternative. For $ K = 9 $ conditions ($ n = 10 $ rounds), where the Shapiro–Wilk test has insufficient power to confirm normality, results were verified with a Wilcoxon signed-rank test; all qualitative conclusions were unchanged. Cohen's $ d $ is computed as $ \bar{\Delta}/s_\Delta $.

Table 4 reports paired-comparison statistics for all twelve model–K–dataset combinations. Applying Bonferroni correction for the twelve simultaneous tests (adjusted threshold $ \alpha' = 0.05/12 \approx 0.00417 $), no condition reaches statistical significance. The $ K = 9 $ conditions ($ n = 10 $ rounds) are underpowered: post-hoc power for a medium effect ($ d = 0.50 $) at $ \alpha = 0.05 $ is approximately 35%, rising to roughly $ 50\% $ at $ d = 0.70 $. Null results for these conditions should therefore be interpreted as inconclusive rather than as positive evidence of equivalence. In particular, GPT-5.4 at $ K = 9 $ ($ p = 0.050 $, $ d = 0.72 $) narrowly misses the uncorrected threshold and exhibits a medium-to-large effect size; the absence of significance here reflects insufficient sample size rather than a reliably small effect. The four CUB-Crows $ K = 3 $ conditions ($ n = 20 $) yield $ p \geq 0.23 $ with small effect sizes ($ d \leq 0.28 $), but these conditions are subject to accuracy ceiling effects ($ \bar{A} \geq 0.84 $ for most models) that suppress any detectable semantic benefit regardless of whether one exists, and cannot therefore be interpreted as evidence of visual primacy.

CUB-Cormorant results. The four CUB-Cormorant conditions ($ K = 3 $, $ n = 20 $) provide the only $ K = 3 $ comparisons free of ceiling effects, with blind-condition means of 67.8%–73.9%. Despite the absence of ceiling suppression, no condition reaches statistical significance ($ p \geq 0.11 $). Effect sizes are inconsistent in direction across models: Gemini shows the largest positive trend ($ d = 0.37 $, $ \bar{\Delta} = {+}10.0 $ pp), GPT shows a small positive trend ($ d = 0.15 $), while Qwen shows a small negative effect ($ d = -0.21 $, $ \bar{\Delta} = {-}5.0 $ pp) and Kimi shows no effect ($ d = 0.00 $). The direction reversal for Qwen, where naming the species decreases accuracy, is inconsistent with a simple semantic enrichment hypothesis and suggests that name disclosure interacts with model-specific response tendencies rather than providing a uniformly additive signal. The cormorant results thus extend the CUB-Crows conclusion (no reliable semantic benefit) to a condition where that conclusion is no longer confounded by ceiling effects, while simultaneously revealing cross-model variability that resists a unified interpretation.

At K = 3 CUB-Crows, signed deltas are centred near zero with symmetric win/loss round counts. At K = 9, accuracy decreases globally, but the blind–named gap narrows rather than grows: Gemini, Kimi, and Qwen show mean deltas of at most +0.015.

Per-class analysis on CUB-Crows reveals a consistent two-tier structure. Visually distinctive species (Groove-billed Ani, Red-winged Blackbird, and Brandt's Cormorant) achieve $ \bar{A} = 1.00 $ across all four models at K = 3, confirming that label semantics add no marginal signal when exemplars are already discriminative. The corvid cluster (American Crow, Fish Crow, Common Raven) constitutes the persistent accuracy floor: Fish Crow is the hardest class for every model at K = 9, with named-condition means of 0.37, 0.33, 0.27, and 0.20 for Gemini, GPT, Kimi, and Qwen respectively, rankings identical to those in the blind condition. This per-class pattern is consistent with the corvid confusion reflecting a visual representational limitation rather than a vocabulary deficit resolvable by species names; however, the possibility of a moderate semantic benefit that the current design lacks power to detect cannot be excluded.

4.3.   Task 3: Prompt ablation study

Table 5 reports mean top-1 and top-2 accuracy across the six prompt variants on SoyLocal for Gemini-3-Flash-Preview and Kimi-K2.5 (n_rounds = 10 per variant per model). All rounds produced parseable output. The elevated standard deviations on the Direct and Minimal variants (17%–22% across both models) reflect genuine round-to-round variability driven by stochastic category sampling rather than output failures: these variants provide little discriminative scaffolding, making performance more sensitive to whether a given round happens to draw easily or poorly separated categories.

Overall level difference. Averaged across all six variants, Kimi-K2.5 achieves a mean top-1 accuracy of 45.0% versus 37.4% for Gemini, a 7.6 percentage point advantage that holds on every individual variant. This model-level gap is the single largest source of variance in the ablation, illustrating that architecture matters more than prompt design within the range of strategies tested here.

Prompt sensitivity is strongly model-dependent. The two models disagree on the optimal prompt variant: Gemini achieves its highest top-1 accuracy under Baseline (41.3%), while Kimi peaks under Interleaved (48.7%). The total span from best to worst variant is 6.3 percentage points for Gemini (41.3% vs 35.0%) and 9.7 percentage points for Kimi (48.7% vs 39.0%), indicating that Kimi is more sensitive to prompt structure within this dataset and condition. No single variant is consistently optimal across both backends.

Chain-of-thought decomposition. The CoT contribution (measured as Baseline minus Direct accuracy) is sign-reversed across the two models: +6.3 percentage points for Gemini and −3.3 for Kimi. For Gemini, the two-stage describe-then-classify pipeline plausibly reduces confirmation bias when categories are visually ambiguous^[45,48]; for Kimi, the added verbosity introduces noise rather than precision at this category count. The sign reversal is sufficient reason not to treat prompt strategies validated on a single architecture as transferable defaults.

Contrastive elimination. The Contrastive variant trails Baseline by 4.0 percentage points for Gemini, suggesting that forced contradiction enumeration imposes generation overhead without a proportional discriminative return at K = 10. For Kimi the direction is reversed: Contrastive exceeds Baseline by 5.7 percentage points, consistent with contrastive priming theory^[46,47] that explicitly surfacing contradictions tightens category boundaries in high inter-class similarity settings.

Instruction placement. Co-located per-category instructions (Interleaved) cost Gemini 5.7 percentage points relative to the global preamble (Baseline), suggesting that global task context is more useful than local cues for this model. The direction reverses for Kimi: Interleaved outperforms Baseline by 6.3 percentage points, the largest single-step gain for either model, pointing to enhanced feature binding when discriminative cues appear alongside their corresponding reference images. Deferred instruction placement (Instructions-Last) follows the same sign-reversal pattern: −3.0 percentage points for Gemini relative to Baseline, but +4.0 for Kimi, suggesting that Kimi benefits from accumulating the full visual context before committing to a task frame. The cross-model spread is largest on Interleaved (13.0 percentage points) and smallest on Baseline (1.0 percentage point), making Baseline the most robust default across the two backends evaluated here.

High-variance variants and top-2 accuracy. Direct and Minimal produce the highest standard deviations (17%–22%), reflecting their minimal scaffolding rather than output failures. Notably, their top-2 accuracies are competitive with or exceed those of the structured variants (64.6% and 65.0% for Gemini; 68.9% and 69.2% for Kimi), indicating that both models generate informative visual feature comparisons even without explicit reasoning guidance; the discriminative deficit lies in the final category selection step rather than in feature extraction.

Lower bound and total prompt engineering contribution. The gap between each model's best-performing variant and Minimal quantifies the ceiling benefit of structured prompt engineering: 4.3% for Gemini (Baseline over Minimal) and 9.7% for Kimi (Interleaved over Minimal). These figures are upper bounds within this exploratory, single-dataset design.

4.4.   Major challenges in ultra-FGVC with MLLMs

Three fundamental challenges currently limit MLLM deployment in Ultra-FGVC tasks.

Visual encoder resolution. The most pervasive challenge exposed across all three tasks is the representational ceiling of current CLIP-based visual encoders. The performance gap between MLLMs and fine-tuned supervised methods widens sharply as $ K $ grows (from a modest deficit at $ K = 3 $ to a 12–34 point deficit at $ K = 10 $), and the preliminary $ K = 50 $ GPT-5.4 data (20.4% and 12.4% on SoyLocal and Cotton80 respectively) are consistent with continued degradation at benchmark scale. This gap is not reduced by richer prompt structure (Task 3). The persistent corvid accuracy floor in Task 2, which does not improve meaningfully under named conditions, is consistent with a visual representational limitation, though the statistical design of Task 2 is insufficient to establish this definitively: the possibility of a moderate semantic benefit that current sample sizes cannot detect remains open. Current CLIP encoders were trained on image-text pairs at coarse semantic granularity and lack the resolution to reliably separate inter-class feature distances at subspecies level^[19].

Output generation reliability. Although all Task 3 rounds produced parseable output under the current parser implementation, structured-output compliance remains a latent deployment risk. The elevated variance on Direct and Minimal ($ \sigma \approx 17 $%–22%) shows that discriminative performance is sensitive to prompt scaffolding, and the schema compliance rates observed here reflect a parser that was hardened iteratively against model outputs. In broader deployment scenarios (different models, longer category lists, or schema changes), constrained decoding mechanisms or function-calling interfaces that enforce JSON schema at generation time remain advisable to eliminate compliance as a confound and ensure reproducible inference pipelines.

Confidence miscalibration. All four models display systematic overconfidence across Tasks 1 and 2, with mean reported confidence of 0.74–0.91 irrespective of accuracy level. This precludes the use of raw model confidence as a deployment rejection threshold or reliability signal in precision agriculture workflows where misclassification has operational consequences. For Gemini, the miscalibration extends across all six Task 3 prompt variants (0.85–0.89) and appears to be an intrinsic property of frontier model confidence estimation in discriminative tasks with subtle inter-class distances. Whether the same holds for Kimi-K2.5 in Task 3 cannot be determined from the current results, as confidence statistics for Kimi were not captured in the Task 3 experimental run.

Several concrete technical directions are well-positioned to address the gaps documented here.

Richer visual representations. Both Task 1 and Task 2 point consistently to the visual encoder as a plausible primary bottleneck, though Task 2's statistical design is insufficient to conclusively rule out a role for label semantics at higher $ K $. Replacing CLIP-based encoders with part-aware or region-grounded backbones^[19] could increase the effective resolution at which fine-grained morphological differences are captured. Augmenting in-context exemplars with model-generated diagnostic captions via the describe-then-classify pipeline^[45,50] may raise the visual resolution ceiling without requiring task-specific fine-tuning. The FineR framework^[50], which extracts part-level visual attributes via VQA and reasons over them with world knowledge, is one concrete instantiation of this direction that could be adapted to cultivar-level agricultural classification.

Scale-up and fine-tuning. Should larger annotated datasets become available, supervised fine-tuning of the vision encoder, or lightweight parameter-efficient adaptation (e.g. LoRA) applied to the cross-modal connector, could significantly raise the accuracy ceiling documented here. The performance gap between MLLM few-shot ICL and fully supervised methods at K = 10 provides a clear and quantified target. Many-shot ICL^[43], which scales log-linearly with demonstration count up to 2,000 shots in frontier models, represents a near-term option that does not require retraining and could be combined with kNN-based exemplar selection^[53] to maximise the information content of each added shot.

Prompt robustness and extended ablation. Task 3 exposes sign-reversed responses to all four tested design dimensions across the two backends, revealing that architecture effects dwarf prompt effects in this setting. Extending the ablation to all four backends and the remaining datasets, with confidence statistics collected for each model, would substantially sharpen these conclusions. The elevated variance on unstructured variants (Direct, Minimal) also warrants investigation at higher round counts to narrow confidence intervals on the lower-bound estimates.

5.2.   Outlook for future work

Several broader directions beyond the current scope merit systematic investigation.

Extended model and dataset coverage, including K = 50. The preliminary GPT-5.4 results at K = 50 establish that the scale is technically within reach for at least one frontier model. Future work should pursue systematic K = 50 evaluation across all four backends, via retrieval-augmented reference selection, multi-turn context management, or model-specific API adaptations, to enable the first complete MLLM–supervised comparison at benchmark scale.

Confidence calibration. All four models display systematic overconfidence (mean scores 0.74–0.91 irrespective of accuracy), precluding raw confidence from serving as a deployment rejection threshold. Post-hoc calibration methods such as temperature scaling or conformal prediction are prerequisites for operational deployment. Conformal prediction is particularly well-suited to the data-scarce conditions of Ultra-FGVC given its distribution-free coverage guarantees.

Many-shot and retrieval-augmented classification. Given the log-linear scaling of accuracy with shot count in frontier models^[43] and the strong effect of exemplar selection^[53,55], retrieval-augmented pipelines that dynamically select maximally informative exemplars may substantially close the gap to supervised methods without fine-tuning, while simultaneously addressing the K = 50 feasibility constraint. More broadly, active learning and semi-supervised methods that exploit the large pools of unlabelled field imagery already captured could substantially reduce the per-category annotation burden that currently bottlenecks Ultra-FGVC dataset construction. Combining such approaches with the few-shot ICL capabilities documented here offers a path toward operational pipelines that require only minimal expert labelling per new cultivar or pest species.

Statistical power in Task 2 follow-up. A conclusive test of the visual-primacy hypothesis requires K = 9 with n ≥ 30 rounds to achieve > 80% power for a medium effect size ( d = 0.50), extended to all four evaluated backends. The CUB-Cormorant design should be replicated with larger n and, if possible, extended to a K = 9 cormorant-analogous subset to avoid ceiling effects at higher category counts.

Accurate subspecies-level classification of crop cultivars and pest species is a prerequisite for targeted interventions in precision agriculture, yet it sits at the intersection of two compounding difficulties: the discriminative features that separate cultivars are highly localised and morphologically subtle, and the labelled data available per category at this granularity is too scarce to sustain standard supervised learning. MLLMs, which encode rich biological knowledge from web-scale training and adapt at inference time through in-context learning, are a principled candidate for closing this gap without retraining. Whether they actually do so, and exactly where they fail, has not previously been characterised at Ultra-FGVC scale.

This paper provides that characterisation through a controlled three-task evaluation of four state-of-the-art MLLMs across three Ultra-FGVC benchmarks. Task 1 establishes that MLLMs are practically viable for low-$ K $ field triage ($ K = 3 $), exceeding human expert accuracy on SoyLocal and matching it on Cotton80, but fall sharply below fine-tuned supervised methods at $ K = 10 $. Further still, in the preliminary $ K = 50 $ evaluation, tracing a degradation trajectory consistent with a visual encoder bottleneck rather than a prompting or in-context-learning failure. Task 2 finds no statistically reliable accuracy gain from disclosing real category names in place of numeric identifiers across any of the 12 evaluated conditions, though the design is underpowered at $ K = 9 $ and inconsistent effect directions across models at $ K = 3 $ preclude a strong visual-primacy conclusion. Task 3 shows that prompt-design effects are strongly model-dependent, with sign-reversed responses to every tested design dimension between the two evaluated backends, cautioning against treating prompt strategies validated on a single architecture as transferable defaults.

All three tasks converge on the same primary constraint: the representational ceiling of current CLIP-based visual encoders, which is insensitive to richer prompt structure and largely insensitive to label semantics within the conditions tested. Output generation reliability and confidence miscalibration are secondary challenges that compound the representational gap in deployment settings. Each has a concrete technical path forward (richer visual backbones, constrained decoding, and post-hoc calibration), and the quantified performance gaps reported here provide explicit targets against which progress on those directions can be measured.