Inconsistent StopIteration error related to batch size vs. training dataset size across datasets

[imb-ai-junghwan]

Hi,
Based on my previous experience, I understood that the StopIteration error was related to the relationship between the training dataset size and the batch size. Specifically, when the training dataset size was smaller than 64, I set the batch size to 16 (a modified option), and for larger datasets I used the default batch size. This approach had worked reliably for me in the past.
However, when running experiments on a total of 36 datasets this time, the behavior was no longer consistent with what I expected. Even though I applied the same rule as before, some datasets completed successfully while others failed with a StopIteration error.
At this point, I cannot identify a clear pattern that explains why certain datasets succeed and others fail under the same batch size logic.
I would appreciate any insight into:

What underlying conditions might cause this inconsistent behavior

Whether there are additional factors beyond dataset size that could trigger this error

Recommended best practices for setting batch size in relation to dataset size to avoid StopIteration
I will include the details of my datasets and the error messages below.

Thank you for your help.

Dataset and results:
Dataset | N_healthy_train | N_disease_train | N_healthy_test | N_disease_test | N_train | N_test | GDmicro Option | Run
1 | 66 | 97 | 8 | 11 | 163 | 19 | default | Fail
2 | 497 | 34 | 56 | 4 | 531 | 60 | default | Fail
3 | 57 | 135 | 7 | 16 | 192 | 23 | default | Fail
4 | 49 | 60 | 6 | 7 | 109 | 13 | default | Fail
5 | 57 | 75 | 7 | 9 | 132 | 16 | default | Fail
6 | 262 | 60 | 30 | 7 | 322 | 37 | default | Fail
7 | 279 | 33 | 32 | 4 | 312 | 36 | default | Fail
8 | 187 | 188 | 21 | 21 | 375 | 42 | default | Fail
9 | 262 | 257 | 30 | 29 | 519 | 59 | default | Fail
10 | 279 | 266 | 32 | 30 | 545 | 62 | default | Fail
11 | 78 | 75 | 9 | 9 | 153 | 18 | default | Fail
12 | 48 | 66 | 6 | 8 | 114 | 14 | default | Fail
13 | 164 | 168 | 19 | 19 | 332 | 38 | default | Fail
14 | 94 | 136 | 11 | 16 | 230 | 27 | default | Fail
15 | 193 | 312 | 22 | 35 | 505 | 57 | default | Fail
16 | 87 | 85 | 10 | 10 | 172 | 20 | default | Fail
17 | 86 | 89 | 10 | 10 | 175 | 20 | default | Fail
18 | 135 | 57 | 16 | 7 | 192 | 23 | default | Fail
19 | 153 | 192 | 18 | 22 | 345 | 40 | default | Fail
20 | 128 | 148 | 15 | 17 | 276 | 32 | default | Fail
21 | 90 | 67 | 10 | 8 | 157 | 18 | default | Fail
22 | 497 | 195 | 56 | 22 | 692 | 78 | default | Success
23 | 56 | 42 | 7 | 5 | 98 | 12 | default | Success
24 | 56 | 41 | 7 | 5 | 97 | 12 | default | Success
25 | 27 | 27 | 3 | 3 | 54 | 6 | modified | Success
26 | 36 | 35 | 4 | 4 | 71 | 8 | default | Success
27 | 23 | 23 | 3 | 3 | 46 | 6 | modified | Success
28 | 18 | 16 | 2 | 2 | 34 | 4 | modified | Success
29 | 19 | 38 | 3 | 5 | 57 | 8 | modified | Success
30 | 27 | 38 | 4 | 5 | 65 | 9 | default | Success
31 | 38 | 44 | 5 | 5 | 82 | 10 | default | Success
32 | 16 | 17 | 2 | 2 | 33 | 4 | modified | Success
33 | 38 | 46 | 5 | 6 | 84 | 11 | default | Success
34 | 27 | 28 | 3 | 4 | 55 | 7 | modified | Success
35 | 9 | 12 | 2 | 2 | 21 | 4 | modified | Success
36 | 46 | 37 | 6 | 5 | 83 | 11 | default | Success

Error message:
Traceback (most recent call last):
File "./GDmicro/run_MLP_embedding_da.py", line 224, in build_graph_mlp
tgt_data,_=tgt_iter.next()
File "./gdmicro/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 652, in next
data = self._next_data()
File "./gdmicro/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 691, in _next_data
index = self._next_index() # may raise StopIteration
File "./gdmicro/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 642, in _next_index
return next(self._sampler_iter) # may raise StopIteration
StopIteration

[liaoherui]

Hi, sorry about this issue.

If it’s convenient, could you share some of the datasets that failed? I’ll use them to debug the code and figure out what’s going on and how to fix it. Thank you so much!

[imb-ai-junghwan]

Thank you for your response.
I’m planning to send one of the datasets that failed via email.
Could you please confirm if I should send it to heruiliao2-c@my.cityu.edu.hk

Thank you in advance.

[liaoherui]

Yes, Either heruiliao2-c@my.cityu.edu.hk or herui728@mit.edu is fine, thanks!

[imb-ai-junghwan]

Ok. I'll send email to 'herui728@mit.edu'. thanks!

[liaoherui]

Thanks so much for sharing your datasets! I tested the pipeline using them and found that the issue is caused by the batch size setting. By default, the training and prediction steps require at least 64 samples (-b 64) in both the training and test sets. If the number of samples in either set is smaller than the specified batch size, this error will occur.

To clarify, I tested the datasets you sent using the command lines below, and they all worked correctly. I recommend setting the batch size to a value smaller than the minimum number of samples in your training or test sets.

Example 1: gdmicro_cn_abundances.csv

n_healthy_train: 128
n_disease_train: 148
n_healthy_test: 15
n_disease_test: 17

The minimum sample count is n_healthy_test = 15, so I used -b 8:
gdmicro -i gdmicro_cn_abundances.csv -d non_healthy -o cn_test -r 0 -b 8

Example 2: gdmicro_hk_abundances.csv

n_healthy_train: 48
n_disease_train: 66
n_healthy_test: 6
n_disease_test: 8

The minimum sample count is n_healthy_test = 6, so I used -b 4:
gdmicro -i gdmicro_hk_abundances.csv -d non_healthy -o hk_test -r 0 -b 4

Please let me know if you have any questions or if the issue persists. I’m happy to help further!

[imb-ai-junghwan]

Thank you very much for your helpful documentation and resources.
I would like to confirm one specific detail regarding the following statement:

"By default, the training and prediction steps require at least 64 samples (-b 64) in both the training and test sets. If the number of samples in either set is smaller than the specified batch size, this error will occur.”

Could you kindly let me know which script should I refer to in order to check the source code related to this part??

I would greatly appreciate your guidance on this matter.

[liaoherui]

Sure, this one -> "GDmicro.py".

[imb-ai-junghwan]

Thank you for the information.
However, when I reviewed GDmicro.py, I couldn't find the relevant part.
If possible, could you let me know which line number I should check?

Thanks!

[liaoherui]

line 1183

[imb-ai-junghwan]

I have a follow-up question regarding your explanation that an error can occur when the number of training or testing samples is smaller than the batch size. (If the number of samples in either set is smaller than the specified batch size, this error will occur.)
In my case, the datasets numbered 23 through 36, which I previously shared, had fewer training or testing samples than the batch size. Despite this, the code ran without any issues.
Because of this, I suspect that there may be another underlying reason for the behavior we observed. Could you please help me look into this possibility?
Sincerely,

[liaoherui]

Sorry for late response.

The StopIteration occurs in the domain adaptation branch (run_MLP_embedding_da), where loaders are created with drop_last=True. If a split has fewer samples than batch_size, that loader can become length 0, and repeated next() calls raise StopIteration.

However, even with identical CLI parameters, the code path can differ because domain adaptation is enabled only when both conditions are true: doadpt==1 and len(test_idx)>12. In the command, doadpt defaults to 1, but test_idx depends on how many samples are labeled as test in metadata. If test samples are 12 or fewer, the pipeline switches to the non-domain-adaptation branch automatically. As a result, some datasets (including small ones) may run fine because they take the non-domain-adaptation branch (run_MLP_embedding instead of run_MLP_embedding_da), which does not use drop_last=True in the same way.

[imb-ai-junghwan]

Thank you for your response.
Thanks to your explanation, I have found a way to resolve the issue I am currently facing.