28S fetching failing

[claumer]

Hi again @SchistoDan ,

Having some trouble using gene-fetch to download 28S records.

For instance, we have https://www.ncbi.nlm.nih.gov/nuccore/KC869866.1

When I do:

gene-fetch -e [myemail] -k [mykey] --header detailed -ms 2000 -ns 2000 -t nucleotide -c -s 6170 -g 28S -o 28S_prorhynchida

It really should be picking this (and many other sequences up, but instead I get:

====== Starting Gene Fetch ======
Version: 1.0.20
Written by Dan Parsons & Ben Price, Natural History Museum London

Validating NCBI credentials: email='myemail', api_key='[mykey]'
Credential validation passed
2025-12-10 16:55:27,084 [INFO] Logging initialised. Log file: 28S_prorhynchida/gene_fetch.log
2025-12-10 16:55:27,084 [INFO] Single-taxid mode activated: using protein size threshold 500 and nucleotide size threshold 2000
2025-12-10 16:55:27,084 [INFO] Using generic search terms for lsu
2025-12-10 16:55:27,084 [INFO] Output directory: 28S_prorhynchida
2025-12-10 16:55:27,084 [INFO] Sequence type: nucleotide
2025-12-10 16:55:27,084 [INFO] Single-taxid mode activated for taxid: 6170
2025-12-10 16:55:27,084 [INFO] Maximum number of sequences to fetch: 2000
2025-12-10 16:55:27,084 [INFO] Retrieving taxonomy details from NCBI for taxID: 6170
2025-12-10 16:55:29,155 [INFO] Successfully retrieved taxonomy information from NCBI for taxID: 6170
2025-12-10 16:55:29,155 [INFO] Attempting search at family level: Prorhynchidae (taxid: 6170)
2025-12-10 16:55:29,155 [INFO] Searching nucleotide database at rank family (Prorhynchidae) with term: (lsu[Title] OR lsu[Gene] OR "lsu"[Protein Name]) AND txid6170[Organism:exp] AND 2000:60000[SLEN]
2025-12-10 16:55:30,199 [WARNING] No sequences found
2025-12-10 16:55:30,200 [WARNING] No sequences found for taxid 6170
2025-12-10 16:55:30,200 [INFO] Single taxid processing completed

I think it's a similar problem to what you were having with the 18S leading to the new version. The terms lsu[Title] OR lsu[Gene] OR "lsu"[Protein Name] just aren't inclusive enough - we should have "28S" at least in there as well, and maybe "large subunit".

There will also presumably be analogous problems when people are searching 12S and 16S mtDNA rRNAs, right?

Regards,
Chris L

[SchistoDan]

Hi @claumer

Apologies for the delay in getting back to you on this - I was stumped for a while. Although my searches were also returning "No sequences found", the search term being constructed was different from the one in the log snippet you shared.
When running the command you shared, I get:

Searching nucleotide database at rank family (Prorhynchidae) with term: (28S ribosomal RNA[Gene] OR 28S rRNA[Gene] OR rrn28[Gene] OR LSU rRNA[Gene] OR 28S ribosomal RNA[rRNA]28S rRNA[rRNA] OR LSU rRNA[rRNA] OR rrn28[rRNA]) NOT methylase[Title] NOT methyltransferase[Title] NOT pseudouridylate[Title] NOT synthase[Title] AND txid6170[Organism:exp] AND 2000:60000[SLEN]

And after some testing, I think the command you shared might not correspond to the log snippet - the command shows -g 28S, whilst the logging shows Using generic search terms for lsu. If I run the same command you shared but with -g lsu, I do get the same generic search term as in your log snippet:

Searching nucleotide database at rank family (Prorhynchidae) with term: (lsu[Title] OR lsu[Gene] OR "lsu"[Protein Name]) AND txid6170[Organism:exp] AND 2000:60000[SLEN]

As lsu wasn't listed in the '_target_aliases' dictionary in core.py, running -g lsu resulted in a fallback to using the generic 'gene_search_term' string (line 528 of core.py) - this was an oversight on my part and searching -g lsu now correctly maps to 28s search term. Similarly, I added ssu, so searching -g ssu now correctly maps to 18s search terms.

This also highlighted another issue... You may have noticed in the 28S search term I mentioned above is malformed, where 28S ribosomal RNA[rRNA]28S rRNA[rRNA] should include an 'OR' operator between the terms. This was resulting in no sequences being found even with -g 28S (or 28s) in the command you shared. I've fixed this, run your command, and now get:

====== Starting Gene Fetch ======
Version: 1.0.21
Written by Dan Parsons & Ben Price, Natural History Museum London

Validating NCBI credentials: email='d.parsons@nhm.ac.uk', api_key='1866f9734a06f26bc5895a84387542ac9308'
Credential validation passed
2025-12-12 15:27:38,727 [INFO] Logging initialised. Log file: 28S/gene_fetch.log
2025-12-12 15:27:38,728 [INFO] Single-taxid mode activated: using protein size threshold 500 and nucleotide size threshold 2000
2025-12-12 15:27:38,728 [INFO] Using rRNA search terms for 28s
2025-12-12 15:27:38,729 [INFO] Output directory: 28S
2025-12-12 15:27:38,729 [INFO] Sequence type: nucleotide
2025-12-12 15:27:38,730 [INFO] Single-taxid mode activated for taxid: 6170
2025-12-12 15:27:38,730 [INFO] Maximum number of sequences to fetch: 2000
2025-12-12 15:27:38,731 [INFO] Retrieving taxonomy details from NCBI for taxID: 6170
2025-12-12 15:27:40,117 [INFO] Successfully retrieved taxonomy information from NCBI for taxID: 6170
2025-12-12 15:27:40,118 [INFO] Attempting search at family level: Prorhynchidae (taxid: 6170)
2025-12-12 15:27:40,118 [INFO] Searching nucleotide database at rank family (Prorhynchidae) with term: (28S ribosomal RNA[Gene] OR 28S rRNA[Gene] OR rrn28[Gene] OR LSU rRNA[Gene] OR 28S ribosomal RNA[rRNA] OR 28S rRNA[rRNA] OR LSU rRNA[rRNA] OR rrn28[rRNA]) NOT methylase[Title] NOT methyltransferase[Title] NOT pseudouridylate[Title] NOT synthase[Title] AND txid6170[Organism:exp] AND 2000:60000[SLEN]
2025-12-12 15:27:40,950 [INFO] Found 16 nucleotide sequence IDs
2025-12-12 15:27:40,951 [INFO] Processing 16 nucleotide IDs
2025-12-12 15:27:40,952 [INFO] Attempting to fetch nucleotide sequence (GI:588313908)
2025-12-12 15:27:41,477 [INFO] Successfully fetched nucleotide sequence of length 3386 (accession KC869866.1)
2025-12-12 15:27:41,478 [INFO] Looking for 28s rRNA features in record KC869866.1
2025-12-12 15:27:41,478 [INFO] Searching for these rRNA name variations in feature table: {'lsu', '28s', 'large subunit', 'l-rrna', 'rrn28'}
2025-12-12 15:27:41,479 [INFO] Found matching rRNA feature with product: 28s ribosomal rna
2025-12-12 15:27:41,482 [INFO] Successfully extracted rRNA of length 3386
2025-12-12 15:27:41,483 [INFO] Successfully extracted 28s rRNA feature of length 3386 from KC869866.1
2025-12-12 15:27:41,483 [INFO] Progress: 1/2000 sequences processed
2025-12-12 15:27:41,484 [INFO] Attempting to fetch nucleotide sequence (GI:588313907)
2025-12-12 15:27:42,000 [INFO] Successfully fetched nucleotide sequence of length 3378 (accession KC869865.1)
2025-12-12 15:27:42,001 [INFO] Looking for 28s rRNA features in record KC869865.1
2025-12-12 15:27:42,002 [INFO] Searching for these rRNA name variations in feature table: {'lsu', '28s', 'large subunit', 'l-rrna', 'rrn28'}
... + 15 additional 28S sequences

I've expanded the name variations (rRNA_alternatives in seqeunce_processor.py, line 1784) for all rRNA targets to be more flexible and inclusive - I hope that solves the current issue and pre-empts similar issues that may have arisen down the line.

Please let me know when you've updated (to v1.0.12) and if the run completes succesfully this time! And of course, jsut let me know if you hit any other issues - thanks for highlighting another necessary improvement!

All the best,
Dan