NLP & Vector Search: Extracting Knowledge from Historical Scientific Literature

Executive Summary

Historical taxonomic literature contains scientific knowledge that is difficult to search and analyze computationally. Species nomenclature and descriptions may be scattered across scanned journal issues, inconsistent layouts, and specialized passages mixing Latin and English. Converting these documents into searchable records requires more than extracting text: it requires identifying species names, associating them with diagnostic descriptions, and preserving their sources.

The Synoptic Key of Life (SKOL) project approaches this problem through two connected capabilities: semantic retrieval of structured taxonomic records and an upstream pipeline for segmenting and classifying unannotated literature. The architectural decision was to prototype the downstream retrieval experience before scaling upstream processing, so that extraction requirements could be informed by the representation the search system actually needed.

I participated as Pair-Programming Pipeline Engineer & Communications Lead, working with project owner, architect and originator La Monte Henry Piggy Yarroll and co-contributor David Caspers. Together, we integrated retrieval components, developed or extended document-processing utilities, and evaluated feature representations and classifiers for taxonomic text. The work illustrates practical tradeoffs in data contracts, distributed text processing, sparse-feature modeling, and system integration.

My Contributions

Role: Pair-Programming Pipeline Engineer & Communications Lead.

Specific contribution areas: Documentation and communication of architecture and implementation. Authoring and presenting communications decks and visual guides. Pair programming of data preparation utilities, pipeline compoents, as well as training, testing and evaluation of classifier models.

---

System Overview and Engineering Scope

The source collection includes more than 1,200 historical journal issues and an estimated 10,000 species descriptions. Before the automated pipeline work, approximately 190 issues had been manually annotated using the YEDDA format, with paragraph labels for Nomenclature, Description, and Misc-exposition. These annotations provided a basis for prototyping and classification experiments.

The remaining collection represented a potential automation target of roughly 1,000 additional issues and 7,000 articles.

We organized the solution into two related phases:

1. Part 1 — Downstream semantic retrieval: Structure taxonomic records, encode text as embeddings, and search for relevant descriptions using vector similarity.
2. Part 2 — Upstream classification and ingestion: Segment unannotated text, classify paragraph functions, and prepare outputs that could feed the structured retrieval representation.

---

Inverting the Pipeline: Prototyping Downstream First

In data engineering, investing in complex pipelines before defining downstream data requirements can create significant and costly re-work. We reduced that risk by testing the consumption side first: what should a useful taxonomic record contain, and could a search application retrieve it in a meaningful way?

This work brought together two existing systems:

1. SKOL: The annotation and classification framework originated by La Monte Yarroll.
2. MycoSearch: A specialized literature-retrieval application forked from dr-drafts, originally developed by Dr. Nick Gisolfi at Carnegie Mellon University’s Auton Lab.

Establishing the Data Contract (Taxon Schema)

In part 1, we developed a standardized Taxon python class in the SKOL application to act as our data contract. This class normalized disparate document structures into a structured serializable schema with the following attributes:

• Nomenclature: Genus, species, author, year, and taxonomic rank.
• Morphological Description: Macroscopic and microscopic diagnostic traits (cap shape, spore size, cystidia, ecology).
• Citation & Metadata: Journal, volume, empirical page numbers, and plates.

The design of the Taxon class enabled clean and streamlined separation of specieies identity, descriptive evidence, and source provenance, which enables the retrieval system to reliably return a traceable context similarity score.

The team then further extended the SKOL application to add a document ingestion utility to read annotated articles and create taxon objects for each unique species and description found.

# Illustrative, abridged schema
class Taxon:
    def __init__(
        self,
        nomenclature: str,
        description: str,
        citation: str,
        metadata: dict | None = None,
    ):
        self.nomenclature = nomenclature
        self.description = description
        self.citation = citation
        self.metadata = metadata or {}

    def to_embedding_payload(self) -> str:
        return f"{self.nomenclature}\n\n{self.description}"

Semantic Retrieval Architecture

The prototype encoded structured descriptions as dense embeddings using the all-mpnet-base-v2 sentence-transformer model. In the intended retrieval flow:

1. A user’s descriptive query is converted into an embedding.
2. The query vector is compared with precomputed record embeddings using cosine similarity.
3. Candidate matches are returned with scores and available citations or source links.

This prototype established the feasibility of the retrieval representation and gave the upstream team concrete extraction targets. It did not, on its own, prove that unannotated literature could be processed accurately at scale.

---

Upstream Automation: Distributed Processing and Classification

With a target record structure established, the second workstream addressed how to identify relevant content in the unannotated issues. The pipeline combined heuristic paragraph reconstruction, domain-aware text features, and supervised classification.

1. Reconstructing Paragraph Boundaries

Historical OCR text did not consistently preserve paragraph structure. We developed heuristics to infer boundaries from features such as nomenclature patterns, short trailing lines, and surrounding layout context. We also identified likely artifacts, including page headers, table separators, and figure or plate references.

This resulted in a stateful heuristic segmentation parser that combines regular expressions (for biological nomenclature detection) with geometric layout cues—such as line-length thresholds, control characters, and keyword-driven state flags to filter tables and figures. The processing approach grouped lines by source document, applied segmentation logic, and expanded candidate paragraphs into rows suitable for subsequent feature extraction.

# Conceptual pipeline outline
parsed_df = (
    raw_df.groupBy("filename")
    .agg(collect_list("value").alias("lines"))
    .withColumn("paragraph", explode(heuristic_paragraph_udf(col("lines"))))
    .filter(trim(col("paragraph")) != "")
)

To associate nomenclature with its description, La Monte Yarroll examined approximately 79,000 paragraphs and observed that descriptions often appeared within five to six paragraphs after a species name. That observation informed a proximity-based grouping heuristic, with the understanding that proximity alone does not guarantee correct association in every document layout.

Original line positions were retained and the collected values are subsequently sorted to prevent Spark aggregation and shuffling resulting in a reordering of rows.

2. Feature Engineering for Mixed-Language Taxonomic Text

Classifying taxonomic descriptions is complicated by specialized vocabulary and Latin inflectional endings. Rather than assuming a general-purpose word representation would be sufficient, we compared word-level features with features constructed from word suffixes.

The PySpark ML approach combined:

• Word TF-IDF: Token-level term frequency and inverse document frequency.
• Suffix features: A custom transformer extracting two-, three-, and four-character word endings as additional morphological signals.
• Feature assembly: Sparse vectors from both channels assembled into a combined feature set.

# Abridged illustration: omitted imports, parameter declarations, and schema checks.
class SuffixTransformer(Transformer):
    def _transform(self, df):
        suffixes_udf = udf(
            lambda words: [
                word[-n:]
                for word in words
                for n in (2, 3, 4)
                if len(word) >= n
            ],
            ArrayType(StringType()),
        )
        return df.withColumn(
            self.getOutputCol(), suffixes_udf(col(self.getInputCol()))
        )

pipeline = Pipeline(stages=[
    tokenizer,
    word_vectorizer, word_idf,
    suffix_transformer, suffix_vectorizer, suffix_idf,
    vector_assembler,
    label_indexer,
])

3. Classifier Evaluation and Interpretation

We evaluated multinomial logistic regression and random forest approaches for three paragraph labels: Nomenclature, Description, and Misc-exposition. The recorded comparison used an 80/20 training/test split.

Classifier	Features	Accuracy	Precision*	Recall*	F1*
Logistic regression	Word + suffix TF-IDF	94.21%	96.40%	96.30%	0.9419
Logistic regression	Word TF-IDF	94.18%	96.03%	96.66%	0.9414
Logistic regression	Suffix TF-IDF	93.96%	95.93%	96.49%	0.9392
Random forest	Word features	78.78%	78.78%	100.0%	0.6943

The combined word-and-suffix model reported the highest accuracy, but its margin over the word-only model was 0.03 percentage points (94.21% versus 94.18%). This is too small, without repeated experiments or uncertainty estimates, to establish a meaningful improvement. More notably, suffix-only features reported 93.96% accuracy, suggesting that morphology-derived features retained substantial information relevant to the classification task.

The pipeline was designed to output classified text in a tagged representation such as [@ ... #Description], providing an intermediate step toward constructing Taxon records and generating embeddings.

---

Technical Challenges and Architectural Decisions

Sparse Features Versus Dense Dimensionality Reduction

We explored principal component analysis (PCA) to reduce the dimensionality of TF-IDF representations exceeding 270,000 feature dimensions. In the reported notebook environment, dense conversion and PCA proved prohibitively expensive; experimental jobs remained incomplete after more than 24 hours.

Instead, we retained sparse features and used L2-regularized logistic regression (regParam=0.01). That approach reportedly converged in minutes and supported the strongest measured results among the alternatives tested. The lesson was not that PCA is categorically unsuitable, but that the transformation cost was difficult to justify for this feature representation and environment.

Preserving Sequence in Distributed Text Processing

Distributed processing does not inherently preserve document row order. For taxonomic literature, sequence is semantically important: a nomenclature entry may be followed by its corresponding morphological description.

The design therefore needed explicit source-position metadata and a deterministic reconstruction strategy for ordered paragraphs. Spark-generated row IDs are not a substitute for source order; they can provide identifiers without guaranteeing the original publication sequence.

Engineering for Collaboration

By designing the retrieval experience around a consistent taxonomic representation, the team could develop the ingestion and search workstreams with a shared understanding of the desired output. That separation also made it possible to evaluate classification techniques without continually changing the web-facing retrieval interface.

---

Next Steps

Since completing our initial two-phase pipeline and proving out 94% classification accuracy, the SKOL architecture has continued to mature under La Monte Yarroll’s stewardship. To explore the latest data models and active codebase, visit the SKOL GitHub repository.