Note: The source code for this project is split across two repositories to enforce a strict separation of concerns. You can view the Frontend Repository here, which also contains links and instructions for the Backend Repository.

Overview

This project is an enterprise-grade, full-stack library management system that modernizes traditional library catalogs. Instead of relying purely on exact title matching (OPAC), this system builds an Agentic Retrieval-Augmented Generation (RAG) pipeline. It dynamically ingests book metadata from multiple academic APIs, enriches it using LLMs, vectorizes it, and provides a conversational AI interface for users to query the library's physical and digital knowledge base.

Technology Stack

Frontend (User Interface & Integration)

Framework: Next.js (App Router), React
Styling: Tailwind CSS (Vanilla CSS focus, modern glassmorphism and bento grids)
Icons: Lucide-React
State Management: React Hooks (useState, useEffect)
API Communication: Native fetch with REST endpoints

Backend (Core Logic & Queueing)

Framework: NestJS (Node.js, TypeScript)
Task Scheduling: @nestjs/schedule (Cron jobs for rate-limit protection)
File Parsing: multer (File Interceptors) & csv-parser
Security: @nestjs/throttler (Rate Limiting), custom API Key Guards, JWT/bcrypt (Auth scaffolded)

Database & Vector Storage

Database: Neon Serverless PostgreSQL
ORM: Drizzle ORM (drizzle-kit for migrations)
Vector Database: pgvector extension for storing and querying semantic embeddings
Schema: UUID primary keys, JSONB for unstructured metadata, Enum statuses.

AI & Enrichment APIs

LLM Engine: Google Gemini API (gemini-3.5-flash for synthesis, text-embedding-004 for vectors)
Metadata Sources:
- Google Books API (Descriptions, Authors)
- OpenLibrary API (Table of Contents)
- Semantic Scholar Graph API (Academic Abstracts, Citations, Fields of Study)

The RAG Flow & Core Mechanics

The true power of this system lies in its ingestion pipeline and hybrid search mechanics.

1. Multi-API LLM Enrichment (The Ingestion Pipeline)

When a book is ingested (either manually or via CSV), the CatalogService executes a highly choreographed data fetch:

Google Books API: Fetches the foundational metadata (Title, Author, Description).
OpenLibrary API: Attempts to fetch the book's Table of Contents.
Semantic Scholar API: If academic, fetches the paper abstract, citation counts, and fields of study.
LLM Synthesis: RagService.synthesizeEnrichment() injects all raw JSON data (and any custom CSV columns) into gemini-3.5-flash. The LLM reads the disparate data and writes a dense, cohesive semantic profile for the book.
Vectorization: The enriched profile is split into semantic paragraphs (chunks). RagService.generateEmbedding() hits the Gemini embedding model, converting chunks into floats.
Storage: The parent metadata is saved to documents, and the vectors are saved to document_chunks in Postgres using pgvector.

2. Rate-Limited Bulk CSV Ingestion

Because the Gemini Free Tier limits requests to 15 Requests Per Minute (RPM), synchronously processing a CSV of 5,000 books would instantly crash the system.

The Queue: Uploading a CSV (POST /catalog/upload/csv) parses the file and inserts every row into the ingestion_queue table as PENDING.
The Worker: CatalogWorker runs a Cron job (@Cron('*/5 * * * * *')) every 5 seconds. It picks the oldest PENDING job, processes it through the pipeline, and marks it COMPLETED. This mathematically caps the system at 12 RPM, running safely 24/7.

3. Hybrid Semantic Search

When a user asks a question in the /chat sandbox:

RagService.hybridSearch(query) is triggered.
Vector Search: Converts the user query into an embedding and calculates the cosine distance (<=>) against all document_chunks in Postgres.
Full-Text Search: Falls back to text matching (ILIKE) on the documents table to catch exact title/author matches.
Context Injection: Fuses the top N results into a prompt template: "You are an expert librarian... Use the following catalog context to answer the user..."
Generation: Gemini reads the contextual chunks and provides a highly accurate, hallucination-free answer with specific book recommendations.

Reverse Engineering & Future Scaling

For a developer taking over this project:

Changing Vector DBs: All vector logic is isolated in src/rag/rag.service.ts and src/database/schema/chunks.ts. You can swap pgvector for Pinecone or Milvus by altering these two files.
Adding MARC 21 Support: The ingestion_queue schema relies on an isbn and customMetadata. To add binary .mrc support, simply add a POST /upload/marc endpoint, use marcjs to parse the ISBNs, and dump them into the same Postgres queue. The CatalogWorker will handle the rest.
Auth: The auth module exists but is purposefully detached from controllers for rapid local testing. Secure the endpoints by enabling @UseGuards(JwtAuthGuard) on the catalog controllers.