Launch HN: Captain (YC W26) – Automated RAG for Files
12 hours ago (runcaptain.com)
Hi HN, we’re Lewis and Edgar, building Captain to simplify unstructured data search (https://runcaptain.com). Captain automates the building and maintenance of file-based RAG pipelines. It indexes cloud storage like S3 and GCS, plus SaaS sources like Google Drive. There’s a quick walkthrough at https://youtu.be/EIQkwAsIPmc.
We also put up this demo site called “Ask PG’s Essays” which lets you ask/search the corpus of pg’s essays, to get a feel for how it works: https://pg.runcaptain.com. The RAG part of this took Captain about 3 minutes to set up.
Here are some sample prompts to get a feel for the experience:
“When do we do things that don't scale? When should we be more cautious?” https://pg.runcaptain.com/?q=When%20do%20we%20do%20things%20...
“Give me some advice, I'm fundraising” https://pg.runcaptain.com/?q=Give%20me%20some%20advice%2C%20...
“What are the biggest advantages of Lisp” https://pg.runcaptain.com/?q=what%20are%20the%20biggest%20ad...
A good production RAG pipeline takes substantial effort to build, especially for file workloads. You have to handle ETL or text extraction, chunking, embedding, storage, search, re-ranking, inference, and often compliance and observability – all while optimizing for latency and reliability. It’s a lot to manage. grep works well in some cases, but for agents, semantic search provides significantly higher performance. Cursor uses both and reports 6.5%–23.5% accuracy gains from vector search over grep (https://cursor.com/blog/semsearch).
We’ve spent the past four years scaling RAG pipelines for companies, and Edgar’s work at Purdue’s NLP lab directly informed our chunking techniques. In conversations with dozens of engineers, we repeatedly saw DIY pipelines produce inconsistent results, even after weeks of tuning. Many teams lacked clarity on which retrieval strategies best fit their data.
We realized that a system to provision storage and embeddings, handle indexing, and continuously update pipelines to reflect the latest search techniques could remove the need for every team to rebuild RAG themselves. That idea became Captain.
In practice, one API call indexes URLs, cloud storage buckets, directories, or individual files. Under the hood, we’re converting everything to Markdown. For this, we’ve had good results with Gemini 3 Pro for images, Reducto for complex documents, and Extend for basic OCR. For embedding models, ‘gemini-embedding-001’ performed reasonably well at first, but we later switched to the Contextualized Embeddings from ‘voyage-context-3’. It produced more relevant results than even the newer Voyage 4 models because its chunk embeddings are encoded with awareness of the surrounding document context. We then applied Voyage’s ‘rerank-2.5’ as second-stage re-ranking, reducing 50 initial chunks to a final top 15 (configurable in Captain’s API). Dense embeddings are just half the picture and full-text search with RRF complete our hybrid retrieval. In the Captain API, these techniques are exposed through a single /query endpoint. Access controls can be configured via metadata filters, and page number citations are returned automatically.
The stack is constantly changing but the Captain API creates a standard interface for this. You can try Captain, 1 month for free, and build your own pipelines at https://runcaptain.com. We’re looking for candid feedback, especially anything that can make it more useful, and look forward to your comments!
I think I've lost count of how many of these start ups I've seen. But what I really cant fathom is that pricing which is completely out of band. You can already talk to files directly with gemini, just wrapping other apis etc makes no sense. This is even stuff now you can easily codegen entire solutions for esp object storage based ones. Don't see actual any value add or differentiators here. It's obviously not that secure, and ingestion pipeline/connectors are also commodity.
You're right that you can chat with files using Gemini or a codegen'd RAG pipeline, and that does work well for a lot of teams.
The problem that Captain really addresses comes when production pipelines need to run continuously over large file corpora with fast, incremental indexing, and reliable latency. The maintenance required in these situations is often quite significant.
Captain focuses specifically on making sure the retrieval layer can operate smoothly so folks don't have to scale & maintain the infrastructure themselves.
Congratulations on the launch, Edgar and Lewis! I tried out https://pg.runcaptain.com/ --- 1. I can't seem to select the text during text generation without it being deselected as soon as more text is generated.
2. It seems like it tries to emit citations, but doesn't emit proper links and instead just wrote [filename].
> one of the most common pieces of advice Y Combinator gives to startups [153_do_things_that_dont_scale.pdf].
Great name - Captain Hook recently hit public domain - would be a sick logo! (Disney stuff like red jacket still copywrited)
No way, that's awesome!
Good looking! I didn't get to watch the video or look at docs in depth, but do the results trace back to the location of the answers in a document? Let's say it finds an answer in a PDF, and I'd like to know where in that PDF the citation is. Is that possible or intended?
Great question, we have deterministic page # citations for PDF results and exact bounding box citations coming very soon.
If you want to check out the Query API response example, here's a link: https://docs.runcaptain.com/api-reference/query/collection-v...
Just a note on the website, I thought at first my browser had been hijacked by a shipping or travel agent. The first impression is how AI has improved ship tracking, so you can now track ships with 98% accuracy, with little to no hint this is AI infrastructure until you scroll down.
If you know what Captain is, this is not an issue. I closed the browser tab at first, thinking "what the hell is this, I don't give a damn about shipping forecasts"
Having tried this a bit I do really like the single api call for all of it.
I also appreciate transparent pricing but I am not 100% sure the sense of scale of costs. It could be helpful to give some ballparks on things for each of the plans. I'm not sure exactly what i could get out of a plan. My guess, trying hard to figure it out, was if i had about 1,000 pages of new/updated content per month, I would pay $295/month for unlimited queries on top of it. Is that roughly correct?
Yes, we don't charge for queries. For $295, you're able to index up to 1000 pages of new content per month into a fully queryable pipeline.
Advanced and Basic do play a difference though. Advanced is for complex graphics or charts in the documents submitted. Basic is sufficient for most document workloads.
This is cool, like qmd as a service with real-time integrations where it matters?
How do you handle more structured data like csv/xlsx/json? Would be cool if it were possible to auto-process links to markdown (e.g. youtube, podcast, arbitrary websites, etc) a la https://github.com/steipete/summarize (which can pull full text in addition to summarizing).
Thanks, we're just starting to optimize more for the semi-structured data. So far, we've been parsing tables into Markdown and running them through the contextualized embedding model with no overlap, taking advantage of how it strings together chunks. This isn't great for big files so we're exploring agentic exploration (slow but good for more structured numerical data) and automated graph creation (promising for more relational data).
Love the auto-process markdown idea, we'll add it to our roadmap :D
Its a nice thought & the outcome as a product. Why would an organization pay for such product, as they can very well build a RAG these days tuning to their business needs. I see that Captain API does everything in one shot without rebuilding the RAG, but why would an organization to pay for this solution as this entire chain of activity can be automated and run at non business hours as a batch with the fraction of a cost. What's the delta efficiency that captain would bring it to the table , have you done any benchmarks, if its negligible, i see no reason for any organization to use the captain
This is an interesting product, thanks for sharing. Can you elaborate on some of your competitors in this landscape and what you might do differently compared to each one?
Thanks! The largest alternative to Captain is folks trying to build file search themselves. As mentioned in the post, it is a lot to manage.
The most similar product I've seen is Vertex File Search. They're hosted inside of GCP which can fit nicely into existing cloud deployments. Captain indexes from more sources (like R2 for example) and anecdotally provides faster indexing.
What about OpenSearch, Onyx, Glean Search, Kore.AI, Sana Labs?
1 reply →
Congrats on the launch. Very quick feedback on the site, since I usually try to check out the blog section, https://www.runcaptain.com/blog layout is broken in mobile, I tried brave/chrome/Firefox.
Thanks, just shipped a fix ;)
Just some unfiltered feedback after checking out the website: from what I understand this is an SaaS only? So basically I’m asked to upload ALL company docs to a company that existed for basically a minute with some questionable SOC2 report. Soc2 is basically dead as a security artefact and the data asked to upload is sensitive by nature. I don’t see that working.
> Soc2 is basically dead as a security artefact
can you expand on that?
not to mention they are using 3p apis for everything.. gemini, reranking etc...
> spotty RAG
:O
Are you writing the integrations listed there, or is are you using something that manages the data connections?
We've built these integrations ourselves.
For larger enterprises that require governance and additional compliance, we've been relying on trusted partners to help establish a connection to Captain
The problem with these kind of tools now is that Codex is so good you can basically build something which is good for 99% of cases in a single day, and it's free...
Look at Tobi vibe-coding QMD, he's not a full-time engineer and vibed that up and now it's used as the defacto RAG engine for OpenClaw.
Yeah QMD is quite impressive! The main difference between us and them is the scale folks would be looking at indexing. The serverless ingestion engine I described in the post is optimized for processing large batch jobs with high concurrency. We depend on a lot of cloud compute for this which isn't something QMD's local-first environment is optimized for. That said, it's a great option for OpenClaw!
Funny you say that.
I spent the last two days building this exact thing for our internal use.
Managed to get a full RAG pipeline integrated and running with all of our company documents in less than two days work.
Chunking, embedding and querying, connected to S3 and Google Drive, and running on our own hardware (and scaling on AWS too if needed).
Interesting to see still solutions being developed for RAG. We developed a solution similar to yours: Automatic indexing from GDrive, SharePoint etc. and then advanced hierarchical chunking, context header based markdown conversion etc... All the tricks that were published last year while RAG was still the "new" kid in town. We finally open sourced everything as the competition from the big players (Notion AI, Google etc.) was daunting. If anyone is interested, this blog post about all the techniques we tried and what actually works is still relevant and up2date: https://bytevagabond.com/post/how-to-build-enterprise-ai-rag...
Thank you so much for this, started reading it a few min ago and already learnt quite a lot!
I like how clean and compressed the info is
[flagged]