Comment by daft_pink

6 hours ago

Wow, will they offers some sort of reduce a web page into a markdown file api as welL so we can get full web pages at reduced token sizes?

2 comments

daft_pink

We do have that in Exa search:

* Extraction of main content from HTMLs, the index stores markdown representations (free of headers/footers/sidebars/menus etc)

* Serving highlights picking the most relevant part for each result to reduce token usage downstream

* Also serving dynamic highlights where we summarize all the sources at once reducing the token counts even further

They already have that[0] it’s called Markdown for Agents and it works on any URL