Comment by bravesoul2
1 day ago
I found your medium article on this subject by hunting around. It's very interesting. Hope you write more.
From a layman's point of view, say this is true and embeddings should be a manifold. Like a horse saddle or sphere for example. Then some adding of vectors the simple way won't make sense. On a sphere west in China plus west in US is double west but in 3d it's zero as they point in different directions. Is this sort of the idea?
Hey man! Thanks a lot for your support! Might sound like just common words, but honestly — knowing this helped or inspired someone really motivates me. Makes me feel a bit less like a madman hahaha.
About your question — I think I get your point. Here’s how I understand it:
In the hypothetical latent manifold, we might want to measure two different things:
1. Distance between two points: This wouldn’t be the usual Euclidean distance, because the space is curved. Like how the shortest path between two cities on Earth isn't a straight line, but an arc on the globe. That’s where geodesics come in — they’re the shortest paths constrained by the shape of the manifold.
2. Similarity between two vectors ("parallel transport") Instead of asking where the vectors point in ambient space (like cosine similarity in R^n), we (should) care about how their directions compare _on the surface itself_. So ideally, we’d compare them along the geodesic — parallel transport — to properly align their frames of reference before measuring any angle or similarity.
That’s the intuition I’m working with, anyway. Let me know what you think, and thanks again for your comment!
Thanks! I am not used to thinking in geodesics. And while I studied vector spaces it was a long time ago and forgotten most of it!
So to get the intuition. Your model we still are dealing with vectors like regualar searches and not positions?
This is the model in my mind. A 2D surface with a steep mound in hidden dimension 3 to the north. 2 vectors at 1 degree apart pointing toward the mound are likely to be more different than 2 vectors 1 degree apart pointing away from it. But you only see this if you are right on the foot of the mound. What if you are further away?
Anyway I think my understanding is off it may not be worth answering! but I can read more about it.
Absolutely — you’re on the right track! The intuition of the "mound" is actually quite powerful.
Let’s imagine the embedding space as a 2D surface with a third hidden dimension — say, curvature pointing ‘up’ toward a peak (like a saddle or a bump).
Then:
Two vectors pointing in the same compass direction (e.g., “north”) might diverge or converge depending on their local curvature.
If you try to compare them using Euclidean cosine similarity, you’d be blind to the underlying geometry.
What you'd want is to "parallel transport" one vector along the surface toward the other — adjusting its direction to the curvature — and then compare them.
In high-curvature regions, even small angular differences can grow rapidly. In flatter regions, directionality might dominate. So yes, context (i.e., local geometry and density) changes everything.
Right now I'm testing some of this with reranking signals based on Ricci curvature + soft k-NN graphs (preserving density), and trying out simple geodesic-aware losses. But it’s still early days.
Your comment actually gave me a better mental image of what happens when you’re “far from the mound” — maybe curvature becomes negligible, and naive methods recover. Thanks for that.