I'm confused by both this blog post, and the reception on HN. They... didn't actually train the model. This is an announcement of a plan! They don't actually know if it'll even work. They announced that they "trained over 50 million neural networks," but not that they've trained this neural network: the other networks appear to just have been things they were doing anyway (i.e. the "Virtual Positioning Systems"). They tout huge parameter counts ("over 150 trillion"), but that appears to be the sum of the parameters of the 50 million models they've previously trained, which implies each model had an average of... 3MM parameters. Not exactly groundbreaking scale. You could train one a single consumer GPU.
This is a vision document, presumably intended to position Niantic as an AI company (and thus worthy of being showered with funding), instead of a mobile gaming company, mainly on the merit of the data they've collected rather than their prowess at training large models.
“Concepts of a plan” is often enough to make people think you know what you’re doing. Think most people, here included, got the impression that they had succeeded already.
They have never been a mobile game company and they have said as much themselves on many occasions. They're a data harvesting company. Guess now they're trying to figure out what to do with all of that data.
This is pretty cool, but I feel as a pokehunter (Pokemon Go player), I have been tricked into working to contribute training data so that they can profit off my labor. How? They consistently incentivize you to scan pokestops (physical locations) through "research tasks" and give you some useful items as rewards. The effort is usually much more significant than what you get in return, so I have stopped doing it. It's not very convenient to take a video around the object or location in question. If they release the model and weights, though, I will feel I contributed to the greater good.
Everything “free” coming from a company means they’ve found a way to monetise you in some way. The big long ToS we all casually accept without reading says so too.
Other random examples which appear free but aren’t: using a search engine, using the browser that comes with your phone, instagram, YouTube… etc.
It’s not always about data collection, sometimes it’s platform lock-in, or something else but there is always a side of it that makes sense for their profit margin.
Hiding shady or unexpected stuff in the TOS is illegal in the EU and other countries for example. So just because some companies behave amoral, that doesn’t mean we just have to accept hundreds of pages of legalese being able to dictate us.
Google actually has released weights for some of their models, but judging by the fact that this model is potentially valuable, they likely will not allow Niantic for this
All companies should be truthful, forthcoming, and specific about how they will use your data, but…
If you enjoy the game, play the game. Don’t boycott/withhold because they figured out an additional use for data that didn’t previously exist.
Another way of viewing this: GoogleMaps is incredibly high quality mapping software with lots of extra features. It is mostly free (for the end user). If no one uses it, Google doesn’t collect the data and nobody can benefit from the analysis of the data (eg. Traffic and ETA on Google Maps)
There’s no reason to hold out for a company to pay you for your geolocation data because none of them offer that service.
I wish it were that simple but I think it's reasonable to hesitate. We don't know what these models are going to be used for. If by playing you're unwittingly letting something powerful fall into the wrong hands, maybe play something else.
(Generally speaking. I'm not trying to throw stones at Niantic specifically here.)
It may surprise you to learn pokemon go is nearly a 10 year old game based on 40 year old beloved IP that when it was released did not exist in the same data hellscape we do today, and even if it did, the attraction of the IP would overrule people thinking about this kind of thing. These kinds of comments are extraordinarily disingenuous sounding, particularly when anyone can spend 3 seconds and figure out their primary market is literal children.
Imagine how those of us who played Ingress (Niantic's first game) feel... We were tricked into contributing location data for the game we loved, only to see it reused for the far more popular (and profitable) Pokemon Go.
Why would anyone take issue with this? Asking as someone who tried both games at different points.
Niantic was always open with the fact that they gather location data, particularly in places cars can't go - I remember an early blog post saying as much before they were unbundled from Google. No one was tricked, they were just not paying attention.
They were pretty up-front about it bring a technology demo for a game engine they were building. It was obvious from the start that they would build future games on the same platform.
Do you honestly feel tricked that a gameplay mechanic which transparently asks you to record 50-100MB videos of a point-of-interest and upload it to their servers in exchange for an (often paid/premium) in-game reward was a form of data collection?
I don't think I've done any in PoGo (so I know it's very optional), but I've done plenty in Ingress, and I honestly don't see how it's possible to be surprised that it was contributing to something like this? It is hardly an intuitively native standalone gameplay mechanic in either game.
> They consistently incentivize you to scan pokestops (physical locations) through "research tasks" and give you some useful items as rewards.
There are plenty of non-scan tasks you can do to get those rewards as well but I do think Poffins (largely useless unless you are grinding Best Buddies) are locked behind scan tasks.
Source: Me. This is the one topic I am very qualified to speak to on this website.
Frankly given the numbers of hours of entertainment most people got out of Pokémon Go, I suspect this might be one of the cases where people have been best compensated for their data collection.
Frankly, with the amount of real-world walking required to progress in Ingress and Pokémon Go, most players were compensated by the motivation to get a decent amount of exercise, which had a net positive impact on their health. Most exercise apps require users to pay subscriptions for the pleasure of using them.
> I have been tricked into working to contribute training data so that they can profit off my labor.
you werent tricked - your location data doesn't belong to you when you use the game.
I don't get why people somehow feel that they are entitled to the post-facto profit/value derived from the data that at the time they're willingly giving away before they "knew of" the potential value.
At some point can we agree that if we don't pay anything for something and we experience something fun, it's ok for the company to get something for investing millions of dollars in creating the experience for us in return?
If you weren't aware until now and were having fun is this outcome so bad? Did you have a work contract with this company to provide labor for wages and they didn't pay you? if not, then I don't think you can be upset that they are possibly profiting from your "labor".
Every time we visit a site that is free, which means 99.9% of all websites, that website bore a cost for our visit. Sometimes they show us ads which sometimes offsets the cost of creating the content and hosting it.
I am personally very glad with this arrangement. If a site is too ad filled, I just leave immediately.
With a game that is free and fun, I would be happy that I didn't have to pay anything and that the creator figured out a way for both parties to get something out of the deal. Isn't that a win-win situation?
Also, calling your experience "labor" when you were presumably having fun (if you weren't then why were you playing without expectation for payment in return?) is disingenuous.
At some point we need to be realistic about the world in which we live. Companies provide things for free or for money. If they provide something for "free", then we can't really expect to be compensated for our "labor" playing the game and that yes, the company is probably trying to figure out how to recoup their investment.
You've also been tricked into making your comment, which will undoubtedly be fed into an LLM's training corpus, and someone will be profiting off that, along with my comment as well. What a future we live in!
If anything, it makes me sad that Google sat and watched OpenAI tarnish public recognition of AI, and meanwhile no one had figured out realistic digital persona reconstruction from Twitter logs applicable to arbitrary users. I've made so much of my "effort" into the Internet digital corpus with that purpose in mind!
Note to future digital me, do as I did 2007-2014. I approve.
Lol, do you really think that? I did it from having a desire to contribute to the conversation and I was aware that that would be a future possibility :) I'm not really getting much in return or being incentivized by Y combinator
But did you really scan the items they wanted? Most people in my local community scan their hands or the pavements around the pokestop.
They have a great map of London pavements if they want to do it.
Yeah, they did the same in Ingress: film a portal (pokéstop/gym) while walking around it to gain a small reward. I've always wondered what kind of dataset they were building with that -- now we know!
Honestly you should have assumed they were using the collected data for such a purpose. It would be shocking if they weren't doing this directly or selling the data to other companies to do this.
Yeah it’s horrible. The other day I made a comment on this website and someone learned something from it without my consent. I explicitly refuse permission for you to read this comment. You do NOT have permission. Our privacy is important and I will protect my rights. If you donate 10% of your annual income to the International Society for Krishna Consciousness, I think I’d understand. Anything less is RAPE of my rights! The 4-equidistant time points can be considered as Time Square imprinted upon the circle of Earth: a higher order of life time cube.
This title is editorialized. The real title is: "Building a Large Geospatial Model to Achieve Spatial Intelligence"
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
My personal layman's opinion:
I'm mostly surprised that they were able to do this. When I played Pokémon GO a few years back, the AR was so slow that I rarely used it. Apparently it's so popular and common, it can be used to train an LGM?
I also feel like this is a win-win-win situation here, economically. Players get a free(mium) game, Niantic gets a profit, the rest of the world gets a cool new technology that is able to turn "AR glasses location markers" into reality. That's awesome.
I'm pretty sure most of the data is not coming from the AR features. There are tasks in the game to actually "scan" locations. Most people I know who play also play the game without the AR features turned on unless there's an incentive.
It's OK to adjust the title to have more relevant facts or to fix a poorly worded one. Editorializing is more like 'Amazing: Niantic makes world-changing AI breakthrough'.
The original title was not poorly worded though. The new one was editorialized to get a certain reaction out of readers — I promise you the responses on this thread would look different with the original title.
Many articles only make it to the front page because the submitted title was editorialized. The rules may say one thing, but the incentives are to a subtle balance between editorialization and avoiding flagging due to extreme editorialization with mods only stepping in to correct the title once it's gotten loads of upvotes and comments already.
> the rest of the world gets a cool new technology
The rest of the world gets an opportunity to purchase access to said new technology, you mean! It's not like they're releasing how they generated the models. It's much more difficult to get excited about paid-access to technology than it is about access to tech itself.
I feel like I'm going mad, if you actually read the article it's a theoretical thing they'd like to lead in, yet literally every comment assumes it launched. The title being "announces model" rather than the actual title certainly doesn't help.
All they needed was a shit ton of pictures. The AR responsiveness (and Pokemon Go) have nothing to do with it. It was just a vehicle for gathering training data.
When users scan their barcode, the preview window is zoomed in so users think its mostly barcode. We actually get quite a bit more background noise typically of a fridge, supermarket aisle, pantry etc. but it is sent across to us, stored, and trained on.
Within the next year we will have a pretty good idea of the average pantry, fridge, supermarket aisle. Who knows what is next
This is outrageously unethical. Someone scanning a barcode would have every reason to think that the code was being parsed locally on their phone. There would be no reason to upload an entire photo to read a barcode. Beyond which, not even alerting the user visually that their camera is picking up background stuff???
What if it's on their desk and there are sensitive legal documents next to it? How are you safeguarding all that private data? You could well be illegally in possession of classified documents, unconsenting nudes, all kinds of stuff. And it sounds like it's not even encrypted.
This post’s replies makes it clear a lot of us don’t recognize humor. Do people really think MyFitnessPal is trying to build a model of the average pantry?
The humor isn’t recognized because the humor isn’t there. To be funny there has to be a setup, a punchline, some kinda joke structure. Humor isn’t just saying false things…
Imagine a comedian saying this on stage, how many laughs would that get?
> Do people really think MyFitnessPal is trying to build a model of the average pantry?
We’ve all seen dumber things that are real. Juicero is my personal favorite example.
The problem is that it's not possible to make a parody of an unethical company so blatant that it wouldn't also be a 100% plausible description of a business practice that some company actually does...
If this is real, I hope MyFitnessPal doesn't operate in the EU.
Or rather, I hope they do, and receive an appropriate fine for this, if not even criminal prosecution (e.g. if the app uploaded nonconsensual pornography of someone visible only in the cropped out space).
Not wanting to over-do it, but is there possibly an argument the data about geospatial should be in the commons and google have some obligation to put the data back into the commons?
I'm not arguing to a legal basis but if it's crowdsourced, then the inputs came from ordinary people. Sure, they signed to T&Cs.
Philosophically, I think knowledge, facts of the world as it is, even the constructed world, should be public knowledge not an asset class in itself.
I’ve been saying this about Google Maps for years, especially their vast collection of public transport loading data and real time road speeds.
People are duped into thinking they’re doing some “greater good” by completing the in-app surveys and yet the data they give back is for Google’s exclusive use and, in fact, deepens their moat.
It's not solely for Google's benefit. They're ("we're" tbh) contributing data that improves services that we use. It has additional selfish and altruistic benefits beyond feeding the Googly beast.
IIRC Google maps basically does not make money. I wonder if there can be a government deal to subsidize it on the condition that the data be open sourced.
No. it should be owned by the owners of the land on which these objects are located. You should be able to provide access at different levels of detail to public or private entities that need said access and revoke it at your own will. May be make some money out of it.
3D artist can create a model of a space and offer rights to the owner of the land, who in turn can choose to create his own model or use the one provided by an artist.
I expect any company which collates information about geospatial datasets to release the substance of them, yes. Maybe there's an IPR lockup window, but at some point the cadastral facts of the world are part of the commons to me.
I would think there's actually a lot of epidemiology data which also should be winding up in the public domain getting locked up in medical IPR. I could make the same case. Cochrane reports rely on being able to do meta analysis over existing datasets. Thats value.
Pokemon Go is built on the same engine as Inverness I think its called. When it launched they even used the same POIs. I think this was ~5-7 years before PGO launched.
Edit: I said inverness and meant ingress. Apologies.
Pokemon Go was launched on the Unity game engine in 2016. Ingress was using a different game engine at the time, and wasn't rewritten into Unity until several years later. Even the backend/server side was significantly different, with them needing to write a shim to ensure compatibility during & after the move to Unity.
I was wondering about the privacy implications: given a photo, the LGM could decode it to not just positioning, but also time-of-day and season (and maybe even year, or specific unique dates e.g. concerts, group activities).
Colors, amount of daylight(/nightlight), weather/precipitation/heat haze, flowers and foliage, traffic patterns, how people are dressed, other human features (e.g. signage and/or decorations for Easter/Halloween/Christmas/other events/etc.)
(as the press release says: "In order to solve positioning well, the LGM has to encode rich geometrical, appearance and cultural information into scene-level features"... but then it adds "And, as noted, beyond gaming LGMs will have widespread applications, including spatial planning and design, logistics, audience engagement, and remote collaboration.") So would they predict from a trajectory (multiple photos + inferred timeline) whether you kept playing/ stopped/ went to buy refreshments?
As written it doesn't say the LGM will explicitly encode any player-specific information, but I guess it could be deanonymized (esp. infer who visited sparsely-visited locations).
(Yes obviously Niantic and data brokers already have much more detailed location/time/other data on individual user behavior, that's a given.)
Hanke’s actually got awards from CIA for his work at In-Q-Tel investing in Keyhole/Niantic, so yeah, safe to assume that the agency invested specifically to have players collect data. Considering many Pokémon were on or near military bases around the world… not hard to assume what CIA’s real goal was.
> For example, it takes us relatively little effort to back-track our way through the winding streets of a European old town. We identify all the right junctions although we had only seen them once and from the opposing direction.
That is true for some people, but I'm fairly sure that the majority of people would not agree that it comes naturally to them.
Interestingly, Pokemon GO only prompts players to scan a subset of the Points of Interest on the game map. Players can manually choose to scan any POI, but with no incentive for those scans I'm sure it almost never happens.
> Today we have 10 million scanned locations around the world, and over 1 million of those are activated and available for use with our VPS service.
This 1 in 10 figure is about accurate, both from experience as a player and from perusing the mentioned Visual Positioning System service. Most POI never get enough scan data to 'activate'.
The data from POI that are able to activate can be accessed with a free account on Niantic Lightship [1], and has been available for a while.
I'll be curious to see how Niantic plans to fill in the gaps, and gather scan data for the 9 out of 10 POI that aren't designated for scan rewards.
I really want to know what the NSA and NRO and Pentagon are doing training deep neural networks on hyperspectral imaging and synthetic aperture radar data. Imagine having something like Google Earth but with semantic segmentation of features combined with what material they are made from. All stored on petabytes of NVMe flash.
I still don't get what LGM is. From what I understood, it isn't actually about any "geospatial" data at all, is it? It is rather about improving some vision models to predict how the backside of a building looks, right? And training data isn't of people walking, but from images they've produced while catching pokemons or something?
P.S.: Also, if that's indeed what they mean, I wonder why having google street view data isn't enough for that.
> It is rather about improving some vision models to predict how the backside of a building looks, right?
This, yes, based on how the backsides of similar buildings have looked in other learned areas.
But the other missing piece of what it is seems to be relativity and scale: I do 3D model generation at our game studio right now and the biggest want/need current models can't do is scale (and, specifically, relative scale) -- we can generate 3d models for entities in our game but we still need a person in the loop to scale them to a correct size relative to other models: trees are bigger than humans, and buildings are bigger still. Current generative 3d models just create a scale-less model for output; it looks like a "geospatial" model incorporates some form of relative scale, and would (could?) incorporate that into generated models (or, more likely, maps of models rather than individual models themselves).
> And training data isn't of people walking, but from images they've produced while catching pokemons or something?
Training data is people taking dedicated video of locations. Only ARCore supported devices can submit data as well. So I assume along with the video they're also collecting a good chunk of other data such as depth maps, accelerometer, gyrometer, magnetometer data, GPS, and more.
The ultimate goal is to use the phone camera to get very accurate mapping and position. They're able to merge images from multiple sources which means they're able to localize an image against their database, at least relatively.
However, I can't fully agree that generating 3d scene "on the fly" is the future of maps and many other use cases for AR.
The thing with geospatial, buildings, roads, signs, etc. objects is that they are very static, not many changes are being made to them and many changes are not relevant to the majority of use cases. For example: today your house is white and in 3 years it has stains and yellowish color due to time, but everything else is the same.
Given that storage is cheap and getting cheaper, bandwidth of 5G and local networks is getting too fast for most current use cases, while computer graphics compute is still bound by our GPU performance, I say that it would be much more useful to identify the location and the building that you are looking at and pull the accurate model from the cloud (further optimisations might be needed like to pull only the data user has access to or needs access to given the task he is doing). Most importantly users will need to have access to a small subset of 3D space on daily basis, so you can have a local cache on end devices for best performance and rendering. Or stream rendered result from the cloud like nVidia GDN is doing.
Most precise models will come from CAD files for newly built buildings, retrospectively going back to CAD files of buildings build in last 20-30 years(I would bet most of them have some soft of computer model made before) and finally going back even further - making AI look at the old 2D construction plans of the building and reconstructing it in 3D.
Once the building is reconstructed (or a concrete pole like shown in the article) you can pull its 3D model from the cloud and place it in front of the user - this will cover 95% of use cases for AR. For 5% of the tasks you might want real time recognition of the current state of surfaces for some tasks or changes in geometry (like tracking the changes in the road quality compared with the previous scans or with reference model), but these cases can be tackled separately and having precise 3D model will only help, but won't be needed to be reconstructed from scratch.
This is a good 1st step to make a 3D map, however there should be an option to go to the real location and make edits to 3D plan by the expert so that the model can be precise and not "kind of" precise.
Somehow I always thought something like that would have been the ultimate use case for Microsoft Photosynth (developed from Photo Tourism research project), ideally with a time dimension, like browsing photos in a geo spatio-temporal context.
I expect that was also some reason behind their flickr bid back then.
I worked on this and yes it was 100% related to the interest in Flickr. At the time Google Street had just become a thing and there was interest in effectively crowdsourcing the photography via Flickr and some of the technology behind Photosynth.
Even before LLMs, I knew they are going to launch a fine grained mapping service with all that camera and POI data. Now this one is actually much better obviously. Very few companies actually have this kind of data. Remains to be seen how they make money out of this
> Today we have 10 million scanned locations around the world, and over 1 million of those are activated and available for use with our VPS service. We receive about 1 million fresh scans each week
Wait, they get a million a week but they only have a total of 10 million, ie 10 days worth? Is this a typo or am I missing something?
A location probably requires like a million scans to be visualized properly. Think of a park near your house - there are probably thousands of ways to view each feature within.
I’ve published research in this general arena and the sheer amount of data they need to get good is massive. They have a moat the size of an ocean until most people have cameras and depth sensors on their face
It’s funny, we actually started by having people play games as well but we expressly told them it was to collect data. Brilliant to use an AR game that people actually play for fun
I'm guessing this can be the new bot that could play competitively at GeoGuesser. It would be interesting if Google trained a similar model and released it using all the Street Map data, I sure hope they do.
Has anyone done something similar with the geolocated WIFI MAC addresses, to have small model for predicting location from those.
I believe I read somewhere that geoguesser AI based on street view data was mostly classifying based on the camera/vehicle set up. As in, a smudge on the lens in this corner means its from Paris.
This crowdsourced approach probably eliminates that issue.
I don’t see why not. Photos are often combined with satellite data for photogrammetry purposes, even on large scale - see the recent Microsoft Flight Simulator (in a couple days, when it actually works)
It's usually aerial data, especially oblique aerial.
Bing Maps is still pretty unique in offering them undistorted and not draped over some always degraded mesh.
I don't think so. I wanted to voice this quickly without a detailed rebuttal as yours is the top comment and I don't think it's correct. Hopefully someone will do my homework for me (or alternatively tell me I'm wrong!).
It may not be Geospatial data at all and I'm not sure how much the users consented but the data collection strategy was well crafted. I remember recommending building a game to collect handwriting data from testers (about a thousand), to the research lab I worked for long time back.
Conversation about ‘players are the product’ of Pokémon go aside… What are some practical applications of an LGM?
Seems like navigation is ‘solved’? There’s already a lot of technology supporting permanence of virtual objects based on spatial mapping?
Better AI generated animations?
I am sure there are a ton of innovations it could unlock…
"It could help with search and rescue" jokes aside [1] this seems really useful for robotics. Their demo video is estimating a camera position from a single image, after learning the scene from a couple images. Stick the camera on a robot, and you are now estimating where the robot is based on what the robot has seen before.
They are a bit vague on what else the model does, but it sounds like they extrapolate what the rest of the environment could look like, the same way you can make a good guess what the back side of that rock would look like. That gives autonomous robots a baseline they can use to plan actions (like how to drive/fly/crawl to the other side) that can be updated as new view points become available.
I wonder if there's a sweet spot for geospatial model size.
A model trained on all data for 1m in every direction would probably be too sparse to be useful, but perhaps involving data from a different continent is costly overkill? I expect most users are only going to care about their immediate surroundings. Seems like an opportunity for optimization.
Going to try to clear this up from speculation as best I can.
Niantic was a spinoff divested from Google Maps roughly a decade ago who created a game called Ingress. This used Open Street Maps data to place players in the real world and they could designate locations as points of interest (POI), which Niantic used human moderators to judge as sufficiently noteworthy. Two years after Ingress was released, Niantic purchased limited rights to use Pokemon IP and bootstrapped Pokemon Go from this POI data. Individual points of interest became Pokestops and Gyms. Players had to physically go to these locations and they could receive in-game items needed to continue playing or battle other Pokemon.
From the beginning, Pokemon Go had AR support, but it was gimmicky and not widely used. Players would post photos of the real world with Pokemon overlaid and then turn it off, as it was a significant battery drain and only slowed down your ability to farm in-game items. The game itself has always been a grind type of game. Play as much as possible to catch Pokemon, spin Pokestops, and you get rewards from doing so. Eventually, Niantic started having raids as the only way to catch legendary Pokemon. These were multiplayer in-person events that happened at prescribed times. A timer starts in the game and players have to be at the same place at the same time to play together to battle a legendary Pokemon, and if they defeat it, they'll be rewarded with a chance to catch one.
Something like a year after raids were released, Niantic released research tasks as a way to catch mythical Pokemon. These required you to complete various in-game tasks, including visiting specific places. Much later than this, these research tasks started to include visiting designated Pokestops and taking video footage, from a large enough variety of angles to satisfy the game, and then uploading that. They started doing this something like four or five years ago, and getting any usable data out of it must have required an enormous amount of human curation, which was largely volunteer effort from players themselves who moderated the uploads. The game itself would give you credit simply for having the camera on while moving around enough, and it was fairly popular to simply videotape the sidewalk and the running game had no way to tell this was not really footage of the POI.
The quality of this data has always been limited. Saying they've managed to build local models of about 1 million individual objects leaves me wondering what the rate of success is. They've had hundreds of millions of players scanning presumably hundreds of millions of POI for half a decade. But a lot of the POI no longer exist. Many of them didn't exist even when Pokemon Go was released. Players are incentivized to have as many POI near them as possible because this provides the only way to actually play, and Niantic is incentivized to leave as much as they can in the game and continually add more POI because, otherwise, nobody will play. The mechanics of the game have always made it tremendously imbalanced in that living near the center of a large city with many qualifying locations results in rich, rewarding gameplay, whereas living out in the suburbs or a rural area means you have little to do and no hope of ever gaining the points that city players can get.
This means many scans are of objects that aren't there. Near me, this includes murals that have long been painted over, monuments to confederate heroes that were removed during Black Lives Matter furors of recent years, small pieces of art like metal sculptures and a mailbox decorated to look like Spongebob that simply are not there any more for one reason or another, but the POI persist in the database anyway. Live scans will show something very different from the original photo that still shows up in-game to tell you what the POI is.
Another problem is many POI can't be scanned from all sides. They're behind fences, closed off because of construction, or otherwise obstructed.
Yet another problem is GPS drift. I live near downtown Dallas right now, but when the game started, I lived smack dab in the city center, across the street from AT&T headquarters. I started playing as something to do when walking during rehab from spine surgeries, but I was often bedridden and couldn't actually leave the apartment. No problem. I could receive sometimes upwards of 50km a day of credit for walking simply by leaving my phone turned on with the game open. As satellite line of sight is continually obstructed and then unobstructed by all the tall buildings surrounding your actual location, your position on the map will jump around. The game has a built-in speed limit meant to prevent people from playing while driving, and if you jump too fast, you won't get credit, but as long as the jumps in location are small enough to keep your average over some sampling interval below that limit, you're good to go. Positions within a city center where most of the POI actually are is very poor.
They claim here that they have images from "all times of day," which is possibly true if they literally mean daylight hours. I'm awake here writing this comment at 2:30 AM and have always been a very early riser. I stopped playing this game last summer, but when I still played, it was mostly in darkness, and one of the reason I quit was the frustration of constantly being given research tasks I could not possibly complete because the game would reject scans made in the dark.
Finally, POI in Ingress and Pokemon Go are all man-made objects. Whatever they're able to get out of this would be trained on nothing from the natural world.
Ultimately, I'm interested in how many POI the entire map actually has globally and what proportion the 1 million they've managed to build working local models of represents. Seemingly, it has to be objects that (1) still exist, (2) are sufficiently unobstructed from all sides, and (3) in a place free from GPS obstructions such that the location of players on the map is itself accurate.
That isn't nothing, but I'm enormously skeptical that they can use this to build what they're promising here, a fully generalizable model that a robot could use to navigate arbitrary locations globally, as opposed to something that can navigate fairly flat city peripheries and suburbs during daylight hours. If Meta can really get a large enough number of people to wear sunglasses with always-on cameras on them, this kind of data will eventually exist, but I highly doubt what Niantic has right now is enough.
I’m intrigued by the generative possibilities of such a model even more than how it could be used with irl locations. Imagine a game or simulation that creates a realistic looking American suburbia on the fly. It honestly can’t be that difficult, it practically predicts itself.
People complaining here that you are somehow owed something for contributing to the data set, or that because you use google maps or reCAPTCHA you are owed access to their training data.
I mean, I'd like that data too. But you did get something in return already. A game that you enjoy (or your wouldn't play it), free and efficient navigation (better than your TomTom ever worked), sites not overwhelmed by bots or spammers.
Yeah google gets more out of it than you probably do, but it's incorrect to say that you are getting 'nothing' in return.
The company was formed as Niantic Labs in 2010 as an internal startup within Google, founded by the then-head of Google's Geo Division (Google Maps, Google Earth, and Google Street View).
It became an independent entity in October 2015 when Google restructured under Alphabet Inc. During the spinout, Niantic announced that Google, Nintendo, and The Pokémon Company would invest up to $30 million in Series-A funding. Not sure what the current ownership is (they've raised a few more times since then), but they're seemingly still very closely tied with Google.
So. I'm not really sure what to do here given that this was exactly and specifically what we were building and frankly had a lot of success in actually building.
Waymo is supposedly geofenced because they need detailed maps of an area. And this is supposedly a blocker for them deploying everywhere. But then Google goes and does something like this, and I'm not sure, if it's even really true that Waymo needs really detailed maps, that it's an insurmountable problem.
I'm confused by both this blog post, and the reception on HN. They... didn't actually train the model. This is an announcement of a plan! They don't actually know if it'll even work. They announced that they "trained over 50 million neural networks," but not that they've trained this neural network: the other networks appear to just have been things they were doing anyway (i.e. the "Virtual Positioning Systems"). They tout huge parameter counts ("over 150 trillion"), but that appears to be the sum of the parameters of the 50 million models they've previously trained, which implies each model had an average of... 3MM parameters. Not exactly groundbreaking scale. You could train one a single consumer GPU.
This is a vision document, presumably intended to position Niantic as an AI company (and thus worthy of being showered with funding), instead of a mobile gaming company, mainly on the merit of the data they've collected rather than their prowess at training large models.
“Concepts of a plan” is often enough to make people think you know what you’re doing. Think most people, here included, got the impression that they had succeeded already.
Maybe its because the current HN title says "trained" in the past tense?
1 reply →
They have never been a mobile game company and they have said as much themselves on many occasions. They're a data harvesting company. Guess now they're trying to figure out what to do with all of that data.
This is pretty cool, but I feel as a pokehunter (Pokemon Go player), I have been tricked into working to contribute training data so that they can profit off my labor. How? They consistently incentivize you to scan pokestops (physical locations) through "research tasks" and give you some useful items as rewards. The effort is usually much more significant than what you get in return, so I have stopped doing it. It's not very convenient to take a video around the object or location in question. If they release the model and weights, though, I will feel I contributed to the greater good.
> I feel … I have been tricked
Everything “free” coming from a company means they’ve found a way to monetise you in some way. The big long ToS we all casually accept without reading says so too.
Other random examples which appear free but aren’t: using a search engine, using the browser that comes with your phone, instagram, YouTube… etc.
It’s not always about data collection, sometimes it’s platform lock-in, or something else but there is always a side of it that makes sense for their profit margin.
Hiding shady or unexpected stuff in the TOS is illegal in the EU and other countries for example. So just because some companies behave amoral, that doesn’t mean we just have to accept hundreds of pages of legalese being able to dictate us.
2 replies →
only a sith speaks in absolute. plenty of especially free AI products out there
4 replies →
> I have been tricked into working to contribute training data so that they can profit off my labor
You were playing a game without paying for it. How did you imagine they were making money without pimping your data?
Niantic made 700 million dollars last year, mostly selling virtual game items.
39 replies →
Lots of people are spending a lot of money on in app purchases in these games already.
> You were playing a game without paying for it.
I CALL BS. We paid ALL THE TIME! We pay even item's capacity so much they need to increase the limit recently[1].
Ref:
[1] https://www.facebook.com/PokemonGO/posts/1102918761192160
1 reply →
They won't. It's the same data collection play as every other Google project
Just for clarity on this comment and a separate one, Niantic is a Google spin out company and appears to still be majority shareholder: https://en.wikipedia.org/wiki/Niantic,_Inc.#As_an_independen...
Google actually has released weights for some of their models, but judging by the fact that this model is potentially valuable, they likely will not allow Niantic for this
29 replies →
I kept wondering why a Google spinoff was named after a river and community in Connecticut, one of the least Googley locales in the country.
The connection is a ship, built in Connecticut, which brought gold rushers to San Francisco and was run aground and converted to a hotel there: https://en.m.wikipedia.org/wiki/Niantic_(whaling_vessel)
The company was named after the ship.
All companies should be truthful, forthcoming, and specific about how they will use your data, but…
If you enjoy the game, play the game. Don’t boycott/withhold because they figured out an additional use for data that didn’t previously exist.
Another way of viewing this: GoogleMaps is incredibly high quality mapping software with lots of extra features. It is mostly free (for the end user). If no one uses it, Google doesn’t collect the data and nobody can benefit from the analysis of the data (eg. Traffic and ETA on Google Maps)
There’s no reason to hold out for a company to pay you for your geolocation data because none of them offer that service.
> If you enjoy the game, play the game
I wish it were that simple but I think it's reasonable to hesitate. We don't know what these models are going to be used for. If by playing you're unwittingly letting something powerful fall into the wrong hands, maybe play something else.
(Generally speaking. I'm not trying to throw stones at Niantic specifically here.)
> All companies should be truthful, forthcoming, and specific about how they will use your data, but…
I'm fairly sure, if you read the terms-and-conditions, it probably said that the company owns this data and can do what they want with it.
> There’s no reason to hold out for a company to pay you for your geolocation data because none of them offer that service.
Well, it can make perfect sense (to some people) to hold out forever in that case.
1 reply →
Were you really tricked? Hard to believe that someone on Hacker News saw Pokemon Go and didn't immediately think of the data collection possibilities.
It may surprise you to learn pokemon go is nearly a 10 year old game based on 40 year old beloved IP that when it was released did not exist in the same data hellscape we do today, and even if it did, the attraction of the IP would overrule people thinking about this kind of thing. These kinds of comments are extraordinarily disingenuous sounding, particularly when anyone can spend 3 seconds and figure out their primary market is literal children.
2 replies →
Imagine how those of us who played Ingress (Niantic's first game) feel... We were tricked into contributing location data for the game we loved, only to see it reused for the far more popular (and profitable) Pokemon Go.
Why would anyone take issue with this? Asking as someone who tried both games at different points.
Niantic was always open with the fact that they gather location data, particularly in places cars can't go - I remember an early blog post saying as much before they were unbundled from Google. No one was tricked, they were just not paying attention.
I didn't feel tricked. Still don't.
They were pretty up-front about it bring a technology demo for a game engine they were building. It was obvious from the start that they would build future games on the same platform.
1 reply →
As long as they make enough money from Pokemon Go to sustain Ingress, I OK with that.
The Google - Niantic - Ingress - borg - kubernetes conspiracy must be unraveled
Do you honestly feel tricked that a gameplay mechanic which transparently asks you to record 50-100MB videos of a point-of-interest and upload it to their servers in exchange for an (often paid/premium) in-game reward was a form of data collection?
I don't think I've done any in PoGo (so I know it's very optional), but I've done plenty in Ingress, and I honestly don't see how it's possible to be surprised that it was contributing to something like this? It is hardly an intuitively native standalone gameplay mechanic in either game.
Oh yes, children, their primary market, definitely consider this. Definitely.
1 reply →
They did at least published their research, and also dataset for 655 places:
https://research.nianticlabs.com/mapfree-reloc-benchmark
This was linked the news post (search for "data that we released").
> They consistently incentivize you to scan pokestops (physical locations) through "research tasks" and give you some useful items as rewards.
There are plenty of non-scan tasks you can do to get those rewards as well but I do think Poffins (largely useless unless you are grinding Best Buddies) are locked behind scan tasks.
Source: Me. This is the one topic I am very qualified to speak to on this website.
Weren't they pretty open about this being their business model?
> and give you some useful items as rewards
Were you tricked, or were you just poorly compensated for your time?
Frankly given the numbers of hours of entertainment most people got out of Pokémon Go, I suspect this might be one of the cases where people have been best compensated for their data collection.
2 replies →
Frankly, with the amount of real-world walking required to progress in Ingress and Pokémon Go, most players were compensated by the motivation to get a decent amount of exercise, which had a net positive impact on their health. Most exercise apps require users to pay subscriptions for the pleasure of using them.
> I have been tricked into working to contribute training data so that they can profit off my labor.
you werent tricked - your location data doesn't belong to you when you use the game.
I don't get why people somehow feel that they are entitled to the post-facto profit/value derived from the data that at the time they're willingly giving away before they "knew of" the potential value.
One of the reasons i never played pokemon go is because there was no guarantee I didnt have my data sold.
I can't tell you why other people wouldn't think of this concern
Now imagine how artists feel – and they didn’t even get any Pocket Monsters in return.
At some point can we agree that if we don't pay anything for something and we experience something fun, it's ok for the company to get something for investing millions of dollars in creating the experience for us in return?
If you weren't aware until now and were having fun is this outcome so bad? Did you have a work contract with this company to provide labor for wages and they didn't pay you? if not, then I don't think you can be upset that they are possibly profiting from your "labor".
Every time we visit a site that is free, which means 99.9% of all websites, that website bore a cost for our visit. Sometimes they show us ads which sometimes offsets the cost of creating the content and hosting it.
I am personally very glad with this arrangement. If a site is too ad filled, I just leave immediately.
With a game that is free and fun, I would be happy that I didn't have to pay anything and that the creator figured out a way for both parties to get something out of the deal. Isn't that a win-win situation?
Also, calling your experience "labor" when you were presumably having fun (if you weren't then why were you playing without expectation for payment in return?) is disingenuous.
At some point we need to be realistic about the world in which we live. Companies provide things for free or for money. If they provide something for "free", then we can't really expect to be compensated for our "labor" playing the game and that yes, the company is probably trying to figure out how to recoup their investment.
You've also been tricked into making your comment, which will undoubtedly be fed into an LLM's training corpus, and someone will be profiting off that, along with my comment as well. What a future we live in!
If anything, it makes me sad that Google sat and watched OpenAI tarnish public recognition of AI, and meanwhile no one had figured out realistic digital persona reconstruction from Twitter logs applicable to arbitrary users. I've made so much of my "effort" into the Internet digital corpus with that purpose in mind!
Note to future digital me, do as I did 2007-2014. I approve.
Baba booy bbaba booy Batman bats badly barring the baristers bearing.
Magic schoolbus!
Yea, take that llm model maker
NooooooooooOooOooOo!
1 reply →
Lol, do you really think that? I did it from having a desire to contribute to the conversation and I was aware that that would be a future possibility :) I'm not really getting much in return or being incentivized by Y combinator
4 replies →
[flagged]
17 replies →
But did you really scan the items they wanted? Most people in my local community scan their hands or the pavements around the pokestop. They have a great map of London pavements if they want to do it.
Yeah, they did the same in Ingress: film a portal (pokéstop/gym) while walking around it to gain a small reward. I've always wondered what kind of dataset they were building with that -- now we know!
As the old adage goes, "if you're not paying for the product, you ARE the product"...
It should just be "you ARE the product" giving that they don't care if you paid them or not.
Please don't tell me you were just now realizing this
“If you're not paying for the product, you are the product”
(I realize you can pay, but are not required to)
Did anyone here on hackernews not seriously assume this was the real reason for the existence of that game since day 1?
I'm not sure about the 'real reason'.
It's perhaps more like: some folks an Niantic wanted to make a Pokemon game, and this way they could make it financially viable?
Honestly you should have assumed they were using the collected data for such a purpose. It would be shocking if they weren't doing this directly or selling the data to other companies to do this.
Assumed … or just read the Terms & Conditions / AUP like we did 10 years ago when they were using "Ingress" for location collection & tracking.
My reaction, also.
"You used me... for LAND DEVELOPMENT! ...That wasn't very nice."
Yeah it’s horrible. The other day I made a comment on this website and someone learned something from it without my consent. I explicitly refuse permission for you to read this comment. You do NOT have permission. Our privacy is important and I will protect my rights. If you donate 10% of your annual income to the International Society for Krishna Consciousness, I think I’d understand. Anything less is RAPE of my rights! The 4-equidistant time points can be considered as Time Square imprinted upon the circle of Earth: a higher order of life time cube.
I mean it was ultimately a research task
The game is free, there has to be some way for them to profit, interesting to see this was it.
This wasn’t it. It was from gems
When ever it's free, it's all about the data.
I recall having a conversation circa 2004/5 with a colleague that Google was an AI company, not a search company.
1 reply →
Really? You feel … tricked? Are you new around here??
Well now by posting your thoughts to hn, you have been tricked yet again to give up free labor to train ai models.
This title is editorialized. The real title is: "Building a Large Geospatial Model to Achieve Spatial Intelligence"
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
My personal layman's opinion:
I'm mostly surprised that they were able to do this. When I played Pokémon GO a few years back, the AR was so slow that I rarely used it. Apparently it's so popular and common, it can be used to train an LGM?
I also feel like this is a win-win-win situation here, economically. Players get a free(mium) game, Niantic gets a profit, the rest of the world gets a cool new technology that is able to turn "AR glasses location markers" into reality. That's awesome.
I'm pretty sure most of the data is not coming from the AR features. There are tasks in the game to actually "scan" locations. Most people I know who play also play the game without the AR features turned on unless there's an incentive.
That's good information, thank you!
It's OK to adjust the title to have more relevant facts or to fix a poorly worded one. Editorializing is more like 'Amazing: Niantic makes world-changing AI breakthrough'.
The original title was not poorly worded though. The new one was editorialized to get a certain reaction out of readers — I promise you the responses on this thread would look different with the original title.
1 reply →
Many articles only make it to the front page because the submitted title was editorialized. The rules may say one thing, but the incentives are to a subtle balance between editorialization and avoiding flagging due to extreme editorialization with mods only stepping in to correct the title once it's gotten loads of upvotes and comments already.
> the rest of the world gets a cool new technology
The rest of the world gets an opportunity to purchase access to said new technology, you mean! It's not like they're releasing how they generated the models. It's much more difficult to get excited about paid-access to technology than it is about access to tech itself.
I feel like I'm going mad, if you actually read the article it's a theoretical thing they'd like to lead in, yet literally every comment assumes it launched. The title being "announces model" rather than the actual title certainly doesn't help.
Google branded AR glasses. Not any AR glasses.
The Harry Potter game had much better AR integration
All they needed was a shit ton of pictures. The AR responsiveness (and Pokemon Go) have nothing to do with it. It was just a vehicle for gathering training data.
We do this at MyFitnessPal.
When users scan their barcode, the preview window is zoomed in so users think its mostly barcode. We actually get quite a bit more background noise typically of a fridge, supermarket aisle, pantry etc. but it is sent across to us, stored, and trained on.
Within the next year we will have a pretty good idea of the average pantry, fridge, supermarket aisle. Who knows what is next
This is outrageously unethical. Someone scanning a barcode would have every reason to think that the code was being parsed locally on their phone. There would be no reason to upload an entire photo to read a barcode. Beyond which, not even alerting the user visually that their camera is picking up background stuff???
What if it's on their desk and there are sensitive legal documents next to it? How are you safeguarding all that private data? You could well be illegally in possession of classified documents, unconsenting nudes, all kinds of stuff. And it sounds like it's not even encrypted.
please don't feed the trolls
This post’s replies makes it clear a lot of us don’t recognize humor. Do people really think MyFitnessPal is trying to build a model of the average pantry?
The humor isn’t recognized because the humor isn’t there. To be funny there has to be a setup, a punchline, some kinda joke structure. Humor isn’t just saying false things…
Imagine a comedian saying this on stage, how many laughs would that get?
> Do people really think MyFitnessPal is trying to build a model of the average pantry?
We’ve all seen dumber things that are real. Juicero is my personal favorite example.
2 replies →
Who knows what is next
The problem is that it's not possible to make a parody of an unethical company so blatant that it wouldn't also be a 100% plausible description of a business practice that some company actually does...
1 reply →
If this is real, I hope MyFitnessPal doesn't operate in the EU.
Or rather, I hope they do, and receive an appropriate fine for this, if not even criminal prosecution (e.g. if the app uploaded nonconsensual pornography of someone visible only in the cropped out space).
Whoa, that's a p crazy admission. Is this known publicly?
I am just assuming the post was sarcasm and the user doesn’t work there.
Otherwise, someone is FIRED
I’d be interested in how your privacy policy allows this. I can’t find where it mentions photos are stored or used for training purposes…
The MyFitnessPal privacy policy says "We use photos, videos, or other data you provide to us to customize our Services." [1]
That's all they need to do to cover themselves.
[1] https://www.myfitnesspal.com/privacy-policy
4 replies →
I would be more interested on why you believe something like this isn't baked into most privacy policies.
I'm not shocked but I'm shocked you are shocked.
3 replies →
I really hope this is a joke, as someone who diligently uses the barcode feature on MFP everyday.......
brother definitely just violated an NDA
For when this is in court:
Hello court jurors ! I hope you're having a great day. One of the attorneys breath smells pretty bad, am I right ?
Was here before comment got removed!
Holy shit thats some big whistleblowing if true
Not wanting to over-do it, but is there possibly an argument the data about geospatial should be in the commons and google have some obligation to put the data back into the commons?
I'm not arguing to a legal basis but if it's crowdsourced, then the inputs came from ordinary people. Sure, they signed to T&Cs.
Philosophically, I think knowledge, facts of the world as it is, even the constructed world, should be public knowledge not an asset class in itself.
Four Square just open sourced their places dataset. https://location.foursquare.com/resources/blog/products/four...
Given how expensive it is to query Google places, would love a crowdsourced open-source places API.
I’ve been saying this about Google Maps for years, especially their vast collection of public transport loading data and real time road speeds.
People are duped into thinking they’re doing some “greater good” by completing the in-app surveys and yet the data they give back is for Google’s exclusive use and, in fact, deepens their moat.
It's not solely for Google's benefit. They're ("we're" tbh) contributing data that improves services that we use. It has additional selfish and altruistic benefits beyond feeding the Googly beast.
IIRC Google maps basically does not make money. I wonder if there can be a government deal to subsidize it on the condition that the data be open sourced.
3 replies →
As a Google maps user, I benefit from that data being in there.
No. it should be owned by the owners of the land on which these objects are located. You should be able to provide access at different levels of detail to public or private entities that need said access and revoke it at your own will. May be make some money out of it.
3D artist can create a model of a space and offer rights to the owner of the land, who in turn can choose to create his own model or use the one provided by an artist.
In the US at least, "facts of the world as it is" are not generally copyrightable, though any creative process in the presentation of them may be.
There's an "illegal child labor" angle to it, I suspect, T&Cs be damned.
Do you expect every company to release all their data to the public as well or it's just because you're not invested in this one?
I expect any company which collates information about geospatial datasets to release the substance of them, yes. Maybe there's an IPR lockup window, but at some point the cadastral facts of the world are part of the commons to me.
I would think there's actually a lot of epidemiology data which also should be winding up in the public domain getting locked up in medical IPR. I could make the same case. Cochrane reports rely on being able to do meta analysis over existing datasets. Thats value.
1 reply →
I can really imagine a meeting with the big brasses of Google/Niantic a few years ago that went along
- We need to be the first to have a better, new generation 3D model of the world to build the future of maps on it. How can we get that data?"
+ What about gamifying it and crowd-sourcing it to the masses?
- Sure! Let's buy some Pokemon rights!
It's scary but some people do really have some long-term vision
Pokemon Go is built on the same engine as Inverness I think its called. When it launched they even used the same POIs. I think this was ~5-7 years before PGO launched.
Edit: I said inverness and meant ingress. Apologies.
I think you are thinking of Ingress. No idea what Inverness is.
Ingress and PGO share the same portals and stuffs and its what PGO got its data from.
2 replies →
Famously for a long time, the best way to get a point of interest into PGO was to play ingress and request it's addition there
Pokemon Go was launched on the Unity game engine in 2016. Ingress was using a different game engine at the time, and wasn't rewritten into Unity until several years later. Even the backend/server side was significantly different, with them needing to write a shim to ensure compatibility during & after the move to Unity.
3 replies →
They definitely had this as a long-term vision
Brian Maclendon (Niantic) presented some interesting details about this in his recent Bellingfest presentation:
https://www.youtube.com/live/0ZKl70Ka5sg?feature=shared&t=12...
I'm sure the CIA already has access. [1] People were raising privacy concerns years ago. [2]
[1] https://www.networkworld.com/article/953621/the-cia-nsa-and-...
[2] https://kotaku.com/the-creators-of-pokemon-go-mapped-the-wor...
I was wondering about the privacy implications: given a photo, the LGM could decode it to not just positioning, but also time-of-day and season (and maybe even year, or specific unique dates e.g. concerts, group activities).
Colors, amount of daylight(/nightlight), weather/precipitation/heat haze, flowers and foliage, traffic patterns, how people are dressed, other human features (e.g. signage and/or decorations for Easter/Halloween/Christmas/other events/etc.)
(as the press release says: "In order to solve positioning well, the LGM has to encode rich geometrical, appearance and cultural information into scene-level features"... but then it adds "And, as noted, beyond gaming LGMs will have widespread applications, including spatial planning and design, logistics, audience engagement, and remote collaboration.") So would they predict from a trajectory (multiple photos + inferred timeline) whether you kept playing/ stopped/ went to buy refreshments?
As written it doesn't say the LGM will explicitly encode any player-specific information, but I guess it could be deanonymized (esp. infer who visited sparsely-visited locations).
(Yes obviously Niantic and data brokers already have much more detailed location/time/other data on individual user behavior, that's a given.)
Hanke’s actually got awards from CIA for his work at In-Q-Tel investing in Keyhole/Niantic, so yeah, safe to assume that the agency invested specifically to have players collect data. Considering many Pokémon were on or near military bases around the world… not hard to assume what CIA’s real goal was.
https://futurism.com/the-byte/pokemon-go-trespassers-militar...
Google maps has more data than PGO could ever hope to have.
But you only use Maps when you need directions.
1 reply →
[flagged]
More like Celesteela, after all, you need jet fuel to melt steel beams.
People have a lot of strange beliefs about the CIA. Why would they even care about this?
https://en.wikipedia.org/wiki/In-Q-Tel is a major investor into Niantic.
Upload a picture of a bad guy in an office lobby to pokegpt and ask it where he is.
2 replies →
> For example, it takes us relatively little effort to back-track our way through the winding streets of a European old town. We identify all the right junctions although we had only seen them once and from the opposing direction.
That is true for some people, but I'm fairly sure that the majority of people would not agree that it comes naturally to them.
Interestingly, Pokemon GO only prompts players to scan a subset of the Points of Interest on the game map. Players can manually choose to scan any POI, but with no incentive for those scans I'm sure it almost never happens.
> Today we have 10 million scanned locations around the world, and over 1 million of those are activated and available for use with our VPS service.
This 1 in 10 figure is about accurate, both from experience as a player and from perusing the mentioned Visual Positioning System service. Most POI never get enough scan data to 'activate'. The data from POI that are able to activate can be accessed with a free account on Niantic Lightship [1], and has been available for a while.
I'll be curious to see how Niantic plans to fill in the gaps, and gather scan data for the 9 out of 10 POI that aren't designated for scan rewards.
1: https://lightship.dev
Impressive, but this is one of those "if this is public knowledge, how far ahead is the _not_ public knowledge" things
I really want to know what the NSA and NRO and Pentagon are doing training deep neural networks on hyperspectral imaging and synthetic aperture radar data. Imagine having something like Google Earth but with semantic segmentation of features combined with what material they are made from. All stored on petabytes of NVMe flash.
I still don't get what LGM is. From what I understood, it isn't actually about any "geospatial" data at all, is it? It is rather about improving some vision models to predict how the backside of a building looks, right? And training data isn't of people walking, but from images they've produced while catching pokemons or something?
P.S.: Also, if that's indeed what they mean, I wonder why having google street view data isn't enough for that.
> It is rather about improving some vision models to predict how the backside of a building looks, right?
This, yes, based on how the backsides of similar buildings have looked in other learned areas.
But the other missing piece of what it is seems to be relativity and scale: I do 3D model generation at our game studio right now and the biggest want/need current models can't do is scale (and, specifically, relative scale) -- we can generate 3d models for entities in our game but we still need a person in the loop to scale them to a correct size relative to other models: trees are bigger than humans, and buildings are bigger still. Current generative 3d models just create a scale-less model for output; it looks like a "geospatial" model incorporates some form of relative scale, and would (could?) incorporate that into generated models (or, more likely, maps of models rather than individual models themselves).
> And training data isn't of people walking, but from images they've produced while catching pokemons or something?
Training data is people taking dedicated video of locations. Only ARCore supported devices can submit data as well. So I assume along with the video they're also collecting a good chunk of other data such as depth maps, accelerometer, gyrometer, magnetometer data, GPS, and more.
The ultimate goal is to use the phone camera to get very accurate mapping and position. They're able to merge images from multiple sources which means they're able to localize an image against their database, at least relatively.
Very cool.
However, I can't fully agree that generating 3d scene "on the fly" is the future of maps and many other use cases for AR.
The thing with geospatial, buildings, roads, signs, etc. objects is that they are very static, not many changes are being made to them and many changes are not relevant to the majority of use cases. For example: today your house is white and in 3 years it has stains and yellowish color due to time, but everything else is the same.
Given that storage is cheap and getting cheaper, bandwidth of 5G and local networks is getting too fast for most current use cases, while computer graphics compute is still bound by our GPU performance, I say that it would be much more useful to identify the location and the building that you are looking at and pull the accurate model from the cloud (further optimisations might be needed like to pull only the data user has access to or needs access to given the task he is doing). Most importantly users will need to have access to a small subset of 3D space on daily basis, so you can have a local cache on end devices for best performance and rendering. Or stream rendered result from the cloud like nVidia GDN is doing.
Most precise models will come from CAD files for newly built buildings, retrospectively going back to CAD files of buildings build in last 20-30 years(I would bet most of them have some soft of computer model made before) and finally going back even further - making AI look at the old 2D construction plans of the building and reconstructing it in 3D.
Once the building is reconstructed (or a concrete pole like shown in the article) you can pull its 3D model from the cloud and place it in front of the user - this will cover 95% of use cases for AR. For 5% of the tasks you might want real time recognition of the current state of surfaces for some tasks or changes in geometry (like tracking the changes in the road quality compared with the previous scans or with reference model), but these cases can be tackled separately and having precise 3D model will only help, but won't be needed to be reconstructed from scratch.
This is a good 1st step to make a 3D map, however there should be an option to go to the real location and make edits to 3D plan by the expert so that the model can be precise and not "kind of" precise.
Somehow I always thought something like that would have been the ultimate use case for Microsoft Photosynth (developed from Photo Tourism research project), ideally with a time dimension, like browsing photos in a geo spatio-temporal context.
I expect that was also some reason behind their flickr bid back then.
https://medium.com/@dddexperiments/why-i-preserved-photosynt...
https://phototour.cs.washington.edu
https://en.wikipedia.org/wiki/Photosynth
at least any patents regarding this will also expire about 2026.
I worked on this and yes it was 100% related to the interest in Flickr. At the time Google Street had just become a thing and there was interest in effectively crowdsourcing the photography via Flickr and some of the technology behind Photosynth.
Even before LLMs, I knew they are going to launch a fine grained mapping service with all that camera and POI data. Now this one is actually much better obviously. Very few companies actually have this kind of data. Remains to be seen how they make money out of this
> Today we have 10 million scanned locations around the world, and over 1 million of those are activated and available for use with our VPS service. We receive about 1 million fresh scans each week
Wait, they get a million a week but they only have a total of 10 million, ie 10 days worth? Is this a typo or am I missing something?
A location probably requires like a million scans to be visualized properly. Think of a park near your house - there are probably thousands of ways to view each feature within.
Scans are not always of new locations. They have ~10m established nodes and they get ~1m node scans per week that might be new and might be old.
Pretty sure there can be multiple "scans" per location is what they are saying
It’s possible they meant 1 million frames from scans.
I’ve published research in this general arena and the sheer amount of data they need to get good is massive. They have a moat the size of an ocean until most people have cameras and depth sensors on their face
It’s funny, we actually started by having people play games as well but we expressly told them it was to collect data. Brilliant to use an AR game that people actually play for fun
Yes it must be almost an exabyte of data.
I'm guessing this can be the new bot that could play competitively at GeoGuesser. It would be interesting if Google trained a similar model and released it using all the Street Map data, I sure hope they do.
Has anyone done something similar with the geolocated WIFI MAC addresses, to have small model for predicting location from those.
I believe I read somewhere that geoguesser AI based on street view data was mostly classifying based on the camera/vehicle set up. As in, a smudge on the lens in this corner means its from Paris.
This crowdsourced approach probably eliminates that issue.
I wonder how this can be combined with satellite data, if at all?
I don’t see why not. Photos are often combined with satellite data for photogrammetry purposes, even on large scale - see the recent Microsoft Flight Simulator (in a couple days, when it actually works)
It's usually aerial data, especially oblique aerial. Bing Maps is still pretty unique in offering them undistorted and not draped over some always degraded mesh.
Genuinely impressed Google had the vision and resources to commit to a 10 year data collection project
So that's why Pokemon was notoriously impactful on battery life. They were recording and uploading our videos the whole time?
I don't think so. I wanted to voice this quickly without a detailed rebuttal as yours is the top comment and I don't think it's correct. Hopefully someone will do my homework for me (or alternatively tell me I'm wrong!).
No, that is unlikely to be the case.
It may not be Geospatial data at all and I'm not sure how much the users consented but the data collection strategy was well crafted. I remember recommending building a game to collect handwriting data from testers (about a thousand), to the research lab I worked for long time back.
Conversation about ‘players are the product’ of Pokémon go aside… What are some practical applications of an LGM?
Seems like navigation is ‘solved’? There’s already a lot of technology supporting permanence of virtual objects based on spatial mapping? Better AI generated animations?
I am sure there are a ton of innovations it could unlock…
"It could help with search and rescue" jokes aside [1] this seems really useful for robotics. Their demo video is estimating a camera position from a single image, after learning the scene from a couple images. Stick the camera on a robot, and you are now estimating where the robot is based on what the robot has seen before.
They are a bit vague on what else the model does, but it sounds like they extrapolate what the rest of the environment could look like, the same way you can make a good guess what the back side of that rock would look like. That gives autonomous robots a baseline they can use to plan actions (like how to drive/fly/crawl to the other side) that can be updated as new view points become available.
1: https://www.xkcd.com/2128/
I hope this tech could help make AR glasses more useful in public, day-to-day life, like a video game HUD.
Applications that I thought of as I read this:
Real-Time mapping of the environment for VR experiences with built-in semantic understanding.
Winning at geoguesser, automated doxing of anybody posting a picture of themselves.
Robotic positioning and navigation
Asset generation for video games. Think about generating an alternate New York City that's more influenced by Nepal.
I'm getting echoes of neural radiance fields as well.
Procedural generation of an alternative planet is the kind of stuff that the No Man's sky devs could only dream of.
AI guided missiles.
I wonder if there's a sweet spot for geospatial model size.
A model trained on all data for 1m in every direction would probably be too sparse to be useful, but perhaps involving data from a different continent is costly overkill? I expect most users are only going to care about their immediate surroundings. Seems like an opportunity for optimization.
Is this related to NeRF (neural radiance fields)?
Going to try to clear this up from speculation as best I can.
Niantic was a spinoff divested from Google Maps roughly a decade ago who created a game called Ingress. This used Open Street Maps data to place players in the real world and they could designate locations as points of interest (POI), which Niantic used human moderators to judge as sufficiently noteworthy. Two years after Ingress was released, Niantic purchased limited rights to use Pokemon IP and bootstrapped Pokemon Go from this POI data. Individual points of interest became Pokestops and Gyms. Players had to physically go to these locations and they could receive in-game items needed to continue playing or battle other Pokemon.
From the beginning, Pokemon Go had AR support, but it was gimmicky and not widely used. Players would post photos of the real world with Pokemon overlaid and then turn it off, as it was a significant battery drain and only slowed down your ability to farm in-game items. The game itself has always been a grind type of game. Play as much as possible to catch Pokemon, spin Pokestops, and you get rewards from doing so. Eventually, Niantic started having raids as the only way to catch legendary Pokemon. These were multiplayer in-person events that happened at prescribed times. A timer starts in the game and players have to be at the same place at the same time to play together to battle a legendary Pokemon, and if they defeat it, they'll be rewarded with a chance to catch one.
Something like a year after raids were released, Niantic released research tasks as a way to catch mythical Pokemon. These required you to complete various in-game tasks, including visiting specific places. Much later than this, these research tasks started to include visiting designated Pokestops and taking video footage, from a large enough variety of angles to satisfy the game, and then uploading that. They started doing this something like four or five years ago, and getting any usable data out of it must have required an enormous amount of human curation, which was largely volunteer effort from players themselves who moderated the uploads. The game itself would give you credit simply for having the camera on while moving around enough, and it was fairly popular to simply videotape the sidewalk and the running game had no way to tell this was not really footage of the POI.
The quality of this data has always been limited. Saying they've managed to build local models of about 1 million individual objects leaves me wondering what the rate of success is. They've had hundreds of millions of players scanning presumably hundreds of millions of POI for half a decade. But a lot of the POI no longer exist. Many of them didn't exist even when Pokemon Go was released. Players are incentivized to have as many POI near them as possible because this provides the only way to actually play, and Niantic is incentivized to leave as much as they can in the game and continually add more POI because, otherwise, nobody will play. The mechanics of the game have always made it tremendously imbalanced in that living near the center of a large city with many qualifying locations results in rich, rewarding gameplay, whereas living out in the suburbs or a rural area means you have little to do and no hope of ever gaining the points that city players can get.
This means many scans are of objects that aren't there. Near me, this includes murals that have long been painted over, monuments to confederate heroes that were removed during Black Lives Matter furors of recent years, small pieces of art like metal sculptures and a mailbox decorated to look like Spongebob that simply are not there any more for one reason or another, but the POI persist in the database anyway. Live scans will show something very different from the original photo that still shows up in-game to tell you what the POI is.
Another problem is many POI can't be scanned from all sides. They're behind fences, closed off because of construction, or otherwise obstructed.
Yet another problem is GPS drift. I live near downtown Dallas right now, but when the game started, I lived smack dab in the city center, across the street from AT&T headquarters. I started playing as something to do when walking during rehab from spine surgeries, but I was often bedridden and couldn't actually leave the apartment. No problem. I could receive sometimes upwards of 50km a day of credit for walking simply by leaving my phone turned on with the game open. As satellite line of sight is continually obstructed and then unobstructed by all the tall buildings surrounding your actual location, your position on the map will jump around. The game has a built-in speed limit meant to prevent people from playing while driving, and if you jump too fast, you won't get credit, but as long as the jumps in location are small enough to keep your average over some sampling interval below that limit, you're good to go. Positions within a city center where most of the POI actually are is very poor.
They claim here that they have images from "all times of day," which is possibly true if they literally mean daylight hours. I'm awake here writing this comment at 2:30 AM and have always been a very early riser. I stopped playing this game last summer, but when I still played, it was mostly in darkness, and one of the reason I quit was the frustration of constantly being given research tasks I could not possibly complete because the game would reject scans made in the dark.
Finally, POI in Ingress and Pokemon Go are all man-made objects. Whatever they're able to get out of this would be trained on nothing from the natural world.
Ultimately, I'm interested in how many POI the entire map actually has globally and what proportion the 1 million they've managed to build working local models of represents. Seemingly, it has to be objects that (1) still exist, (2) are sufficiently unobstructed from all sides, and (3) in a place free from GPS obstructions such that the location of players on the map is itself accurate.
That isn't nothing, but I'm enormously skeptical that they can use this to build what they're promising here, a fully generalizable model that a robot could use to navigate arbitrary locations globally, as opposed to something that can navigate fairly flat city peripheries and suburbs during daylight hours. If Meta can really get a large enough number of people to wear sunglasses with always-on cameras on them, this kind of data will eventually exist, but I highly doubt what Niantic has right now is enough.
Lunduke is happy: “I told you so!”
https://www.youtube.com/watch?v=EVmZy95vMUc
This seems like it’d be quite handy to have in an autonomous vehicle of any kind
Don't quite understand the application of this?
Google Maps uses this tech for AR navigation: https://www.pocket-lint.com/what-is-google-maps-ar-navigatio...
I’m intrigued by the generative possibilities of such a model even more than how it could be used with irl locations. Imagine a game or simulation that creates a realistic looking American suburbia on the fly. It honestly can’t be that difficult, it practically predicts itself.
People complaining here that you are somehow owed something for contributing to the data set, or that because you use google maps or reCAPTCHA you are owed access to their training data. I mean, I'd like that data too. But you did get something in return already. A game that you enjoy (or your wouldn't play it), free and efficient navigation (better than your TomTom ever worked), sites not overwhelmed by bots or spammers. Yeah google gets more out of it than you probably do, but it's incorrect to say that you are getting 'nothing' in return.
I'm not sure quite what the ownership is, but Niantic isn't a subsidiary of Alphabet or Google.
The company was formed as Niantic Labs in 2010 as an internal startup within Google, founded by the then-head of Google's Geo Division (Google Maps, Google Earth, and Google Street View).
It became an independent entity in October 2015 when Google restructured under Alphabet Inc. During the spinout, Niantic announced that Google, Nintendo, and The Pokémon Company would invest up to $30 million in Series-A funding. Not sure what the current ownership is (they've raised a few more times since then), but they're seemingly still very closely tied with Google.
This is literally what I built my first company around starting in 2012, when Niantic was still working on Ingress
I describe it here during 500 Startups demo day: https://youtu.be/3oYHxdL93zE?si=cvLob-NHNEIJqYrI&t=6411
I further described it on the Planet of the Apps episode 1
Here's my patent from 2018: https://patents.google.com/patent/US10977818B2/en
So. I'm not really sure what to do here given that this was exactly and specifically what we were building and frankly had a lot of success in actually building.
Quite frustrating
Very interesting. What is the current state of this tech?
Call an intellectual property attorney?
Waymo is supposedly geofenced because they need detailed maps of an area. And this is supposedly a blocker for them deploying everywhere. But then Google goes and does something like this, and I'm not sure, if it's even really true that Waymo needs really detailed maps, that it's an insurmountable problem.
The data marginally better than what google already have
[dead]
Fucking cool. Hi old Niantic teammates, it's me Mark Johns ;).
The cia has to be all over this.
https://en.wikipedia.org/wiki/Keyhole_Markup_Language
I’m not sure why you are getting downvoted. Niantic has ties with the CIA.