Adding a [date] to a small query causes it to exhaust memory

[gboeing]

If I run a simple query in a small ~400x400 meter bounding box, it completes successfully as expected in <2 seconds (link):

[out:json][timeout:180];((node['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;););(way['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;););(relation['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;);););out;

But if I add a [date] (dating back to just a few hours ago) to the otherwise-same query, it runs out of memory after running for ~25 seconds (link):

[out:json][timeout:180][date:'2025-06-22T00:00:00Z'];((node['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;););(way['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;););(relation['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;);););out;

"An error occurred during the execution of the overpass query! This is what overpass API returned: runtime error: Query run out of memory using about 2048 MB of RAM."

If I do the same thing, with the same date filter, but querying for bus stops instead of residential roads, everything works fine (link):

[out:json][timeout:180][date:"2025-06-22T00:00:00Z"];((node['highway'='bus_stop'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;););(way['highway'='bus_stop'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;););(relation['highway'='bus_stop'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');(._;>;);););out;

Is this a bug? Why would adding a [date] to that first query exhaust memory, when adding the same [date] to other similar queries works fine?

[mmd-osm]

Adding [date] to a query results in a much different evaluation strategy, because the data needs to be reconstructed from previous and current object versions to match the requested point in time. Without [date], only the current object versions are considered, which cuts down the data volume significantly.

My guess is that the second query failed because it was requiring slightly more memory than permitted by default. In this particular case, adding a maxsize parameter seems to help: [maxsize:3G][out:json][timeout:180].

By the way, I've checked the query on another implementation with some performance improvements in place. Both date queries took a second or two with 512M maxsize.

[gboeing]

Thanks @mmd-osm. It is interesting that Overpass's query evaluation strategy is so memory-inefficient when given a [date] parameter.

Out of curiosity, what is the other performance-optimized implementation that you tested this on?

FYI I passed your info along to the StackOverflow question that originated this issue.

[mmd-osm]

I've tested it on my fork.

[drolbr]

Thank you for reporting the issue.

Is this a bug? Why would adding a [date] to that first query exhaust memory, when adding the same [date] to other similar queries works fine?

We do not know for sure. It all boils down to running a public instance in a maximum resilient way.

One scarce or potentially scarce resource is memory. The tricky point of memory is that it is too late to act when the memory is already filled, because there are then no good options for concurrent or deadlocked other requests. For this reason, the interpreter wants to know ahead of time whether there is a risk of taking too much memory.

The public instance is therefore designed to estimate ahead of fetching data the amount of data to expect. If the resulting data is a combination of multiple conditionals then the only safe assumption is that the full data from the most restrictive filter will fully pass through. That is most likely the full content of the polygon (or more precisely of the shingles that cover the polygon), because highway=residential is a weak filter on ways.

Now a couple of things might happen:

• the estimation is applied on globally all historic residential ways instead of the polygon (bug, and unlikely)
• the polygon is estimated to have had much more ways in the past than currently (artifact of former data vandalism, and should be prevented by improvement of the estimator)
• a different strategy kicks in because some optimizations are not available on historic data (not a bug, but might be worth an improvement)
• a simple miscalculation (bug, possible)

I does take time to investigate what is actually going on and to rebalance the size estimation. This investigation competes with other bugfixes and improvements for developer attention. There is current effort underway to shrink the total database, to deal with aggressive request patterns from clouds of servers, and to enable some redaction mechanism, and this is lower in importance than the aforementioned issues.

I'll keep the issue open to come back to it as soon as time allows.

In the meantime I would like to encourage you to test around a little bit:

• run the request for each type individually and test whether it applies to ways only or each of nodes, ways, and relations
• test what happens with an enclosing bounding box instead of the polygon to probe the polygon's influence
• test what happens when the tag filter is fully dropped

This can help to shed some light on which of the potential mechanisms is at work here or whether it is something completely different.

[drolbr]

[out:json][timeout:180][date:"2025-06-22T00:00:00Z"];
way['highway'='residential'](poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');
out;

is a smaller example. It are indeed the ways.

[drolbr]

Second note:

[out:json][timeout:180][date:"2025-06-22T00:00:00Z"];
way
   (poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');
way._(if:t['highway']=='residential');
out count;

completes within seconds while

[out:json][timeout:180][date:"2025-06-22T00:00:00Z"];
way
   (poly:'37.822573 -122.233912 37.822573 -122.229358 37.826170 -122.229358 37.826170 -122.233912 37.822573 -122.233912');
way._['highway'='residential'];
out count;

fails. The two requests are supposed to have the same results. Thus there is definitely a bug around the tag filter in the highway=residential case in combination with the date setting.

[drolbr]

It is also fixed in the (misnamed) touched_timeline branch. That one is the base for the next release and will come live with that.

[gboeing]

Great! Thanks @drolbr.