Carl Morris, Author at Mapio Cymru

We are building an open public map of Wales with all the names in Welsh. Because of recent work, the map will load much faster for you now.

Here’s what we did to improve the map loading speed.

When you load the map, what you’re seeing is a grid of tiles. Each is a square image file, like this:

Here’s a small bit of Tyddewi / St David’s at zoom level 17.

A JavaScript library called Leaflet manages your navigation around the map (panning and zooming). The main point is, it’s ultimately made up of images.

These tile images are rendered from the underlying map data in OpenStreetMap, which is stored as points, ways, and relations. As well as the data, the OpenStreetMap software stack developed by the project is also freely licensed.

The most time consuming part of showing an up-to-date map to a user is converting the data into images. This tends to be done when the user loads the map: the images are generated and served, and also stored in a cache on the server.

If the map were completely finished and final then that would help. We could make sure the server has the tile images all rendered and stored, and serve them every time. But that’s not an option for the whole map for a couple of reasons.

At the moment the Mapio Cymru map is updated automatically once every night when most people in Wales are asleep, and server capacity tends to be higher. These updates are necessary because geographical map features and names change often, whenever somebody makes an edit to OpenStreetMap. This is often an improvement to the map, e.g. somebody adding a name to a feature. The open data elements are constantly being revised, making it a bit like a Wikipedia of maps. The edit can also be a response to something changing in the physical world, e.g. a café changing its name or perhaps a lovely new railway station.

Therefore we can’t preserve the map in aspic, it’s changing all the time.

It turns out there’s another snag to the idea of pre-rendering and storing the whole map to speed up loading. There is a different set of image tiles at each zoom level. For the furthest zoom levels it is possible to store all the tile images. But for closer zoom levels, the total number of tiles grows exponentially. Pretty soon we need a vast amount of time and storage space, much more than we have.

For example all of Wales at zoom level 17 took a little over seven hours to render overnight. That’s too much.

Can we pre-render and store some selected tiles, and then render any others on demand? It turns out that we can. The challenge is to figure out what to pre-render for maximum speed advantage, given the constraints of time and storage.

What are the map areas of ‘interest’ or ‘relevance’, and how do we codify this more precisely?

Initially we had a hypothesis that for the map sparsely populated areas would be less frequently visited than densely populated areas. One method would be to pre-render areas above a certain population density threshold.

I then realised that there was another solution much more ready to go, and even better. We could refer to aggregated browser requests for tiles to see which parts of our map were visited most. This allows us to look at the historical popularity of areas right down to individual tile level. This was data we already had, lying in the server logs.

Here’s a heat map produced by Ben Proctor.

Heat map for one year of visits at all zoom levels

The popular areas do seem to correspond to population density. There may also be a relationship with the number and/or percentage of Welsh speakers in different areas, which is available from Census data.

I’ve instructed the server to pre-render these tiles and store them. We have chosen these areas:

Zoom levels 3 to 16 are now entirely pre-rendered.

Zoom levels 17 and 18 are now partially pre-rendered.

Now the server automatically pre-renders these areas every night, immediately after importing the up-to-date data.

The difference was very noticeable when I loaded the site before and after the change. Beforehand I’d been a bit embarrassed about the huge blank areas and the apparent freeze-ups of the map, while the server wheezed along. I am not experiencing that anymore – at least for now!

On average, tiles are loading in 40% of the time when pre-rendered. That’s a dramatic improvement, although the degree of speed-up is highly dependent on how many users are accessing the server at once.

Even tiles that are not pre-rendered are loading faster because there is usually more capacity on the server.

Incidentally the smooth running of the server also depends on choosing the right settings and configuration. (We briefly considered nginx as an alternative to Apache but it appears not to have an equivalent of the mod_tile module.)

As we gain more interest and users for the map I expect to have to visit this again. Your contributions to project costs are always useful.

Map: places in Wales that have ‘llan’ in their names

Would you like to get castles, rivers, post boxes, or cycleways in Wales from a map?

How about investigating place names in Welsh in your local area?

How about getting other features in Wales and further afield, as open data from a map?

This blog post will show you how to get open data from OpenStreetMap, with a particular emphasis on Welsh-language data.

It is intended as a fun introduction, not as a comprehensive reference guide. No previous experience is necessary.

We will be passing queries to the Overpass API, and it’s easy to get started. The queries can be run from your web browser in Overpass Turbo, which is one seriously cool app. Other than that your curiosity is the only prerequisite!

Introductory concepts

Feel free to skip this section if you want to head to the practical bit straightaway.

OpenStreetMap is a global map which has been built by thousands of people. It uses a wiki-like approach to mapping – anybody can edit and re-use the content. Because it’s all open data, you can use it however you want in your own learning, work, and leisure.

There is a huge amount of Welsh-language data in OpenStreetMap.

It’s independent of proprietary mapping providers, allowing you freedom to work with the data in your own projects.

The underlying code is also freedom-respecting software and open source. As the Mapio Cymru project we have built a showcase map which shows Welsh-language names for features including places, roads, rivers, and so on.

How to run an Overpass query

The quickest way to try Overpass queries is to visit the Overpass Turbo website.

The screen will be divided into an editor panel and a map/data viewer panel. Now do this:

Write (or paste!) a query into the editor.
Click the Run button.
The results are shown in the data viewer.
Within the data viewer you can select Map tab or the Data tab.

You’ll be following these same steps every time you run a query.

Towns query

Here is a simple query you can use. First drag the map and zoom until it shows an area you want to investigate, e.g. a part of Wales. Then follow the above steps using this query.

node["place"="town"]({{bbox}});
out;

Bingo, you should now see towns plotted on the map area you’ve selected. Congratulations on accomplishing your first Overpass query!

The data

Select the Data tab in the data viewer to see the data. It will be in the default format, which is XML.

Here’s a portion of the XML data you’ll see for the results of the above query, for two towns:

<node id="8997358" lat="51.5912466" lon="-2.7517629">
  <tag k="name" v="Caldicot"/>
  <tag k="name:cy" v="Cil-y-coed"/>
  <tag k="place" v="town"/>
  <tag k="population" v="11200"/>
  <tag k="postal_code" v="NP26 4"/>
  <tag k="wikidata" v="Q722585"/>
  <tag k="wikipedia" v="en:Caldicot, Monmouthshire"/>
</node>

<node id="21413062" lat="51.8591257" lon="-4.3115907">
  <tag k="is_in" v="Wales"/>
  <tag k="name" v="Carmarthen"/>
  <tag k="name:br" v="Caerfyrddin"/>
  <tag k="name:cy" v="Caerfyrddin"/>
  <tag k="name:en" v="Carmarthen"/>
  <tag k="name:ja" v="カーマーゼン"/>
  <tag k="name:la" v="Moridunum"/>
  <tag k="name:ru" v="Кармартен"/>
  <tag k="place" v="town"/>
  <tag k="population" v="14185"/>
  <tag k="population:date" v="2011"/>
  <tag k="source" v="NPE"/>
  <tag k="source:population" v="Census"/>
  <tag k="wikidata" v="Q835835"/>
</node>

As you can see, the name:cy tag has the town’s name in Welsh. There are equivalent tags for other languages. There’s also a tag called name without a language code, here’s the definition of the name key.

In general name:cy will provide the name in Welsh for anything on the map – if it’s been submitted.

The other data in the examples above should be fairly self-explanatory, and include latitude and longitude, Wikidata item identifier, and other things.

Note that OpenStreetMap is always a work in progress. You’ll see pretty good data for many queries although some others will display gaps. (You can edit/add place names on the map, and other features and their tags.)

Change your Overpass Turbo map to Mapio Cymru

Within Overpass Turbo your underlying map will probably be the main OpenStreetMap. This is OK but it won’t always display all names in Welsh.

You can change it to the Mapio Cymru map server, like this:

Select Settings menu
Select Map
In the Tile-Server box put: //openstreetmap.cymru/osm_tiles/{z}/{x}/{y}.png

Please note that when you click on map pins any links will still go to the main OpenStreetMap.

Farms, cities, and other places

You can take the query above and modify it:

node["place"="farm"]({{bbox}});
out;

Spot the difference between this query and the one above. Alternatively use one of the possible key values for place. For example you can use “village”, “city”, “island” and so on.

Your bounding box

In general:

If your query refers to a bbox (bounding box) the query will run on the visible map, the portion of the map you’ve selected.
You can also reduce the width of the map: drag its edge to reduce its size, and increase the size of the editor.
If your query has a lot of results, there may be too much data to plot on the Overpass Turbo map in your browser. Try zooming in to reduce the size of the bounding box.

Towns in Wales only

No matter how much you move the bounding box it’s not possible to get all of Wales, and Wales only. Our query needs to change.

This time, click the Wizard button and type ‘towns in Wales’ then click Build Query. When I ran it it suggested ‘town in Wales’ then gave the following query, and yours will be similar or the same.

/*
This has been generated by the overpass-turbo wizard.
The original search was:
“town in wales”
*/
[out:json][timeout:25];
// fetch area “wales” to search in
{{geocodeArea:wales}}->.searchArea;
// gather results
(
  // query part for: “town”
  node["place"="town"](area.searchArea);
  way["place"="town"](area.searchArea);
  relation["place"="town"](area.searchArea);
);
// print results
out body;
>;
out skel qt;

Where possible the Wizard will take the English you type and give a query in Overpass query language. As far as I know the Wizard is only available in English at the moment.

searchArea above is a variable containing our geocode area for Wales. It is set for the life of the query. We don’t have to call it searchArea, we can call it almost anything – as long as there’s no clash with other reserved terms.

The above query contains comments which have no effect on the query. There are two styles:

/* comment within slash star delimiters */

// comment between double slash and end of line

Llan place names

As well as Llanelwy this will return Rhosllannerchrugog in the results – and so on. It’s a case-insensitive search.

[out:json][timeout:50];
(
  node["name"~"Llan",i][place]({{bbox}});
);
out center;

This is a narrower search for Llan with a capital L.

[out:json][timeout:50];
(
  node["name"~"Llan"][place]({{bbox}});
);
out center;

Here’s a search that includes the tags name a name:cy for a comprehensive map which includes places which currently lack a name:cy tag and names like Llanandras (Presteigne) and Llanllieni (Leominster) (diolch/thanks for your replies via Twitter!).

(
node({{bbox}})["name:cy"~"Llan"][place];
node({{bbox}})["name"~"Llan"][place];
);
out;

This will give all places in Wales with Llan in the name. It gives data only – in Overpass Turbo the map tab will be blank. You can use the CSV results data in a project, e.g. in a spreadsheet.

[out:csv("name:cy", "name", ::lat, ::lon, "place", ::id; true; ",")][timeout:50];
{{geocodeArea:wales}}->.searchArea;
(
node["name"~"Llan"][place](area.searchArea);
node["name:cy"~"Llan"][place](area.searchArea);
);
out;

You could modify one of the above for ‘Aber’, ‘Caer’, ‘Tre’ and so on.

Castles in any area

Now try this query.

[out:json][timeout:25];
// gather results
(
  // query part for: “castle”
  node["historic"="castle"]({{bbox}});
  way["historic"="castle"]({{bbox}});
  relation["historic"="castle"]({{bbox}});
);
// print results
out body;
>;
out skel qt;

This is OK but how about all the castles in Wales only? Use this:

[out:json][timeout:25];
{{geocodeArea:wales}}->.searchArea;
// gather results
(
  // query part for: “castle”
  node["historic"="castle"](area.searchArea);
  way["historic"="castle"](area.searchArea);
  relation["historic"="castle"](area.searchArea);
);
// print results
out body;
>;
out skel qt;

Here are some others to try. In each case you should edit the three statements above to cover all nodes, ways and relations in the search. Let’s look up the definitions of those in a jiffy…

"natural"="peak"

"site_type"="megalith"

"historic:civilization"="ancient_roman"

"amenity"="bicycle parking"

"amenity"="recycling"

"amenity"="bus station"

The last one will identify, among others, the National Express coach station in Cardiff – currently the only bus station in the city.

Elements of OpenStreetMap

There are millions of possible Overpass queries.

You can play around with basic queries without having a comprehensive understanding of OpenStreetMap. The wizard may help.

Sooner or later though you might want more context to help you write that special query for your own interest. This portion from the documentation on elements has some vital definitions will help:

Elements are the basic components of OpenStreetMap’s conceptual data model of the physical world. They consist of

nodes (defining points in space),
ways (defining linear features and area boundaries), and
relations (which are sometimes used to explain how other elements work together).

All of the above can have one or more associated tags (which describe the meaning of a particular element).

If you want to see some examples of nodes, use this query.

In Overpass Turbo this will only work for small bounding boxes, because the amounts of data are relatively large.

Show the Wales Coastal Path

This is a simple query that only shows one relation – the northern part of the Wales Coastal Path.

relation(1850847);>;out;

This shows the entire Wales Coastal Path. (Because this is stored as a relation of relations, the query uses a ‘recurse down relations’ operator >> to display the relations, ways, and nodes within the overall relation. Here’s more info on the recurse down relations operator.)

[out:json][timeout:25];
rel(1820890);
(._;>>;);
out;

Hiking routes

Taken from the Overpass API examples. You probably need to zoom into a bounding box.

[bbox:{{bbox}}];
(
  relation[route=hiking][network~"^.wn$"];
  way(r);
  >;
);
out;

{{style:
way
{ color:green; fill-color:green; }

relation[network=lwn] way
{ color:blue; fill-color:cyan; }

relation[network=iwn] way
{ color:red; fill-color:red; }

relation[network=nwn] way
{ color:green; fill-color:green; }

relation[network=rwn] way
{ color:yellow; fill-color:yellow; }

}}

Rectangular buildings in Wales that are taller than they are wide

This one’s adapted from the examples.

[out:json][timeout:25];
{{geocodeArea:wales}}->.searchArea;
// Find rectangular buildings that are taller then they are wide
(
  // Compare the height to the average length of a side
  way["building"]["height"](area.searchArea)(if:
    count_members() < 6 && is_closed() &&
    number(t["height"]) > length() / 4);
  // Assume a floor is 3 m tall
  way["building"]["building:levels"](area.searchArea)(if:
    count_members() < 6 && is_closed() &&
    number(t["building:levels"]) * 3 > length() / 4);
);

// Print results
out body;
>;
out skel qt;

What’s next? Edit the map

If you notice any deficiencies in the data then you can edit the map. Welcome to the open data mapping community!

Overpass can be used deliberately to look for opportunities to improve the map.

Overpass Turbo in Welsh?

Overpass Turbo’s interface is available in a few languages but it doesn’t offer Welsh as an interface language yet. If you’d like to contribute to the translation head to its Transifex project.

Write your own queries

Start with the Overpass Turbo examples and Overpass API examples provided by the OSM community.

You can even delve into the user manual for Overpass.

Using Python instead of Overpass Turbo

If you can use Python you can run Overpass queries in your code using this simple wrapper instead of the Overpass Turbo web interface. Write an app and wow us!

Alternatively check out these other methods of querying Overpass via code.

In any case Overpass Turbo is handy for perfecting your queries.