Engineering Working Group + Hack Weekend

743 - Cogs - Pattern

cc-by2.0 Patrick Hoesly on flickr

Back in August we announced the formation of the Engineering Working Group, tasked with trying to attract more developers by lowering barriers to entry. Since then we’ve seen some good technical coding work and other achievements in and around the activities of this group:

  • OpenStreetMap is now rolling with rails version 3, thanks largely to the hard-working Tom Hughes. Besides deploying it, and ironing out a few nasty problems with sessions, he did the work of porting the code over. The website and API code needed to take account of differences and new features of rails 3, particularly the use of AREL for database querying.
  • Kai Krueger has packaged this rails code, and also Mapnik and mod_tile, into an unbuntu PPA. This packaging system offers a very simple way to install these tools (and keep them up to date) on ubuntu/debian . We’re currently testing this, and hope it’ll make it much easier for developers to hack on the code for openstreetmap.org.
  • Working with Mike Migurski we have a more attractive, and more helpful page sitting at planet.openstreetmap.org, the OpenStreetMap data downloads site. Mike, and stamen design, are also now providing “metro extracts” – more manageable (smaller) files for OpenStreetMap data, one city at a time.

In addition to these, the Engineering Working Group has dived head first into the big tasks of improving technical documentation, and tidying up the bug tracker.

Clearly these are things which would happen anyway within the OpenStreetMap developer community, and the achievements are down to the hard work of individual people. But the Engineering Working Group lends a little structure, and provides a forum for taking a step back and looking at these kinds of meta-development. Development which helps development!

You can find out more about the Engineering Working Group on the OSMF site, and anyone is welcome to join in their discussions, which take place every Monday on IRC (details)

London Hack Weekend – 26th, 27th

Perhaps you’d prefer to join in face-to-face? Come along to a “hack weekend”! EWG is also involved in this, and trying to get more developer events happening, in more locations. For the moment though we have one coming up at the end of the week…

See the London Hack Weekend details on the wiki (and sign up there)

Previous hack weekend at the same venue (MapQuest offices)

“An OpenStreetMap ‘hack weekend’ is a meet-up where we bring along laptops to an office space and spend the weekend doing some technical work to improve OpenStreetMap. This may be development of the “core” components, the editors, or any other side projects and pet projects we fancy hacking on. OpenStreetMap has development tasks sprouting from it in all directions. There’s work to do in almost any programming language, as well as tasks like documentation, and even some non-technical graphics design and translation tasks.

We mostly take a fairly unstructured free-form format. People turn up and start beavering away on something, or they turn up and see what they can help with. However we can also run more structured workshops if there is demand”

Whether you can make it to a hack weekend or not, we are always looking for more technical people to help with improving OpenStreetMap.

Posted in Uncategorized | 2 Comments

ODbL progress

We’re planning the final stages of the switch over to the Open Database License for OpenStreetMap data. The OpenStreetMap Foundation Board discussed the license upgrade process and many other aspects of the project at their recent board meeting, and we’ll have more information about that from the board shortly.

One item that came out of the board meeting was the deadline to complete the license upgrade by 01 April 2012 and to publish the first OpenStreetMap planet file under the ODbL by 04 April 2012. The License Working Group supports this target date as a reasonable goal.

There are still many things to do before we are ready to publish the first OpenStreetMap planet file as an ODbL database. As always, community engagement and your participation are important. There will be more information and details on your favourite OSM community channels including the mailing lists and IRC. For now the process of contacting mappers yet to respond and remapping non-compliant data is still the priority.

There are various tools to help you get an idea of ODbL coverage in your country, or your local area. In particular, you can enable a view within Potlatch2 or install a plugin for JOSM to see the license status of elements.

Posted in Board | Tagged , , , , | 7 Comments

What’s new on OpenStreetMap.org?

Swan photo by Tony the Misfit is licensed CC-By

We spend a lot of time talking about the amazing map our users are producing, but perhaps not enough about the technology that enables it all. But it’s worth shouting about: from the reliable Ruby code that puts stuff into our super-efficient Postgres database setup, to the speedy hardcore cgimap code that sends it out to editors, to the wizardry of Mapnik that makes it all into beautiful maps… like the swan, there’s a lot of effort underneath the surface, but it all seems serene above water.

In fact, there’s a constant stream of changes aimed at making OpenStreetMap easier to use – big ones, yes, but also “little things that mean a lot”. So, for example, in the last month we’ve improved the “users near you” map (thanks to Martijn van Exel) and fixed potential security issues (thanks, as ever, to Tom Hughes). We’re also finishing up a move to Rails 3, which will help us make more user interface improvements, and keep the code clean. We’ll tell you more about these changes as they happen.

Have you noticed that osm.org adjusts to smaller screens with smaller tabs? It does now. Map CC-By-SA

Our Potlatch 2 editor has also had a whole bunch of improvements. It’s now much easier to draw shapes with holes (“multipolygons”), the GPS track handling is cleverer, and we have a clever feature where you can replace a node with a whole new one (select the old one and press O). We’ve incorporated a number of suggestions from usability research, and there’s still more to come.

If you’re a developer, we’d love to have your help with OpenStreetMap. There’s so much to do! You can find out about our codebase at http://wiki.openstreetmap.org/wiki/Rails, and join the rails-dev mailing list to bounce ideas off other developers. Or if you’d like to help with Potlatch 2, see http://wiki.openstreetmap.org/wiki/Potlatch and the potlatch-dev mailing list.

Posted in osm.org | Tagged , , , , , , | 2 Comments

Tile Usage Policy

Tile image by vidalia_11 is licensed CC-By-SA

We’ve had to block some uses of the OpenStreetMap Foundation tile servers. This article describes what is happening and why. This article also describes how you can adapt if you are affected.

We’re really proud of the increased popularity of OpenStreetMap. We’ve seen seven-plus years of project growth in every measurable area. As the project has grown we’ve learned and adapted in many ways. The use of our tile server has grown faster than every other aspect of the project. One way that we are adapting now, is by restricting how our map tiles may be used.

Slow sign by DaveCrosby is licensed CC-By-SA

You’ll still be able to use our tiles in creative and interesting ways but the volume of use will be limited. We need to limit access to our tile server to only those users who don’t overburden our resources.

Those users who make large demands on our tile server will be slowed down by our throttling mechanism. This throttling mechanism is rarely triggered by mappers.

Problematic applications may show this image instead of a map.

Those applications which make exceptional demands in aggregate from their users will be blocked. The tile usage policy is on the OSM wiki.

So what can you do about this? How can you get the wonderful OpenStreetMap tiles for your mobile device?

You’ll find more advice about potential tile source alternatives on the wiki.

Read on if you would like to know more about the history of the OpenStreetMap tile server.

OSM tile server background

We started creating rendered images of our map data as a way to encourage our data contributors. Mappers enjoy seeing the results of their surveys on the OpenStreetMap web site, and they can be inspired to map in their neighbourhood the things they see other mappers surveying in other places. Mappers loved the tile server when it first appeared. Potential users often looked at the map in 2006 and said, “Hey, why is there a huge blank spot where my town should be?” Some of those potential users became the long-time contributors that we all know and love.

As more contributors mapped more neighbourhoods, more blank spots started to fill in. More potential users became actual users, and OpenStreetMap tiles started to appear in more places. The tile server became even more popular when rapid updates were enabled. Rather than updating the map every week, parts of the map could update as contributors added data. If you remember the weekly updates you also remember that funny tingle you had the first time you mapped something and it appeared on the map immediately; it seemed like magic, didn’t it?

The OpenStreetMap Foundation has had a tile usage policy for some time. From September 2008 it has been explicitly stated that bulk downloading of tiles was discouraged. OpenStreetMap kept growing. More people came to understand the awesomeness of OSM tiles.

Also in 2008, the Ordnance Survey started serving map tiles to users through their OS Openspace program. In July of 2011, Ordnance Survey served their one billionth tile to a user.

OpenStreetMap serves a Billion tiles every eleven days.

So we know a thing or two about providing awesome maps to users. We do it all with the crowd-sourced data from our contributors around the world, the volunteered time of our sysadmins who keep our servers running, and the generous donations of servers and bandwidth and funds.

We’ve had to become more restrictive of the use of our tile server over time. We’ve limited how many tiles you can consume in a period of time. These restrictions only affected the most-demanding of tile consumers. The everyday mapper never ran into a problem getting tiles to add data to OSM. That allowed the growing number of users to continue to have access to OpenStreetMap tiles without our resources being monopolized by one or two bad actors.

More and more mobile applications started using OpenStreetMap tiles. Many of them included a bulk downloading method so that tiles could be saved on the device at home, rather than downloading tiles at a punitive data rate. That bulk-downloading has always been problematic for OpenStreetMap because a single user will consume hundreds of times the resources of an average user. There are so many applications using OSM tiles, with so many users making unreasonable demands on our resources that it is affecting the quality of service for the average user. And that’s not fair.

So we’ve started blocking the applications that are causing us the most trouble, in addition to blocking users with problematic specific behaviours. We regret it, in a way. After all, we map because we want people to be able to use our data. But our resources have to be used in a way that everybody can share. We can’t have a small number of people consuming all of our resources.

So that’s why some people have started to see the “prohibited” tiles on their maps. Overuse. Or mobile applications that are causing overuse by a group.

Posted in History | Tagged , , , | 5 Comments

State of the Map 2011 wrap up

State of the Map 2011 group photo

The fifth annual OpenStreetMap conference, State of the Map, wrapped up in Denver, Colorado recently. Even if you were not able to join the fun in Denver, you can still enjoy parts of the event. Conference sessions were recorded by our friends at FOSSLC and many of those recordings are already available. You’ll also find audience video for some of the sessions.

Tagging a conference

If we were tagging SotM11 to add it to the OSM database, we might consider tags like:

  • conferencename = State of the Map, 2011
  • attendance = 273
  • attendance:prepaid = 258
  • attendance:walkin = 15
  • attendance:attendee_countries = 34

State of the Map 2012

The call for venues for State of the Map 2012 is expected in November 2011. The SotM 2012 venue will be announced early in 2012.

Posted in SOTM | Tagged , , , | Comments Off

ODbL data.gov.au permission granted

data.gov.au logoThe Licensing Working Group has obtained explicit special permission to incorporate geographic datasets from data.gov.au in the OpenStreetMap project database published under any free and open license, including ODbL, provided that…
a) we provide primary attribution in a reasonable manner (currently the Attribution wiki page), and…
b) that we explicitly list there each dataset used to give useful feedback within the Australian government on how folks are using open data.

We have been careful to point out that (under ODbL) we are not asking folks who make visual maps from OpenStreetMap to provide secondary attribution to each and every contributor, so would not be in compliance with the CC-BY Australia 2.5 and 3.0 license their data is normally provided under. They have raised no objection to this.

The LWG would like to publicly thank data.gov.au both for providing open geographic data and for providing this permission.

(Message on the talk-au mailing list from Grant Slater of the License Working Group)

Posted in Legal | Comments Off

More new servers

Dragon sculpture on the Dragon Bridge in Ljubljana. Photo CC-By-SA, dani_7C3


The OpenStreetMap Foundation, and the Operations Working Group, would like to thank Nokia UK Limited for the donation of some of their redundant server hardware. This hardware has found new purpose in the form of “soup“[1] and “fiddlestick“[2], two new web front end servers. A third server “eustace“[3] will be used initially as a trial web statistics server.

The web front-end servers, soup and fiddlestick, replace puff and fuchur who had performed that role since 2008. Web front-end servers in OpenStreetMap provide the data browser and data layer, as well as user diaries and other “social” functions.

Eustace will debut in a new role for OpenStreetMap by collecting web statistics. The OpenStreetMap Foundation wants to know more about how users experience the OSM web site in an effort to improve the way that OSM services are delivered.

[1] Character from The Clangers, a UK children’s TV programme.
[2] Strangewood (1999): Fiddlestick, a small musically emotive dragon.
[3] Turns into a dragon in The Voyage of the Dawn Treader (Chronicles of Narnia) after slipping on a gold bracelet.

Posted in Operations | Tagged , , , , | 5 Comments

Introducing Zark

Zark, during installation.

Zark is the newest OpenStreetMap server. Give Zark a warm welcome. Continuing the in the tradition of naming OSM servers after dragons, the name “Zark” is taken from the Eidolon Chronicles/Shadow World books by Jane Johnson.

The first task for Zark will be to serve as a trial / evaluation server for the OWL – OpenStreetMap Watch List service. OWL’s popularity on the dev server has lead to performance problems and long update delays. After more than a year of development and increasing popularity of OWL’s ability to follow local changes without distracting “Big” changesets, moving OWL to Zark will make this service even more effective for mappers.

Many thanks to bitfolk.com for donating this server.

Posted in Operations | Tagged , , , | 2 Comments

New Board Members

Final votes were cast today at the Foundation AGM, and the results are in. Congratulations to our new board members:

Richard Fairhurst, Matt Amos, and Dermot McNally

Mikel Maron was also re-elected onto the board.

[Update] : The votes tallied up as follows: 98 votes for Richard Fairhurst, 81 Matt Amos, 74 Mikel Maron, 68 Dermot McNally, 47 Kate Chapman, 44 Eugene Usvitsky, 42 Derick Rethans, 41 Niccolo Rigacci, 27 Serge Wroclawski.

Thankyou to all of our candidates. This year we really had an excellent group of highly dedicated OSMers to choose from, and the choice was a difficult one. The relatively even spread of votes is reflective of this. We hope and expect that you will all play an active role in the foundation in 2011-2012.

Posted in Board, SOTM | 4 Comments

Vote for OpenStreetMap Foundation candidates for the Board

The 2011 election is under way for members of the OpenStreetMap Foundation Board of Directors. Further to our previous post, there’s now nine candidates to choose from, which is great! Members of the OSMF can vote in person at the annual general meeting at State of the Map or by proxy via email. Probably many people will be doing the email option. The list of candidates and voting details are found on the wiki.

Please read the email instructions carefully. Note that the deadline for voting by email is coming in a couple of days time: Thu 8th at 17:00 UTC.

If you need to join the foundation first then time is definitely running out! (There’s a delay for processing by the membership secretary) Join the OpenStreetMap Foundation here.

Questions to candidates and discussions can be found on the OSMF-talk@ archives and on the Election discussion page on the wiki.

Vote photo by Marc Tarlock is licensed CC-By-SA

Posted in Board, Organisation | Comments Off