MapBox - Development Seed http://www.developmentseed.org/blog/mapbox The latest developments in our open source mapping stack. en Welcome Robin Tolochko! http://www.developmentseed.org/blog/2015/05/29/welcome-robin <p><img src="https://cloud.githubusercontent.com/assets/7108211/7865443/d1d95942-0536-11e5-8884-402e92de668f.jpg" alt="Robin with bike" /></p> <p><a href="https://developmentseed.org/team/robin-tolochko/">Robin Tolochko</a> loves maps. Her favorite map is a 1868 map of South America. We know that because it’s on her resume.</p> <p>Robin shares her map love. She directed a mapping lab in Bogota and teaches others to make maps at Maptime Madison. Come see her <a href="http://stateofthemap.us/lightning-talks-sun/">lightning talk on Mapathons</a> at State of the Map US.</p> <p>At Development Seed, Robin will make beautiful maps that are intuitive and purposeful. Robin brings curiosity and a detail-oriented eye to all the <a href="http://tolomaps.com">work she does</a>. She’s committed to advancing women’s rights and renewable energy. She also owns a <a href="http://restrepoleather.com/">small business</a> that sells handmade leather goods from Colombia.</p> <p>Hit up Robin on <a href="https://twitter.com/tolomaps">twitter</a> or <a href="https://github.com/tolomaps">GitHub</a> to chat map projections, hiking, or ultimate frisbee.</p> 2015-05-29T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/29/welcome-robin Introducing OpenAerialMap http://www.developmentseed.org/blog/2015/05/27/introducing-openaerialmap <p class="dropcap">Today we’re releasing a beta version of <a href="http://beta.openaerialmap.org/">OpenAerialMap</a>. OpenAerialMap makes it easy to share and find open satellite and drone imagery. This is critical to the work of the disaster response community. We are launching this tool in close partnership with the <a href="http://hotosm.org/">Humanitarian OpenStreetMap Team (HOT)</a>.</p> <figure class="align-center"> <img src="https://farm8.staticflickr.com/7756/17967098000_3de88e8ee2_b.jpg" /> </figure> <p>OpenAerialMap is a set of tools for searching, sharing, and using open satellite and drone imagery. This initial release includes the core infrastructure to catalog petabytes of open imagery. It also includes an extremely usable API and an elegant web interface to submit, search and download available imagery.</p> <figure class="align-center"> <img src="https://farm8.staticflickr.com/7785/17967099320_05cb684a4e_b.jpg" /> <figcaption>Search for available imagery.</figcaption> </figure> <figure class="align-center"> <img src="https://farm8.staticflickr.com/7741/17534235363_fc92fcaf98_b.jpg" /> <figcaption>Select scenes by grid.</figcaption> </figure> <figure class="align-center"> <img src="https://farm8.staticflickr.com/7760/18128334246_2efce9e93f_b.jpg" /> <figcaption>Preview imagery and get metadata information.</figcaption> </figure> <h3 id="rebooting-a-great-concept">Rebooting a great concept</h3> <p>The OpenAerialMap concept has bounced between several attempts over half a decade. Previous attempts failed to take off. HOT <a href="http://www.elrha.org/hif-blog/openaerialmap/">reinitiated the concept</a> this year with funding from the <a href="http://www.elrha.org/hif/home/">Humanitarian Innovation Fund</a>. We worked with the HOT and others in the open imagery community to reimagine an approach to OpenAerialMap that we expect to be much more successful.</p> <ul> <li>We focused on simple, usable toolset that meets the clear needs of the humanitarian response community. The underlying architecture is flexible enough to be immediately useful to research, resource management, urban planning, and other communities. However, we decided to first build a frictionless interface for the clear needs of the disaster response community.</li> <li>We are extremely focused on community. From day one, we involved other organizations and developers like <a href="http://www.azavea.com/">Azevea</a>, <a href="http://planet.com">Planet Labs</a>, <a href="http://cadasta.org/">Cadasta</a>, <a href="http://opendronemap.github.io/odm/">OpenDroneMap</a> and HOT’s own developers.</li> <li>With these and other groups, we reimagined OpenAerialMap along a network model. Rather than try to house all the open imagery out there, OpenAerialMap is a node and index for a larger network or open imagery - the <a href="https://github.com/openimagerynetwork">Open Imagery Network</a>. </li> </ul> <h3 id="a-network-approach-to-open-imagery">A network approach to open imagery</h3> <p><a href="https://github.com/openimagerynetwork">Open Imagery Network (OIN)</a> is a simple framework and license for placing imagery into an open source license. Participants in the OIN adopt a common metadata scheme to describe the imagery they are making available, and standardized ways to broadcast and access that data. This allows us to build tools that search across all open imagery data without requiring one entity to host all of it. We’re working with HOT, Planet Labs, Cadasta, Azavea, OpenDroneMap, and others to develop OIN and to build OpenAerialMap as the first node in that network.</p> <h3 id="your-feedback">Your feedback</h3> <p>Community input is critical to the development of OpenAerialMap. Let us know what you think about the beta version and send us feedback on <a href="http://twitter.com/developmentseed">Twitter</a> or open an issue <a href="https://github.com/hotosm/OpenAerialMap/issues">on Github</a>. We’ll work to add functionality and features toward an initial release candidate later this summer. </p> 2015-05-27T12:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/27/introducing-openaerialmap Open Geospatial Happy Hour http://www.developmentseed.org/blog/2015/05/27/open-geospatial-happy-hour <p>The <a href="http://geospatialworldforum.org/">Geospatial World Forum</a> is well under way in Lisbon and we’re impressed with the diversity of people and talks at the event.</p> <p>On Thursday night, we are organizing an Open Geospatial Happy Hour at Café Fábulas with our friends from <a href="http://planet.com">Planet Labs</a>. This Happy Hour is a great opportunity to meet people and talk about open data, mapping and satellites in a more informal setting.</p> <p>You don’t have to be attending the conference to join the happy hour, just let us know if you’re coming. You can <a href="http://opengeospatialhappyhour.splashthat.com/">RSVP here</a>.</p> <p>If you are participating in the GFW, make sure to check out our <a href="https://developmentseed.org/blog/2015/05/25/meet-us-at-geospatial-world-forum/">workshop and talk</a> in the exhibition hall on Thursday morning.</p> 2015-05-27T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/27/open-geospatial-happy-hour Oil-Climate Index Launches http://www.developmentseed.org/blog/2015/05/26/oci-launch <p class="dropcap">Nearly 100 million barrels of oil flow through the global oil supply chain every day. But not all oils are created equal. When you consider the full oil processing lifecycle, some types of oil are responsible for nearly twice as much greenhouse gas as others. This is important information for oil procurement and energy policy. Smarter oil selection can lead to significant reduction in greenhouse gases without even touching overall oil consumption. We worked with the Carnegie Endowment to launch the <a href="http://oci.carnegieendowment.org/">Oil Climate Index website</a> to help consumers and policymakers make smarter decisions on oil.</p> <h3 id="extracting-oil-data">Extracting Oil Data</h3> <p>Oil emission data has traditionally been extremely hard to find. The source data for most oil fields is often secret and some of the models needed to calculate total greenhouse gas emissions have been proprietary. The Carnegie Endowment made a major contribution to understanding the climate impacts of fossil fuels with the <a href="http://carnegieendowment.org/specialprojects/OilClimateIndex/?lang=en">Oil-Climate Index</a>. The Oil Climate Index is the first study using entirely open-source models for evaluating greenhouse gas emission. These models were developed in a collaboration by Carnegie Endowment’s Energy and Climate Program, Stanford University, and the University of Calgary. In addition, the Oil Climate Index also collected model input data for 30 popular and emerging oils.</p> <figure class="align-center"> <img src="https://cloud.githubusercontent.com/assets/7108211/7837981/6a90bbec-0458-11e5-9153-5b71a08f20b8.png" alt="Scatter plot" width="759" height="587" /> <figcaption>Explore relationships between oil properties</figcaption> </figure> <p>The data is extremely complex and nuanced. Depending on what you want you want to do with an oil, making diesel vs jet fuel, different oils may be better. An improvement in an extraction or refinement process may have a significant climate benefit for one oil but not for another. To make this data useful to scientists, investors, policy makers, and interested citizens we built a <a href="http://oci.carnegieendowment.org/">flexible data exploration tool</a>. The tool makes reasonable assumptions to allow immediate comparison, but also allows users to explore how specific factors change the overall climate impact of each oil. Most importantly, all of the data and the modeling methodologies are open and available for download.</p> <h3 id="designing-for-complexity">Designing for complexity</h3> <p>We limited graphing options to the most meaningful properties to provide flexibility without unnecessary complexity. To visualize the Oil-Climate Index, we weren’t just showing static data, we were visualizing results of complex models. Processing all the data in the browser is impractical. Instead, we picked several model input parameters that are most meaningful and calculated their data up front. We built processing tools behind the site to recalculate this data as Carnegie extracts data on new oils.</p> <figure class="align-center"> <img src="https://cloud.githubusercontent.com/assets/848934/7825084/f6d005fe-03d6-11e5-8d93-59fb7964f586.png" alt="Model parameters" width="370" height="172" /> <figcaption>Model parameters allow the user to customize the model and data output</figcaption> </figure> <p>For more information on the Oil-Climate Index hit us <a href="https://twitter.com/developmentseed">on Twitter</a>, ask <a href="https://twitter.com/DxGordon">@DxGordon</a>, or check out these resources:</p> <ul> <li><a href="http://carnegieendowment.org/2015/05/05/avoiding-oil-climate-collision/i876">http://carnegieendowment.org/2015/05/05/avoiding-oil-climate-collision/i876</a></li> <li><a href="http://carnegieendowment.org/2015/03/11/know-your-oil-creating-global-oil-climate-index/i3oy">http://carnegieendowment.org/2015/03/11/know-your-oil-creating-global-oil-climate-index/i3oy</a></li> <li><a href="http://www.koomey.com/post/119950474783">http://www.koomey.com/post/119950474783</a></li> </ul> 2015-05-26T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/26/oci-launch We're sending a delegation to State of the Map US http://www.developmentseed.org/blog/2015/05/26/gearing-up-for-sotm <p><a href="http://stateofthemap.us/">State of the Map US</a> is around the corner. We’re sending a robust delegation to the United Nations for the biggest OpenStreetMap conference ever.</p> <p>This is a critical meeting for OSM and for open data. OpenStreetMap is becoming too big to fail. It is now critical infrastructure for everything from urban planning to disaster response. We’ll join 1000 open mappers from dozens of countries at a truly international venue to discuss new opportunities and responsibilities for OSM.</p> <p>We’ll be talking about:</p> <ul> <li><strong>OSM-as-a-platform</strong> - What if governments could roll OSM internally as an open source stack to manage road data between agencies? Will wider use of software behind OSM lead to better OSM data? Better OSM software? Check out <a href="http://stateofthemap.us/osm-as-a-platform/">Olaf’s talk</a> on our experience <a href="https://developmentseed.org/blog/2015/04/15/openstreetmap-for-government/">rolling out OSM-as-a-platform in the Philippines</a>. Or talk to <a href="https://twitter.com/obirdman">Olaf</a>, <a href="https://twitter.com/dereklieu">Derek</a>, or <a href="https://twitter.com/anandthakker">Anand</a>.</li> <li><strong>Organizing mapathons</strong> - <a href="https://twitter.com/tolomaps">Robin</a> will deliver a <a href="http://stateofthemap.us/lightning-talks-sun/">lightning talk on organizing mapathons</a>. Hit her up to talk about making mapathon magic.</li> <li><strong><a href="https://github.com/hotosm/OpenAerialMap">OpenAerialMap</a></strong> - OpenAerialMap will be an open catalog of satellite and aerial imagery. Find <a href="https://twitter.com/nas_smith">Nate</a>, <a href="https://twitter.com/scisco7">Alireza</a>, and <a href="https://twitter.com/kamicut">Marc</a> to talk about our contributions to OAM.</li> <li><strong>OSM Meta Data</strong> - Want to understand the OSM community? The answers are in the metadata. Find <a href="https://twitter.com/kamicut">Marc</a> and <a href="https://twitter.com/drewbo19">Drew</a> to talk about our efforts to <a href="https://developmentseed.org/blog/2015/04/14/tracking-osm-metadata/">open OSM changeset metadata</a> and <a href="https://developmentseed.org/blog/2014/07/01/osm-in-five-fastest-cities/">measure OSM growth</a></li> <li><strong>Humanitarian OpenStreetMap Team</strong> - <a href="https://twitter.com/nas_smith">Nate</a> is always scheming about how to improve the HOT imagery pipeline. We all have a ton of HOT love and thoughts on OSM in disaster response.</li> </ul> <p>We hope to see you there. There are <a href="http://www.eventbrite.com/e/state-of-the-map-us-2015-new-york-city-tickets-15437946313">still a few tickets left</a> if you want to join us in New York and geek out on open mapping.</p> 2015-05-26T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/26/gearing-up-for-sotm Geospatial World Forum kicks off http://www.developmentseed.org/blog/2015/05/25/meet-us-at-geospatial-world-forum <p class="dropcap">The <a href="http://geospatialworldforum.org/">Geospatial World Forum</a> kicks off today in Lisbon. We will be talking with NGOs, government agencies, and researchers about how they can use open geospatial data and open source software. Our Lisbon team will host an open geospatial happy hour on Thursday.</p> <p>Open geospatial data is some of the most valuable data that government produce, generating billions of dollars in economic value and tremendous social good. Open geospatial data underlies everything from critical climate change research to many of the apps in your favorite app store.</p> <p><a href="https://developmentseed.org/projects/libra/">Libra</a> and <a href="https://developmentseed.org/projects/libra/">Landsat-util</a> are powerful open source tools for processing and analyzing Landsat data. On Thursday at 11:00 I’ll talk at the <a href="http://geospatialworldforum.org/sessions.asp?Sp_Department=Open%20Data">Open Data track</a> about using these tools in combination with other open software like <a href="http://qgis.org/en/site/">QGIS</a> to work with open satellite data. At 9:00 Thursday <a href="https://twitter.com/scisco7">Alireza</a> and <a href="https://twitter.com/nas_smith">Nate</a> will offer a hands on training on open tools for satellite data at the <a href="http://geospatialworldforum.org/workshop.asp?Sp_Department=Open%20Source%20Imagery%20Tools">the Open Source Imagery Tools workshop</a>.</p> <p>We’ll continue the conversation about open data, mapping and satellites at an <a href="http://opengeospatialhappyhour.splashthat.com/">Open Geospatial Happy Hour</a> that we’re organizing with our friends from <a href="http://planet.com">Planet Labs</a> on Thursday evening. If you are interested in open geospatial data, please join us. You don’t need to be at the conference to attend.</p> <p>We look forward to meeting old and new geo friends while you are in Lisbon. If you want to meet me and our Lisbon team <a href="https://twitter.com/obirdman">hit me up on Twitter</a>.</p> 2015-05-25T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/25/meet-us-at-geospatial-world-forum GitHub cards http://www.developmentseed.org/blog/2015/05/14/github-cards <p class="dropcap">We love <a href="https://github.com/">GitHub</a>. We work in the open. We have well over 100 public projects on GitHub and contribute to many others. We’ve helped dozens of NGOs, International Organizations, and Government Agencies to open source their code on GitHub.</p> <p>When we redesigned our website, we wanted to promote open source and to make it easy to find our code on GitHub. So we created GitHub cards.</p> <h3 id="whats-a-github-card">What’s a GitHub Card?</h3> <p>A GitHub card is a styled card that clearly calls out that this is an open source project, links to the project on GitHub, and shows statistics like forks and stargazers that show the depth of the community around that tool.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/github-card-example@2x.png" alt="GitHub card" width="370" height="172" /> <figcaption>A GitHub card example</figcaption> </figure> <p>GitHub cards are a visual way to promote open source and to make it easier for others to participate in our projects by requesting features or contributing code.</p> <h3 id="let-a-thousand-github-cards-bloom">Let a Thousand GitHub Cards Bloom</h3> <p>GitHub cards aren’t natively supported by GitHub. We built them into our site using a simple visual wrapper and a bit of code to pull live data from our public GitHub repos.</p> <p>We’d love to see more sites use GitHub cards. A simple visual flag for open projects across websites can reinforce how much of the web is built on open technologies. There are solid open source options for nearly any technology need. GitHub cards can make them more obvious.</p> <p>If you want to add GitHub cards to your own site, feel free to use our code. Its <a href="https://creativecommons.org/publicdomain/zero/1.0/">open</a>.</p> <p data-height="268" data-theme-id="0" data-slug-hash="wBVOXe" data-default-tab="result" data-user="danielfdsilva" class="codepen">Check out the source code on <a href="http://codepen.io/danielfdsilva/pen/wBVOXe/">codepen</a>.</p> <script async="" src="//assets.codepen.io/assets/embed/ei.js"></script> <p><br /> </p> <p><em>Update: Within a few hours or posting, <a href="https://github.com/gmaclennan">@gmaclennan</a> from <a href="http://ddem.us/">Digital Democracy</a> built a tool to generate github cards. Awesome. Check it out:</em></p> <iframe width="320px" height="155px" frameborder="0" src="http://lab.digital-democracy.org/github-card/?repo=digidem/github-card&amp;title=Github Card&amp;link=http://lab.digital-democracy.org/github-card"></iframe> 2015-05-14T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/14/github-cards The Connectivity Atlas http://www.developmentseed.org/blog/2015/05/01/con-atlas <p>We recently launched the first iteration of the <a href="https://atlas.developmentseed.org/">Connectivity Altas</a>, a project to map infrastructures on a global scale. From roads and rivers to internet and electricity lines, these intricate and vast networks exist everywhere we are. Considering that infrastructure is a broad classification, mapping out these different systems provides a unique insight into how these networks are dispersed onto the globe. Since the Connectivity Atlas was our first significant dive into <a href="https://www.mapbox.com/mapbox-gl/">Mapbox GL</a>, we wanted to share some notes about our experience.</p> <h4 id="the-challenge-of-complex-data">The challenge of complex data</h4> <p>I started to build the map using <a href="https://www.mapbox.com/mapbox-studio/#darwin">Mapbox Studio</a> importing each layer into Studio as a separate data source and styling each layer using <a href="https://www.mapbox.com/tilemill/docs/manual/carto/">Carto CSS</a>. This workflow worked fine for a while, but over time might have led to hundreds of sources and styles pushing the limit of our Mapbox account and our organizational skills. This approach would also lead to performance issues as our data sets increase.</p> <p><img src="https://camo.githubusercontent.com/b1271ec1effe868597161b716ca7511650ac354a/68747470733a2f2f6661726d392e737461746963666c69636b722e636f6d2f383733342f31373036363335313939325f376630386436353731345f622e6a7067" /></p> <h4 id="rendering-vector-data-in-the-browser">Rendering vector data in the browser</h4> <p>We considered alternative workflows and ultimately decided to use Mapbox GL for various reasons.</p> <ul> <li>We like vector tiles (super crisp)</li> <li>We like rendering styles in the browser (it makes for faster styling / instant results)</li> <li>Better organization - we used a jekyll collection to add styles into a master JSON</li> </ul> <p><img src="https://cloud.githubusercontent.com/assets/10968362/7027986/f3fccf92-dd20-11e4-9ad1-3dc64050d8a6.gif" /></p> <h4 id="what-we-learned">What we learned</h4> <p>Vector tile platforms like GL allow for much faster rendering of complex data by using the graphics processor in the same way that a video game does. The tools for working with Mapbox GL are still limited. For example, we had to write our own tooltip functionality to expose the meta information in each layer. But we know the Mapbox folks are cooking up some new tools for GL, and we look forward to using GL for other projects.</p> <h4 id="open-infrastructure-data">Open Infrastructure Data</h4> <p>All of the data on the Connectivity Atlas is open and available for reuse. This is a collaborative project, and you can participate by sharing and suggesting data. This way we can produce a map of our profound inter-connectedness as well as an oddly beautiful web of global infrastructure.</p> 2015-05-01T02:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/05/01/con-atlas Our new website - still Jekyll http://www.developmentseed.org/blog/2015/04/20/building-with-jekyll <p class="dropcap">We recently redesigned our website from the ground up. While the look and organization changed, the underlying technologies have not. We are still using Jekyll to power the site, just as <a href="https://developmentseed.org/blog/2011/09/09/jekyll-github-pages/">we’ve done since 2011</a>. Jekyll has come a long way and we’re taking full advantage of some of its new features.</p> <h3 id="collections">Collections</h3> <p>Collections are the <a href="http://jekyllrb.com/">Jekyll</a> way to separate content types. They allow us to create new content types and to treat them differently. We are using collections for our team and project pages. Collection items have the attributes of “posts” in Jekyll with some subtle differences, like being ordered alphabetically in ascending order or not needing to have the date on the name. With collections we can achieve a greater content separation thus making the whole site more organized.</p> <h3 id="list-views">List views</h3> <p>Jekyll still has some challenges when it comes to paginate collections or categories. We solved this problem by rendering our list views (projects and blog posts) client side.</p> <p>Jekyll outputs a JSON file that we use to render the content with ejs templates. This allows us to have greater control over content output and prevents your browser from blowing up when trying to render our 650+ blog posts at once.</p> <h3 id="wrangling-the-json-api">Wrangling the JSON API</h3> <p>Jekyll’s JSON output can lead to a familiar comma problem. When Jekyll is generating a JSON file is common to see code like this:</p> <pre><code>{% raw %}[ {% for post in site.projects %} { "title": "{{ post.title }}" } {% unless forloop.last %},{% endunless %} {% endfor %} {% endraw %}] </code></pre> <p>This will work fine when every element of the array is being printed, but if you start introducing conditions the <code>forloop.last</code> can’t be used to control the comma anymore.</p> <p>For example we’re only showing posts that are not hidden. Image we’ve only three posts of which the third one is hidden. The resulting JSON would be:</p> <pre><code>[ { "title": "Project one" } , { "title": "Project two" } , ] </code></pre> <p>See the trailing comma? That little thing would break the JSON. It’s there because our last visible project is <code>Project two</code> but that is not the last element in the array so <code>forloop.last</code> doesn’t kick in and the comma is printed anyways.</p> <p>The workaround for this situation is easy. You just have to print the comma before the element except for the first time:</p> <pre><code>{% raw %}[ {% assign comma = false %} {% for post in site.projects %} {% unless post.hidden == true %} {% if comma %},{% endif %} {% assign comma = true %} { "title": "{{ post.title | escape_once }}" } {% endunless %} {% endfor %} {% endraw %}] </code></pre> 2015-04-20T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/20/building-with-jekyll A Modern Imagery Processing Pipeline http://www.developmentseed.org/blog/2015/04/17/ad-platform-launch <p>Satellite data is a tremendously powerful resource for governments and development organizations. We built a <a href="https://developmentseed.org/blog/2015/03/10/releasing-landsat-v05/">suite</a> <a href="https://developmentseed.org/blog/2015/01/22/announcing-libra/">of</a> <a href="https://github.com/developmentseed/landsat-api">tools</a> to make open Landat data more accessible and useable. These allow our development partners to process imagery and perform analysis quicker, and that can make all the difference in rapidly evolving situations.</p> <p>Often our partners need commercial imagery with greater resolution and refresh times than what Landsat 8 offers. We have great partnerships with commercial imagery providers, to offer all sorts of imagery. Too often receiving and processing commercial imagery is a huge pain point that slows us down and makes it harder to make use of the data. As developers we know it could be better. </p> <p><a href="https://medium.com/@astrodigital/imagery-processing-pipeline-launches-3f92c30932ec">Astro Digital</a> gave us an opportunity to rebuild this workflow from the ground up. We’ve worked closely with their team to build a satellite imagery pipeline for developers and end users. We just launched a browsing and publishing platform with Astro Digital to allow anyone to discover, process, and share satellite imagery in an incredibly quick and intuitive manner. A process that previously could take days has now been cut down to minutes.</p> <h3 id="api-first">API first</h3> <p>We built an end-to-end data processing pipeline that feeds a powerful data API that unlocks possibilities for others. We broke down the fundamental goals of the Platform and built API calls around each. Those goals were to search, process, and publish. </p> <p>We built and exposed a perfomant Elasticsearch-powered endpoint, based on our previous <a href="https://github.com/developmentseed/landsat-api">landsat-api</a> work, that will allow for complex queries to find exactly the data that is needed. </p> <p>{% highlight bash %}<br /> $ curl https://api.astrodigital.com/v1/search?search=cloudCoverageFull:[0+TO+20]<br /> {% endhighlight %}</p> <p>But how to process the imagery? We extended our existing open source <a href="https://github.com/developmentseed/landsat-util">landsat-util</a> tool to handle varying band combinations and the API offers several including true color, vegation health false color and urban false color.</p> <p>{% highlight bash %}<br /> $ curl https://api.astrodigital.com/v1/methods<br /> {% endhighlight %}</p> <p>And finally, there is a simple request that can be made to process the imagery and receive a tiled map URL. This URL can be used with tools like <a href="https://mapbox.com">Mapbox</a> or <a href="http://leafletjs.com">Leaflet</a> to build upon the processed imagery in any way. Full documentation, including interactive samples, can be found at <a href="http://docs.astrodigital.com">docs.astrodigital.com</a>.</p> <p>{% highlight bash %}<br /> $ curl -X POST –data “sceneID=LC80430332014262LGN00&amp;process=urbanFalse&amp;satellite=l8” https://api.astrodigital.com/v1/publish<br /> {% endhighlight %}</p> <p>{% highlight json %}<br /> {“status”:”Image is being processed.”}<br /> {% endhighlight %}</p> <h3 id="frictionless-publishing">Frictionless publishing</h3> <p>We are using this data pipeline to power an extremely easy and visual imagery browser and publishing tool. We started with <a href="https://developmentseed.org/blog/2015/01/22/announcing-libra/">Libra</a> as a base and modified it to meet the Astro Digital specific workflow. Libra was already designed to be quick and intuitive. We added a simple publish workflow that will process and publish images and email a link to the tiled map after processing has completed. For images processed within this visual workflow, the email contains a link to an embeddable map that can be used anywhere across the web. </p> <p><img src="https://cloud.githubusercontent.com/assets/848934/7208412/996efb4a-e50e-11e4-8214-4d63c477944e.gif" alt="publish" /></p> <p>Working with Astro Digital, we built a modern publishing pipeline that we hope will push the entire industry to build more usable tools. This is good for the industry and good for users, particularly the small governments and development organizations that are the next wave of power satellite data users.</p> 2015-04-17T15:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/17/ad-platform-launch An OpenStreetMap for Government http://www.developmentseed.org/blog/2015/04/15/openstreetmap-for-government <p>The software that powers OpenStreetMap (OSM) is a fantastic open source tool for governments to manage road data. We are working with the World Bank and the Philippine government to deploy OSM software to create a collaborative tool for national and municipal authorities to manage road data.</p> <p>OSM-as-a-platform gives governments a powerful open source option for enterprise management of road data. The OpenStreetMap community benefits with more investors in the OSM ecosystem and more OSM-ready data.</p> <h3 id="osm-and-government">OSM and Government</h3> <p>OpenStreetMap is a powerful tool for collaborative mapping of critical infrastructure. Over two million OSM users have contributed 280 million road, rail, and waterway segments.</p> <p>Governments are ideal OSM users. Governments build and maintain roads and manage public transit. Governments have powerful incentives to provide road data in a manner most useful to citizens. However legal, policy, and cultural considerations prevent governments from engaging in OSM.</p> <ul> <li>Government may have a requirement for authoritative control of data for 911 purposes.</li> <li>OSM’s share-alike requirements may run foul of government publishing obligations.</li> <li>Agency IT Departments may be hesitant to authorize external services for maintaining critical infrastructure data.</li> <li>Governments may have additional data or workflow requirements that aren’t supported by OSM.</li> </ul> <h3 id="openroads---osm-for-the-philippines">OpenRoads - OSM for the Philippines</h3> <p>The National Government in the Philippines needs the ability to manage and collaboratively edit road data in much the same way as OSM, but with their own unique set of data and users. Different portions of the road network are managed by authorities at the national, provincial, municipal, or barangay. These authorities also need tools to manage data on road improvement projects, from evaluating proposed resurfacing projects to tracking the proper completion of a funded road expansion.</p> <p>The OSM software provides fantastic tools for a whole-of-government approach to road management. Working closely with the World Bank, we are building the OpenRoads Network Editor, an OSM-based system for road management across Philippines’ various road authorities. We built this using OSM-as-a-platform and utilizing tools like <a href="http://ideditor.com/">iD</a> and <a href="https://github.com/osmlab/to-fix">to-fix</a>.</p> <p><img src="https://cloud.githubusercontent.com/assets/7108211/7166919/1d3c0972-e37f-11e4-9b44-1be85d5569dd.png" alt="" /></p> <p>This tool will allow the Government to create a full map and inventory of the countries 200,000 KM of roads. Ultimately it will also provide the analytical tools to allow the Philippines to make better decisions on infrastructure investments with real-time data.</p> <p>The World Bank recognizes the value of government managing data with collaborative open source tools. This week the World Bank is introducing this platform at a summit of the Philippine’s 1490 mayors. Today we are at the World Bank for <a href="http://www.worldbank.org/en/events/2015/03/31/big-data-for-a-more-resilient-future"><em>Big Data for a More Resilient Future</em></a> to present OpenRoads and how OSM-as-a-platform can benefit other governments.</p> <h3 id="open-tools-for-open-government">Open Tools for Open Government</h3> <p>OpenRoads is part of Development Seed’s ongoing efforts to create <a href="//developmentseed.org/set/open-tools-for-open-data/">open tools to support open government</a>.</p> <p>The OSM software ecosystem provides governments with a powerful open source alternative for managing road data. The OpenStreetMap community also benefits. These deployments create additional investors in the OSM software and more OSM-ready data.</p> 2015-04-15T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/15/openstreetmap-for-government Tracking Metadata in Real-time http://www.developmentseed.org/blog/2015/04/14/tracking-osm-metadata <p>This week we’re releasing more tools to track OpenStreetMap metadata. Together with the American Red Cross, we’re launching <a href="https://github.com/osmlab/osm-meta-api">OSM Metadata API</a>, a tool to help enable analysis of OSM’s rich metadata at the user and comment level. By using hashtags in changeset comments such as <a href="http://www.missingmaps.org/">#MissingMaps</a>, API access can enable groups like the Red Cross to gain feedback from OSM deployments. </p> <p>We’re building off our <a href="https://developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata/">osm-meta-util work we released several weeks ago</a> to store and index OSM metadata. We built the API on <a href="https://www.elastic.co/">Elasticsearch</a> and <a href="https://nodejs.org/">Node.js</a>. Using Elasticsearch, we index and store metadata information such as changeset_id, user, created_at, bounding box, comment. You can filter these logs by hashtag and keyword as they come in real-time, as well as build a database of historical metadata logs. </p> <p>We’re rolling out a sample API today with the code. You can access the endpoint here: <a href="https://api.developmentseed.org/osm">https://api.developmentseed.org/osm</a>. </p> <p>You can use the full power of Apache Lucene Search to browse the data. For example, to browse for the #missingmaps hashtag between two dates:</p> <pre><code>https://api.developmentseed.org/osm?search=created_at:[2015-04-08T00:00:00Z+TO+2015-04-09T00:00:00Z]+AND+comment:"missingmaps" </code></pre> <p>In the past week, #missingmaps events have helped make 69,865 edits to OpenStreetMap in places like Haiti, Iraq, Zimbabwe, and Tanzania. </p> <p><img src="/assets/graphics/content/osm-meta-api-release-cover.png" alt="" /></p> <p>Check out the <a href="https://github.com/osmlab/osm-meta-api/wiki/API-Guide">API guide</a> for full endpoint documentation. </p> <p>All code is available via a <a href="https://github.com/osmlab/osm-meta-api">GitHub repository on OSM Lab</a>. To get started quickly, you can <a href="https://github.com/osmlab/osm-meta-api#osm-meta-api">deploy the API as a Heroku app</a>. Try it out and contribute back to the OSM community.</p> 2015-04-14T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/14/tracking-osm-metadata Howdy, Anand Thakker! http://www.developmentseed.org/blog/2015/04/02/howdy-anand <p><img src="https://farm9.staticflickr.com/8751/16827357019_7b62379f1f_b.jpg" alt="Anand looking good" /></p> <p><a href="https://developmentseed.org/team/anand-thaaker/">Anand Thakker</a> likes the messy stuff. Whether he’s using historic satellite imagery to measure electrification or finding better ways to teach high school math, Anand embraces hard problems. He brings creativity, energy, and thoughtfulness to every project.</p> <p>Anand is going to help us to build powerful tools that make a real difference. Anand has always been engineering. He built a search engine with his high school friends. At a startup right out of college, he worked on tools for analyzing and debugging XML-based web services. Then, for seven years, he devoted his energy to teaching high school math and computer science in Baltimore. There too, Anand engineered curriculum and teaching methods for improving students’ problem solving skills.</p> <p>Anand studied math and computer science at Carnegie Mellon University and has a masters in education from Harvard University. Give him a shout on <a href="https://github.com/anandthakker">github</a> or <a href="https://twitter.com/anandthakker">twitter</a>.</p> 2015-04-02T16:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/02/howdy-anand Welcome Emily Bremner! http://www.developmentseed.org/blog/2015/04/02/welcome-emily <p><img src="https://farm9.staticflickr.com/8728/16825813978_9dcf8abc28_b.jpg" alt="Emily and the tree" /></p> <p>International development is hard. <a href="https://developmentseed.org/team/emily-bremner/">Emily Bremner</a> first saw this at a school in Kenya for Somali refugees. She was 16. Over more than a decade since, Emily has pushed to improve peoples’ lives in places from El Salvador to Afghanistan. She’s been teargassed in Djibouti and battled bureaucracy back in DC.</p> <p>Emily has worked on all sides of international development and has seen its successes and its failures. Emily is going to work on our business strategy and operations. She is going to help us to better solve hard social problems and to make sure our work is impactful.</p> <p>We had the opportunity to work with Emily when she was at <a href="http://democracyinternational.com/">Democracy International</a> to open up election data in <a href="https://developmentseed.org/blog/2014/06/23/open-data-lebanon/">Lebanon</a> and <a href="https://developmentseed.org/blog/2014/05/22/tunisia-open-election-data/">Tunisia</a>. She understood then the opportunities and the limits of data and technology to make meaningful change. We look forward in working with her to expand those limits.</p> <p>Say Bienvenue or Hola to Emily <a href="https://twitter.com/emilybremner23">on twitter</a>.</p> 2015-04-02T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/02/welcome-emily Optimizing Landsat-util http://www.developmentseed.org/blog/2015/03/28/twice-as-fast <p>Two weeks ago we launched a <a href="https://developmentseed.org/blog/2015/03/10/releasing-landsat-v05/">new version of Landsat-util, v0.5</a>, our utility for searching and processing Landsat satellite imagery. This version is now faster than it was before. To do this we rewrote most of the internals to use flexible python frameworks.</p> <p>Below is a deep dive into how we’ve improved Landsat-util to make it faster and easier to use. </p> <h2 id="dependency-hell">Dependency Hell</h2> <p>Landsat-util downloads Landsat files, pulls out the individual bands representing wavelengths of light, corrects contrast, warps them and combines them to make a colored image you can add on a web map.</p> <p>Initially, landsat-util was written as a command line wrapper to <a href="https://www.mapbox.com/blog/processing-landsat-8/">existing pipelines</a>:</p> <ol> <li>Scale bands with <code>gdal-translate</code></li> <li>Warp with <code>gdal-warp</code> to the correct projection</li> <li>Combine bands with <code>ImageMagick</code></li> <li>Contrast correct with <code>OpenCV</code></li> <li>Pansharpen with <code>orfeoToolbox</code></li> <li>Gamma correct with <code>ImageMagick</code></li> <li>Add geographic information back with <code>gdal_edit</code></li> </ol> <p>We needed a lot of dependencies to process an image, and they’re not particularly optimized for scripting. ImageMagick, GDAL, orfeoToolbox and openCV are monolithic frameworks that don’t allow for cherry-picking functions. Installing all these frameworks can be a <a href="https://github.com/developmentseed/landsat-util/issues?utf8=%E2%9C%93&amp;q=install">painful experience</a>.</p> <p>This toolchain combination is also tedious because it creates inherent bottlenecks. To communicate between all tools, we need to write temporary files to disk and read them back in at every step.</p> <h2 id="enter-rasterio">Enter rasterio</h2> <p><a href="https://github.com/mapbox/rasterio">Rasterio</a> is a great python library written by <a href="https://twitter.com/sgillies">Sean Gillies</a> at Mapbox to work with raster data. It wraps around gdal and abstracts the band data as <a href="http://www.numpy.org/">Numpy</a> arrays. </p> <p>By standardizing the input, rasterio allows us to minimize our dependencies and use fast, in-memory, scientific libraries like <a href="http://scikit-image.org/">scikit-image</a>. </p> <p>Here’s the guts of our new process:</p> <h3 id="read-in-bands-with-rasterio">Read in bands with rasterio</h3> <p>{% highlight python %}<br /> with rasterio.drivers():<br /> with rasterio.open(‘LC82040522013123LGN01_B4.TIF’) as band4:<br /> with rasterio.open(‘LC82040522013123LGN01_B3.TIF’) as band3:<br /> with rasterio.open(‘LC82040522013123LGN01_B2.TIF’) as band2:<br /> with rasterio.open(‘LC82040522013123LGN01_B8.TIF’) as band8:<br /> band4_s = band4.read_band(1)<br /> band3_s = band3.read_band(1)<br /> band2_s = band2.read_band(1)<br /> band8_s = band8.read_band(1)<br /> {% endhighlight %}</p> <h3 id="scale-bands-with-scikit">Scale bands with <code>scikit</code></h3> <p>{% highlight python %}<br /> from skimage import transform as sktransform</p> <p>band4_s = sktransform.rescale(band4_s, 2)<br /> band3_s = sktransform.rescale(band3_s, 2)<br /> band2_s = sktransform.rescale(band2_s, 2)<br /> {% endhighlight %}</p> <h3 id="warp-with-rasterio">Warp with <code>rasterio</code></h3> <p>{% highlight python %}<br /> for color, band in zip([r,g,b,b8], [band4_s, band3_s, band2_s, band8_s]):<br /> reproject(band, color, <br /> src_transform = src.transform,<br /> src_crs = src.crs,<br /> dst_transform = dst_transform, <br /> dst_crs = dst_crs,<br /> resampling = RESAMPLING.nearest)<br /> {% endhighlight %}</p> <h3 id="pansharpen-using-numpy-operations">Pansharpen using <code>numpy</code> operations</h3> <p>{% highlight python %}<br /> m = r + b + g<br /> pan = 1/m * b8<br /> r = r * pan<br /> b = b * pan<br /> g = g * pan<br /> {% endhighlight %}</p> <h3 id="contrast-correct-and-gamma-correct-using-scikit-image">Contrast-correct and gamma correct using <code>scikit-image</code></h3> <p>{% highlight python %}<br /> # using CLAHE<br /> from skimage.exposure import equalize_adapthist<br /> for band in [r,g,b]:<br /> band = equalize_adapthist(band, clip_limit=0.02)<br /> {% endhighlight %}</p> <h3 id="write-to-disk-using-rasterio">Write to disk using <code>rasterio</code></h3> <p>{% highlight python %}<br /> with rasterio.open(<br /> tiffname,’w’, driver=’GTiff’,<br /> width=dst_shape[1],height=dst_shape[0],<br /> count=3,dtype=numpy.uint8,<br /> nodata=0,<br /> transform=dst_transform,<br /> photometric=’RGB’,<br /> crs=dst_crs) as dst:<br /> for k, arr in [(1, r), (2, g), (3, b)]:<br /> dst.write_band(k, arr)<br /> {% endhighlight %}</p> <p>The toolchain consists of only python libraries, and no other dependencies. Rasterio inherently supports GeoTiff so we don’t lose geo-information along the way.</p> <p>Landsat-util is open source, and we encourage developers to improve on our process. Fork <a href="https://github.com/developmentseed/landsat-util">our repo</a>!</p> <h2 id="how-fast">How fast?</h2> <p>By using <code>rasterio</code>, <code>numpy</code> and <code>scikit</code>, we eliminate the disk bottleneck, and we regain transparent control over the pipeline.</p> <p>We ran tests on a couple of AWS machines. Each test was conducted 5 times and the resulting times were averaged. </p> <ul> <li>R3.large: 2-core 2.5 GHz Intel Xeon (Ivy Bridge) with 15GB RAM and 32GB SSD.</li> <li>C4.2xlarge: 8-core 2.9GHz Intel Xeon (Haswell) with 15GB RAM and 32GB SSD. </li> </ul> <h3 id="results">Results</h3> <div class="table-responsive"> <table> <thead> <tr> <th>instance type</th> <th>process type</th> <th>old-landsat-util</th> <th>new-landsat-util</th> <th>speedup</th> </tr> </thead> <tbody> <tr> <td>R3.large</td> <td>non-pansharpened</td> <td>252.7s</td> <td>122.05s</td> <td><strong>2x</strong></td> </tr> <tr> <td>R3.large</td> <td>pansharpened</td> <td>846.9s</td> <td>349.17s</td> <td><strong>2.4x</strong></td> </tr> <tr> <td>C4.2xlarge</td> <td>non-pansharpened</td> <td>216.6s</td> <td>106.67s</td> <td><strong>2x</strong></td> </tr> <tr> <td>C4.2xlarge</td> <td>pansharpened</td> <td>438s</td> <td>290s</td> <td><strong>1.5x</strong></td> </tr> </tbody> </table> </div> <p>On all but the largest machine, the new landsat-util is at least twice as fast as previously.</p> <p>And it gets better: a significant amount of time (about a minute on average) is used up to decompress the NASA bundle after downloading it. Landsat-util v0.5 takes advantage of <a href="https://developmentseed.org/blog/2015/03/19/aws-landsat-archive/">AWS’s new landsat archive</a> of unzipped bands, saving even more time.</p> <p>We hope you enjoy the new landsat-util! Fork it, modify it, break it or just use it and tell us about all the ways you’re incorporating satellite data in your apps. </p> 2015-03-28T02:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/28/twice-as-fast We're in the Philippines http://www.developmentseed.org/blog/2015/03/27/in-the-philippines <p class="dropcap">This week, <a href="/team/ian-schuler/">Ian</a> and I are in the Philippines working alongside the World Bank to set up collaborative, open source tools for government to track critical infrastructure, particularly road data.</p> <p>Good road data is crucial for smart development and disaster response. To help the national and local governments manage their road infrastructure, we’ll be building on open tools such as <a href="http://ideditor.com/">iD</a>, the popular editor for <a href="http://www.openstreetmap.org/#map=18/14.08530/121.14943">OpenStreetMap</a>. Our goal is to use open-source tools to create better data infrastructure, at an enterprise level, tailored to the information needs of government.</p> <p>We are <a href="https://developmentseed.org/blog/2014/06/30/understanding-risk-forum/">confessed fans</a> of OpenStreetMap and the ecosystem that surround it. These tools are battle-tested and optimized for the quantity of data that flows daily through OSM. We think these tools can make government work better, and so far we’ve seen terrific responses from officials here.</p> <p>We’ll be in the Philippines until next Tuesday. Ping us on <a href="https://twitter.com/dereklieu">Twitter</a> if you’d like to meet and talk OSM and managing infrastructure data.</p> 2015-03-27T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/27/in-the-philippines Bem-vinda Caroline Portugal http://www.developmentseed.org/blog/2015/03/24/bem-vinda-caroline <p>Welcome Caroline! Designer and architect <a href="/team/caroline-portugal/">Caroline Portugal</a> is joining Development Seed.</p> <p><img src="/assets/graphics/content/caroline-portugal-card.jpg" alt="" /></p> <p>Caroline is going to build thoughtful, beautiful software. Caroline hails from Brazil, where she studied and practiced Architecture and Urban Planning. She worked on architecture projects in Brazil and the US ranging from <a href="http://estudioporto.com/379741/5800883/gallery/rio-2016-olympic-park-master-plan">Rio’s Olympic Park Master Plan</a> to the National Museum of African American History &amp; Culture. Caroline’s latest work has been in visual design. She recently completed a graduate degree in Graphic Design from MICA.</p> <p>Caroline’s work is moving. It is visually stunning. It is grounded in solving practical challenges with humanity, creativity and fun. Check out her work on <a href="http://estudioporto.com/379741/5686749/gallery/bike-in-park-out">bike lanes</a> and <a href="http://estudioporto.com/379741/5762373/gallery/the-future-of-contraception">the future of contraception</a> to get a sense of why we are delighted to have her on the team.</p> <p>Caroline speaks fluent Portuguese. Say bem-vinda to <a href="https://twitter.com/portugal_moura">Caroline on twitter</a>.</p> 2015-03-24T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/24/bem-vinda-caroline Powering Landsat Power Tools http://www.developmentseed.org/blog/2015/03/19/aws-landsat-archive <p class="dropcap">Amazon Web Services just opened Landsat on AWS, a publicly available archive of Landsat 8 imagery hosted on their reliable and accessible infrastructure. This investment by the AWS open data team has a big impact on our work to make satellite imagery more accessible.</p> <p><a href="https://libra.developmentseed.org/">Libra</a> is an open source Landsat imagery browser that we built with <a href="http://astrodigital.com/">Astro Digital</a>. Libra now has options on some scenes to download individual bands related to specific types of imagery analysis like NDVI, or Urban False Color. The most recent Landsat–8 images are now available for download up to two days sooner. Last week we rolled out a <a href="/blog/2015/03/10/releasing-landsat-v05/">new version of landsat-util</a>, our open source utility for processing Landsat imagery. The new version is much faster and allows you to build <a href="https://en.wikipedia.org/wiki/False_color">false color composites</a> on the fly. These improvements to Libra and landsat-util are possible because we started using Landsat on AWS, which is a publicly available archive of Landsat 8 imagery hosted on Amazon S3 that is <a href="http://aws.amazon.com/public-data-sets/landsat/">publicly available today</a>.</p> <p><a href="https://libra.developmentseed.org/"><img src="https://farm8.staticflickr.com/7636/16864977481_2e31af08b6_o.png" alt="libra-band-download" /></a></p> <p><em>Libra now allows you to download the satellite bands for advanced analysis</em></p> <p>Our newest releases of Libra and landsat-util utilize Landsat on AWS for 2015 imagery. Landsat on AWS provides 2015 imagery as unzipped individual bands. AWS makes this imagery available extremely quickly, often within hours of capture. We can pull only the data that we need and to work with it immediately.</p> <p>Landsat 8 imagery is an incredibly powerful resource. It is some of the most valuable open data produced by the US Government. Our partners rely on Landsat data for everything from evaluating droughts to tracking conflict. However, until now, individual bands of Landsat imagery has never been available via predictable download endpoints that we can integrate into our tools.</p> <p>Libra and landsat-util now allow our partners to get imagery sooner and process it faster. Speed and ease are critical to our partners who use this data to respond to natural disasters, prevent hunger, and monitor elections.</p> <p><img src="https://farm9.staticflickr.com/8683/16866164475_1b90a33b72_h.jpg" alt="gulf of guinea" /></p> <p><em>Landsat images of the Gulf of Guinea, processed with <a href="https://github.com/developmentseed/landsat-util">landsat-util</a></em></p> <p>Thanks to the AWS team and collaborators–<a href="https://twitter.com/nfwarmerdam">Frank Warmerdam</a> at <a href="https://www.planet.com/">Planet Labs</a>, <a href="https://twitter.com/vruba">Charlie Loyd</a> at <a href="https://www.mapbox.com/">Mapbox</a>, and Peter Becker and others from <a href="https://www.esri.com/">ESRI</a>–for building a Landsat archive with developers in mind.</p> <p><a href="https://github.com/AstroDigital/libra">Libra</a> and <a href="https://github.com/developmentseed/landsat-util">landsat-util</a> are open source and on Github. Go ahead and fork or contribute.</p> 2015-03-19T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/19/aws-landsat-archive Launching v0.5 of landsat-util http://www.developmentseed.org/blog/2015/03/10/releasing-landsat-v05 <p class="dropcap">We just released a new version of landsat-util, <a href="https://github.com/developmentseed/landsat-util/releases/tag/v0.5.0">version 0.5</a>. This version is lighter and has fewer dependencies. Landsat-util v0.5 downloads and processes images much faster and gives users more control.</p> <p><a href="https://pypi.python.org/pypi/landsat-util/0.5.0">Installing landsat-util</a> now is considerably easier for Mac and Linux users. We are still working to make it as easy to run on Windows.</p> <p>This new version reflects a significant change in our thinking on landsat-util. We’ve moved to using faster and simpler frameworks to optimize processing. We removed heavy dependencies, such as ImageMagick and Orfeo Toolbox, that were causing installation problems. We leveraged faster processing frameworks like <a href="https://github.com/mapbox/rasterio">Rasterio</a>, <a href="https://github.com/numpy/numpy">numpy</a>, and <a href="https://github.com/scipy/scipy">scipy</a>. These changes significantly optimize disk and memory usage resulting in faster and less error-prone processing. Landsat-util can now process images 3x faster.</p> <p><img src="{{ site.baseurl }}/assets/graphics/content/landsat-util-v05.gif" alt="Running the utility" /></p> <p>We’ll post more of the technical details on how we rebuilt landsat-util and what’s under the hood. In the meantime hit us up at FOSS4G in San Francisco all week to learn more.</p> 2015-03-10T15:30:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/10/releasing-landsat-v05 Collect and verify mobile reports http://www.developmentseed.org/blog/2015/03/10/mobile-reporting-workflow <p>A common scenario for mobile reporting looks like this:</p> <ul> <li>A group wants to collect reports from their own trusted network and also to crowdsource reports from the public.</li> <li>The group has some process to try to verify crowdsourced reports and needs to track verification.</li> <li>The group wants to publish this data using simple visuals that answer their core question and invites comparison.</li> </ul> <p>We recently worked with Pursue Lebanon to build a system for reporting breakdowns in service delivery in Lebanon’s Palestinian refugee camps. Pursue has been working with community organizations in all twelve refugee camps for the past five years. With more attention on the refugee situation in Lebanon, there are opportunities to push for improved service delivery with better, timely data. With so many tools for mobile data collection, you’d think that this should be easy to do with open source tools. It’s not. Here is our experience.</p> <h3 id="collecting-data">Collecting Data</h3> <p><a href="https://opendatakit.org/">OpenDataKit</a> is a great open source tool for data collection on Android. When you have control over the device your volunteers are using, ODK is great. <em>(Side note: If you like ODK, check out <a href="https://github.com/AmericanRedCross/openmapkit">OpenMapKit</a> an exciting project of the American Red Cross, SpatialDev, and Ona).</em></p> <p>Like many organizations, Pursue relies on reports from volunteers using their own phones and can’t expect them to report through an Android app. This is why the ODK ecosystem and tools like <a href="https://enketo.org/">Enketo</a> and <a href="https://formhub.org/">Formhub</a> are interesting. Enketo allows us to serve a simple web form on any device. Formhub gives us the ability to convert and manage Excel-based forms. While this ecosystem is great for authoring forms, we found that this system had some shortcomings:</p> <ul> <li>managing multiple datasets can be a challenge,</li> <li>API lacks some features,</li> <li>system is resource hungry, and</li> <li>difficult to deploy and maintain.</li> </ul> <p>We use these tools for what they are best at, generating web forms from Excel documents. We pull out the data via the API to use in other open source tools.</p> <p><img src="https://farm8.staticflickr.com/7651/16803250171_c8525e5500_o.png" alt="" /></p> <h3 id="verification">Verification</h3> <p>No tool that we looked at had good workflow for verification. Most had no verification or had only a simple yes/no toggle. Many organizations need to append further verification data and notes to each report. Pursue’s partners are collecting and verifying thousands of reports. To make this manageable, the system must have a workflow for quickly reviewing whether reports are verified true, verified false or are unverified.</p> <p><img src="https://farm8.staticflickr.com/7633/16181960354_f01be0bf2c.jpg" alt="" /></p> <p>We used Django to build a verification platform that is useful and usable for the data verifier who is triaging hundreds of reports a day and for the field organizer who is trying to track the status of single report.</p> <h3 id="publishing">Publishing</h3> <p>To publish and visualize the collected data, we designed a map and report interface that highlights the trends in service breakdowns between camps and within neighborhoods in camps.</p> <p>It is important to clearly show users whether information is verified. For simplicity, we use a simple checkbox to filter out all reports that are “unverified” or “verified false”, even though we distinguish between these in the verification platform. We showed the verification status prominently on all report listings.</p> <p>Reporters can select a neighborhood and choose a level of geographic precision. To protect the privacy of reporters, the system does not require reports to submit precise location information. Trusted administrators use the precise location when it is provided to verify reports. Regardless of the location precision, the public platform only shows reports to the neighborhood level.</p> <p><img src="https://farm9.staticflickr.com/8697/16182028384_eb77344140_c.jpg" alt="" /></p> <p>Join <a href="https://2015.foss4g-na.org/session/empowering-refugees-data">our talk on Wednesday morning</a> to hear more. We’ll follow up later in the day with a Birds of a Feather session discussing how the open source data collection community continues to grow.</p> 2015-03-10T14:30:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/10/mobile-reporting-workflow Hello San Francisco; Ready for FOSS4G http://www.developmentseed.org/blog/2015/03/09/in-sf-for-foss4g <p class="dropcap">We’re out in San Francisco this week for <a href="https://2015.foss4g-na.org/">FOSS4G North America</a>. Look for <a href="http://developmentseed.org/team/alireza-j/">Alireza</a> and I at the conference or at our session <a href="https://2015.foss4g-na.org/session/empowering-refugees-data">Wednesday morning at 10:30am</a>. Alireza will discuss using the OpenDataKit ecosystem to build mobile reporting and verification tools for refugee camps in Lebanon. We’ll also be at the Birds of a Feather session discussing the future of OpenDataKit.</p> <p><img src="https://farm9.staticflickr.com/8632/16581540300_e1a3d465be_c.jpg" alt="SF Landsat" /><br /> <em>Landsat 8 image over San Francisco on December 31, 2014</em></p> <p>This week we will release new versions of <a href="https://github.com/developmentseed/landsat-util">landsat-util</a> and <a href="http://libra.developmentseed.org/">Libra</a> with some powerful new features. We are also working on some tools for <a href="http://www.developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata/">managing OpenStreetMap data</a>. We look forward to collaborating with old and new friends in the open source community to improve the tools for mobile data collection, Open Geo, and satellite imagery processing. Tweet us at <a href="https://twitter.com/nas_smith">@nas_smith</a> or <a href="https://twitter.com/scisco7">@scisco7</a> to meet up. </p> 2015-03-09T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/09/in-sf-for-foss4g Tapping into OpenStreetMap Metadata http://www.developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata <p class="dropcap">We just launched <a href="https://github.com/osmlab/osm-meta-util">v0.1 of a new tool</a> to tap into OpenStreetMap changeset metadata. We built the tool in partnership with the American Red Cross as part of the infrastructure for tracking efforts such as <a href="http://www.missingmaps.org/">#MissingMaps</a>.</p> <p><a href="http://wiki.openstreetmap.org/wiki/Changeset">OpenStreetMap changesets</a> give us access to a wealth of metadata information that is not specifically geographic but incredibly rich. Metadata is helpful in understanding the changing nature of OSM. This is different from using geographic APIs like <a href="https://github.com/drolbr/Overpass-API">Overpass</a> because metadata contains commit text, number of edits, which editor was used, etc. With metadata, we can <a href="http://resultmaps.neis-one.org/osm-missingmaps">track hashtags</a>, analyze commit text or aggregate user metrics.</p> <p>In 2014 alone, users committed over 6 million changesets to OSM. As OpenStreetMap’s metadata grows, dealing with the sheer amount can be daunting. We built <a href="https://github.com/osmlab/osm-meta-util">osm-meta-util</a> as an experiment in making OSM metadata easier and faster to use.</p> <p><img src="{{ site.baseurl }}/images/gifs/osm-meta.gif" alt="Running the utility" /></p> <p>osm-meta-util focuses on two core functions: downloading the minutely compressed metadata files and serializing into JSON. We convert compressed OSM XML files containing multiple commits to a stream of JSON objects that can be piped to any tool or API. </p> <p>You can use the library in a Node application or as a command-line utility to download all the data between two dates: </p> <p>{% highlight javascript %}<br /> MetaUtil({<br /> ‘start’: process.argv[2],<br /> ‘end’: process.argv[3],<br /> ‘delay’: process.argv[4]<br /> }).pipe(process.stdout)<br /> {% endhighlight %}</p> <p>In combination with <a href="https://stedolan.github.io/jq/">jq</a>, to get a commit history we can simply run: </p> <p>{% highlight sh %}<br /> node app 001181708 001181721 1000 | jq -c ‘{date: .created_at, text: .comment}’<br /> {% endhighlight %}</p> <p>If you don’t give the tool any parameters, it will get the latest changesets and update every minute.</p> <p>We’re using this utility to experiment building a metadata API with the American Red Cross. But we know there are many more uses of the rich OSM metadata and want to see what others can do with the tool. Together with the American Red Cross we’ve put this on <a href="https://github.com/osmlab">OSM Lab</a>, a Github organization for OSM related projects. Follow the development of osm-meta-util on <a href="https://github.com/osmlab/osm-meta-util">Github</a>. </p> 2015-02-19T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata Open Data Day Garage Party http://www.developmentseed.org/blog/2015/02/13/odddc-garage-party <p class="dropcap">We love open data. And we love to talk about it over drinks with other open data lovers. Next Friday, February 20th at 7:00pm we are hosting an <a href="http://opendatadaydcparty.splashthat.com/">Open Data Day celebration</a> in the Mapbox Garage. Head over to the Garage after the first day of <a href="http://dc.opendataday.org/">Open Data Day DC</a> to talk about open tools for open data.</p> <p><a href="https://www.flickr.com/photos/developmentseed/16332412278/"><img src="https://farm9.staticflickr.com/8589/16332412278_ea7bcf4db8_c.jpg" alt="" /></a></p> <p>Even if you can’t make it to Open Data Day DC (or didn’t grab a spot on the now closed list), come over and share the #opendatalove.</p> <p>Let us know if you’re in by <a href="http://opendatadaydcparty.splashthat.com">RSVPing now</a>.</p> 2015-02-13T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/13/odddc-garage-party Data Hungry Happy Hour http://www.developmentseed.org/blog/2015/02/06/data-hungry-happy-hour <p class="dropcap">We’ll be at the <a href="http://www.tffchallenge.com/summit">Thought For Food Global Summit</a> next week working with some brilliant people on the biggest challenges in agriculture and feeding the hungry. Look for <a href="http://www.developmentseed.org/team/olaf-veerman/">Olaf</a> at the Summit.</p> <p>If you are in Lisbon for the Summit, please join us for a <a href="http://datahungryhappyhour.splashthat.com/">Data Hungry happy hour</a> on Thursday evening. We’ll be geeking out on better data, satellites, and sensors can contribute to better food policy. The Happy Hour kicks off at 18.30 in Fabulas in the center of Lisbon. The first couple of rounds are on us. You can <a href="http://datahungryhappyhour.splashthat.com/">RSVP here</a>.</p> <p><a href="http://datahungryhappyhour.splashthat.com/"><img src="https://cloud.githubusercontent.com/assets/751330/6078058/07dbb68a-aded-11e4-8d49-15a409237d70.png" alt="Data Hungry Happy Hour" /></a></p> 2015-02-06T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/06/data-hungry-happy-hour Welcome Mariano Arrien-Gomez http://www.developmentseed.org/blog/2015/02/02/welcome-mariano <p><a href="/team/mariano-arrien-gomez/">Mariano Arrien-Gomez</a> is an artist and designer. He builds <a href="https://instagram.com/mariano.ag/">beautiful graphics</a> and data visualizations that make our products more compelling, creative, and humane. Mariano directs his design skills toward the issues and topics he is passionate about, from preserving local parks, to genetically modified food, to soccer. He utilizes a range of <a href="http://marrieng.com/">visual mediums</a> including illustration, painting murals, and photography.</p> <p><img src="http://marrieng.com/images_dcbrau/dcbrau_pics_s06.jpg" alt="mariano" /></p> <p>In addition to his work at Development Seed, Mariano is active in the DC art scene. Mariano received a Bachelors of Fine Arts in Graphic Design from Virginia Commonwealth University. He is fluent in Spanish.</p> 2015-02-02T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/02/welcome-mariano Announcing Libra - the Landsat imagery browser you will love http://www.developmentseed.org/blog/2015/01/22/announcing-libra <p class="dropcap">We’ve been <a href="https://medium.com/@astrodigital/browsing-large-sets-of-satellite-imagery-7096db1a807f">working with Astro Digital</a> to produce <a href="http://www.developmentseed.org/blog/2015/01/15/dauria-image-search/">the most usable imagery browser</a>. Today we are releasing <a href="http://libra.developmentseed.org">Libra</a>, a fork of the Astro Digital browser for open Landsat data. Libra allows you to browse, sort, and download more than 275 Terabytes of open Landsat imagery as easily as booking an Uber.</p> <p><a href="http://libra.developmentseed.org/"><img src="https://farm9.staticflickr.com/8682/16157701429_8f417679a4_o.png" alt="libra" /></a></p> <h3 id="liberating-landsat">Liberating Landsat</h3> <p>We love open imagery. The global development organizations and developing governments that we work with use open satellite imagery for everything from evaluating disaster response, to tracking deforestation, to planning for drought. For our partners, open imagery isn’t just a matter of cost; it is a matter of licensing and distribution. They get immediate access to Landsat images and can analyze, manipulate, and distribute with almost no restrictions.</p> <p>To make Landsat data more useful, we’ve made it easier to use. We built two open source tools for working with Landsat data - <a href="https://github.com/developmentseed/landsat-util">Landsat-util</a> and <a href="https://github.com/developmentseed/landsat-api">Landsat API</a>. It used to take all day for Development Seed’s imagery specialists to turn Landsat data into imagery layers for online maps. With these two tools, any developer can do it in a matter of minutes.</p> <p>These tools gave us a huge head start in building Libra. Libra relies heavily on Landsat API to quickly query by date, geography, and cloud cover and get image URLs, scene centroids, scene boundaries, and other metadata. Using Landsat API as a backbone of Libra also encouraged us to make improvements and configuration changes on Landsat API such as changing the limits on requests and returned data and some error handling.</p> <p>Have a look at <a href="http://libra.developmentseed.org">Libra</a> and hit us with your feedback <a href="https://twitter.com/developmentseed">@developmentseed</a>.</p> <p><em>Feb 16 2015: We’ve updated this post to reflect the Dauria Geo name change to Astro Digital.</em></p> 2015-01-22T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/22/announcing-libra Introducing Development Seed Lisbon http://www.developmentseed.org/blog/2015/01/20/introducing-development-seed-lisbon <p class="dropcap">Last week we announced the opening of <a href="http://www.developmentseed.org/blog/2015/01/15/development-seed-opens-office-in-lisbon/">Development Seed Lisbon</a>. To kickstart our operations in Europe, we brought on our friends from Flipside, an experienced team working on meaningful open data projects for organizations around Europe. This move allows us to connect with partners and talent in the region, and also deliver quality work right out of the gate.</p> <h2 id="olaf-veerman">Olaf Veerman</h2> <p>Olaf will lead the Lisbon office, run projects, and help us establish a strong presence in Europe. He lived for many years in Latin America, working with small business networks, cooperatives and small farmer groups in Brazil, Uruguay and Venezuela. His experience in working with civil society organizatons around the world allows him to quickly understand our partners’ needs and help them use technology to increase their social and economic impact. </p> <p><img src="https://farm8.staticflickr.com/7570/16138839469_890fdabaa6_b.jpg" alt="" /></p> <p>You can connect with Olaf through <a href="http://twitter.com/obirdman">Twitter</a>.</p> <h2 id="daniel-da-silva">Daniel da Silva</h2> <p>Daniel brings solid engineering skills that he applies to anything from building light-weight frontends with well structured APIs, to deploying tools for offline/online data collection. He is a quick learner and problem solver whose technology expertise spans PHP, Node, Angular, Jekyll and Mongodb. Daniel is going to help us pick the right tool for the job and deliver quality work to our partners.</p> <p><img src="https://farm9.staticflickr.com/8613/16137670630_c7a9b6da29_b.jpg" alt="" /></p> <p>Find Daniel on <a href="https://github.com/danielfdsilva">Github</a>.</p> <h2 id="ricardo-mestre">Ricardo Mestre</h2> <p>Ricardo is a talented designer and front-end developer who pushes how modern technologies can be used to craft usable and engaging websites. He worked for some of the biggest companies in Portugal, but is most passionate about free culture and equality, which he contributes to through his music, art and <a href="http://duplos.org/">other projects</a>. Ricardo is going to have a big impact on the design and usability of our work.</p> <p><img src="https://farm8.staticflickr.com/7551/15705150643_5bb48dac7d_z.jpg" alt="" /></p> <p>You can find Ricardo on <a href="http://twitter.com/ricardoduplos">Twitter</a>.</p> 2015-01-20T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/20/introducing-development-seed-lisbon Building a Better Satellite Imagery Browser http://www.developmentseed.org/blog/2015/01/15/dauria-image-search <p class="dropcap">Satellite imagery companies spend billions of dollars acquiring pictures of the planet. However, anyone who has worked with that imagery knows that the tools for searching and accessing this imagery are painful to work with. That’s why we are excited to be working with <a href="http://astrodigital.com/products/">Astro Digital</a> to build a fast, easy, imagery browser. To do that, we are relentlessly reducing friction and creating something that feels more like Airbnb or Pinterest than traditional GIS-based imagery browsers.</p> <p>Astro Digital’s strategy is built on <a href="https://medium.com/@astrodigital/browsing-large-sets-of-satellite-imagery-7096db1a807f">making imagery easy to integrate across every industry</a>. Here is how we worked with them to bring usability to image browsing:</p> <h3 id="discover">Discover</h3> <p>A primary goal is to eliminate the steps required to start looking at imagery. Astro Digital discussed the “<a href="https://medium.com/@astrodigital/browsing-large-sets-of-satellite-imagery-7096db1a807f">bounding box problem</a>” – traditional imagery browsers require you to provide a bounding box to start a search. The user either has to outline a polygon or upload a file with the bounding box. If the bounding box is too big or returns too many results these sites often require you to start over.</p> <p>Our solution? Just zoom.</p> <p><a href="http://cl.ly/image/072m1k042a3z/dauria-just-zoom.gif"><img src="http://cl.ly/image/072m1k042a3z/dauria-just-zoom.gif" alt="just-zoom" /></a></p> <p>We will explore options for advanced search that may include boundary uploads, and those interactions will follow the same simple design concepts foundational to a good user experience. As we add those features, we will preserve the ability of users to immediately start interacting in a manner they expect from a modern website.</p> <h3 id="responsive-filter-and-sort">Responsive filter and sort</h3> <p>Once the user has identified their area of interest, the next step is sifting through all the returned images to find the best one for their needs. We’ve made this easier through visual filters and sorting tools that help you quickly get to the best image. Familiar icons allow users to filter and sort without having to learn the interface. We included visual tools like a histogram on our filters to help users intuitively understand the implications of their filtering choices. Sorting assumes “best to worst” to make sorting easy and to surface the best images. All along, we provide thumbnail previews of each image so that you can immediately see what you are getting.</p> <p><a href="http://cl.ly/image/0E3C230f370i/dauria-filter-sort.gif"><img src="http://cl.ly/image/0E3C230f370i/dauria-filter-sort.gif" alt="filter-sort" /></a></p> <h3 id="making-the-interactions-obvious">Making the interactions obvious</h3> <p>Our aim is to create interactions that seem natural without the need for a website tour. Moving around the map shows coverage areas. Selecting an area filters results to that geography. We benefited here by using Landsat imagery that has a consistent bounding area, but we believe we can keep this intuitive even with different shaped scenes. To keep user interaction from breaking, we did a lot of work under the hood to make the site fast and responsive even while loading a lot of data and images. With a streamlined request process, once you’ve found the right image, you can download or purchase in one click.</p> <p><a href="http://cl.ly/image/3Z0B2r1y1Y2m/dauria-obvious-ui.gif"><img src="http://cl.ly/image/3Z0B2r1y1Y2m/dauria-obvious-ui.gif" alt="obvious-ui" /></a></p> <p>We were able to quickly stand up a fast mapping application by building off the <a href="https://www.mapbox.com/">Mapbox</a> stack. Simple, intuitive tools make it easier for more sectors to benefit from satellite imagery. As we help global development organizations and emerging economies to use imagery products, usable tools will be critical. We think existing imagery users will also appreciate having a more enjoyable relationship with their imagery finder.</p> <p><em>Feb 16 2015: We’ve updated this post to reflect the Dauria Geo name change to Astro Digital.</em></p> 2015-01-15T11:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/15/dauria-image-search Development Seed opens office in Lisbon http://www.developmentseed.org/blog/2015/01/15/development-seed-opens-office-in-lisbon <p class="dropcap">Development Seed is opening an office in Lisbon. Our team grows by a continent today in beautiful Portugal where we will continue to build data tools and solve complex development challenges. Establishing an office in Europe puts us closer to our partners in Europe, Middle East, and Africa. It will also allow us to better connect to the talented open data hacker movement in the region.</p> <p><img src="https://farm8.staticflickr.com/7536/16100603687_40714edc2c_b.jpg" alt="" /></p> <p>To bootstrap our European team we are immediately bringing on all our talented friends at Flipside. The Flipside team have been doing fantastic work on projects ranging from opening data on clean energy, building mobile monitoring tools with Text To Change, and tracking forest fires around Portugal. The entire team joins Development Seed today. Olaf Veerman from Flipside will lead the Portugal office and help us to grow the team.</p> <p>If you are in Lisbon, come and celebrate with us tonight at our <a href="http://opendatahappyhour.splashthat.com/">Open Data Happy Hour</a>.</p> 2015-01-15T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/15/development-seed-opens-office-in-lisbon Open Data Happy Hour in Lisbon http://www.developmentseed.org/blog/2015/01/09/open-data-happy-hour <p><a href="http://opendatahappyhour.splashthat.com/"><img src="https://farm8.staticflickr.com/7473/16238390085_f53e289df1_c.jpg" alt="" /></a></p> <p class="dropcap">Next week we’ll be in Lisbon where we’re hosting an <a href="http://opendatahappyhour.splashthat.com/">Open Data Happy Hour</a> with our friends from <a href="http://flipside.org">Flipside</a>. If you’re in the neighborhood and want to talk Open Data, know more about our satellite imagery work, or geek out on sensors, make sure to drop by and have a drink with us.</p> <p>The Happy Hour will be hosted at <a href="http://liberdade229.com">Liberdade 229</a> and start around 19:00. We hope to see you there. <a href="http://opendatahappyhour.splashthat.com/">RSVP here</a>.</p> 2015-01-09T11:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/09/open-data-happy-hour Welcome Joe Flasher http://www.developmentseed.org/blog/2015/01/08/howdy-joe <p class="dropcap"><a href="https://www.developmentseed.org/team/joe-flasher">Joe Flasher</a> is joining Development Seed to help us run faster and to broaden our technology stack. Whether he’s launching Delta II rockets, growing the open data community in Mongolia, or connecting an arcade claw game to a Facebook app, Joe is always looking for better ways to bend technology to solve real problems. He is constantly seeking the solution that is more elegant, more practical, and more human.</p> <p>Joe is the perfect person to help Development Seed grow and explore. He’s going push us to build better products and to make bigger open source contributions across satellite imagery, sensors, drones, and devices.</p> <p>Give Joe a holler on <a href="https://twitter.com/joseph_flasher">Twitter</a> or <a href="https://github.com/jflasher">Github</a>.</p> <p><img src="https://farm8.staticflickr.com/7523/16043833640_bb95464a00_c.jpg" alt="" /></p> 2015-01-08T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/08/howdy-joe 2014 Web Index measures the Internet's openness and impact http://www.developmentseed.org/blog/2014/12/12/launching-web-index <p>The World Wide Web Foundation released their <a href="http://thewebindex.org/">2014 Web Index</a>, an annual accounting of how the internet has changed the economic, political, and social lives of citizens across every continent.</p> <p>This is an ambitious and complicated story. The impact of the internet on the 84 countries in the 2014 report is difficult to isolate from culture, politics, and economics. Ultimately, the report found that open and inclusive internet correlate to equitable societies. The report is rich with data and worth reading <a href="http://thewebindex.org/">in full</a>.</p> <h4 id="visualizing-liquid-data">Visualizing liquid data</h4> <p>Making useful and appropriate comparisons between 84 countries is hard. To create the Web Index, the Web Foundation pulled a range of primary and secondary data into normalized indicators, which together comprise the index. The report, produced online in collaboration with <a href="http://www.weso.es/">WESO</a>, gives a global overview in addition to nuanced subviews on topics such as internet censorship and gender equality.</p> <p>When we first began collaborating with Web Foundation to visualize this information, they were in the process of collecting and interpreting their data. They had identified key themes that they knew would be important: economic equality, neutrality, censorship and surveillance, and gender-based violence, among others.</p> <p>Using these leads, we dived into the preliminary results in search of effective, telling visuals. Quick feedback loops and frequent communication with research specialists at Web Foundation helped us to build better visuals. It also helped Web Foundation better understand their data and how to communicate it.</p> <p><img src="https://c1.staticflickr.com/9/8626/15992468205_b8979f541d_c.jpg" alt="" /></p> <h4 id="comparing-countries">Comparing Countries</h4> <p>We ran up against the challenge of showing 84 countries in interesting ways, while doing justice to the data. Although the data is country-level, not every story lends itself to a map. In our visuals, we use flags, bubbles, and names to indicate countries.</p> <p>In plotting gender-based violence, we ran into the problem of page size. The graph examines countries with stark differences in how they support victims of online gender-based violence, and how frequently they prosecute perpetrators of that violence. Each graphic had to be single page and embeddable. To make this fit, we used a <a href="http://bost.ocks.org/mike/fisheye/">fisheye</a> effect that moves with your cursor.</p> <p><img src="http://i.imgur.com/QM5sM9g.gif" alt="" /></p> <h3 id="open-data-about-the-open-internet">Open Data about the open Internet</h3> <p>The Web Index is fully open. You can download <a href="http://thewebindex.org/downloads/">all of the data here</a>. The Web Foundation and WESO built an open API with data from this year and previous years. The code for <a href="https://github.com/weso/webindex-prototype">their site</a> and <a href="https://github.com/weso/webindex-prototype/tree/master/app/visualisations">our four visualizations</a> are open as well.</p> 2014-12-12T01:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/12/12/launching-web-index A Home for Open Housing Data http://www.developmentseed.org/blog/2014/11/17/housing-in-illinois <p>The housing crisis deeply impacted millions of Americans, and today the <a href="http://www.npr.org/2014/11/18/364131391/firm-accused-of-illegal-practices-that-push-families-into-foreclosure">effects are still being felt</a>. For the many problems facing communities, from wage disparity to affordable housing, there isn’t a single fits-all solution.</p> <p>That’s why we are very excited about the work that <a href="http://www.woodstockinst.org/about">Woodstock Foundation</a> is doing to support fair housing policies. Today they launch a nuanced look at housing and income disparity in Illinois in the form of a new <a href="http://www.woodstockinst.org/content/woodstock-institute-interactive-map">map-based open data website</a>.</p> <p>The site brings together 74 datasets on the well-being of local communities. It is a good roadmap for anyone working in housing justice in Illinois. Community organizations can explore <a href="http://woodstockinst.github.io/woodstock-maps/#mortgage/census_tract/avg_loan_value/2012">the average amount of mortgage debt people take on</a> and the rate of foreclosure filings in <a href="http://woodstockinst.github.io/woodstock-maps/#chicago_foreclosure/census_tract/foreclosure_filings_per_1000/2013">the Chicago six-county region</a> and <a href="http://woodstockinst.github.io/woodstock-maps/#rest_of_illinois_foreclosure/census_tract/nos_per_1000/2013">elsewhere in Illinois</a> to inform their decisions on where to focus their work. Some <a href="http://woodstockinst.github.io/woodstock-maps/#housing/census_tract/long_term_vacancy/2008">data sets go back to 2008</a>.</p> <p><img src="https://c2.staticflickr.com/6/5606/15813741921_6b51750a2a_b.jpg" alt="high-income" /><br /> <em>Where the highest income census tracts are in the Chicago six-county region</em></p> <h3 id="serving-complex-data-through-static-json">Serving complex data through static JSON</h3> <p>Housing data is complex, and Woodstock has gathered some amazingly granular statistics about housing data in Illinois. Splitting this data into a format that we could serve over the web proved a difficult challenge. Woodstock is also a small nonprofit, and we wanted to ease as much as possible the burden of maintaining a complicated website.</p> <p>So we wrote a python library to break up their spreadsheets into <a href="http://en.wikipedia.org/wiki/JSON">JSON</a>. Every time you switch to a new facet of housing data, every time you view a different year or category of that data, your browser incrementally downloads a new JSON file. Although those files in aggregate would take ages to load, individually they are manageable. Those scripts, along with the rest of the site, are <a href="https://github.com/woodstockinst/woodstock-maps">open-source</a>.</p> <p>To further reduce load times, we use the <a href="https://github.com/mbostock/topojson">topojson</a> spec to reduce the size of geographic boundary data. This allows us to separate geographic data from numerical data, so you only download those complex census tract boundaries once. The code that runs in your browser than re-connects those boundaries to the housing or mortgage data you select on the fly.</p> <p>Using vector-based geographical boundaries has other benefits. It allowed us to use a mouse click on an overlaying geographic boundary and a <a href="http://stackoverflow.com/questions/217578/point-in-polygon-aka-hit-test">point-in-polygon</a> test to find, for example, which congressional representative is responsible for which census tract.</p> <p><img src="https://c2.staticflickr.com/6/5611/15195642794_9075b6af59_b.jpg" alt="collision-detection" /><br /> <em>Detecting congressional boundaries</em></p> <h3 id="census-tracts-over-multiple-years">Census tracts over multiple years</h3> <p>Census tracts change a lot over census years, and this can be a problem when mapping a multi-year data set that covers more than a single census geography file. Attempting to compare the data between those years that use different tract definitions can be tricky. Tract ID, or FIPS codes can either refer to a different neighborhood or disappear entirely.</p> <p>The Census releases <a href="https://www.census.gov/geo/maps-data/data/relationship.html">relationship files</a> that show where these changes and additions occur. Using this, we created a tool that overlays 2000 and 2010 census tracts, and <a href="http://www.developmentseed.org/blog/2014/04/25/making-sense-of-census-changes/">shows differences between the two years</a>.</p> <p>Currently the project covers only Illinois, but the code is available on <a href="https://github.com/developmentseed/census-tracts/tree/gh-pages">GitHub</a> and ready for you to fork and contribute your own state.</p> 2014-11-17T12:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/17/housing-in-illinois Join us at EcoHack this weekend http://www.developmentseed.org/blog/2014/11/13/hosting-ecohack <p><img src="https://farm9.staticflickr.com/8625/15781769355_8877307f0c_b.jpg" alt="" /><br /> <em>Toxic algal bloom in Lake Erie</em></p> <p>To address climate change and promote environmental justice, we need better tools to understand our changing planet. That’s why we are delighted to help host the <a href="https://nvite.com/EcoHack/b41">DC EcoHack</a> with <a href="http://www.wri.org/">WRI</a> at the Mapbox Garage.</p> <p>EcoHack is an event to bring together a diverse community of scientists, hackers, designers and others who want to tell stories and create tools to protect our environment. The event is open to people of all skill levels. As long you’re interested in using technology to improve and better understand our natural environment, we’d love to see you there.</p> <p>If you’re in DC, <a href="https://nvite.com/EcoHack/b41">register</a> and come join us at the <a href="https://www.mapbox.com/">MapBox</a> Garage in DC. Our friends at Mapbox are also hosting the <a href="https://www.mapbox.com/blog/mapbox-ecohack/">San Francisco EcoHack</a> and EcoHacks will also take place in Sydney, Cambridge, New York, and Madrid.</p> <p><a href="http://www.developmentseed.org/team/drew-bollinger/">Drew</a> and <a href="http://www.developmentseed.org/team/marc-farra/">Marc</a> will be using some of the time to work on <a href="http://www.developmentseed.org/blog/2014/08/29/landsat-util/">landsat-util</a>, an open source tool that makes it easier to work with open satellite imagery. We are also keen to help on projects using <a href="http://www.developmentseed.org/blog/2014/10/28/climatescope/">green energy investment data</a> and <a href="http://www.developmentseed.org/blog/2014/09/15/flood-mapping/">tracking natural disasters</a>.</p> <p>Hope to see you there!</p> 2014-11-13T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/13/hosting-ecohack International Conference of Crisis Mappers storms NYC http://www.developmentseed.org/blog/2014/11/06/iccm-storms-nyc <p>Today, <a href="http://www.developmentseed.org/team/marc-farra/">Marc</a> and I are heading up to New York to attend the <a href="http://crisismappers.net/">International Conference of Crisis Mappers</a>. Open crisis mapping is growing up. We are seeing greater demand for maps and data for crisis response and preparation. ICCM 2014 will be an important place to discuss how we can grow to handle this need, how we better generate real collaboration from data, and how we build infrastructure that is usable and inclusive.</p> <p><img src="https://farm4.staticflickr.com/3944/15726853685_0edd86b2c7_c.jpg" alt="" /><br /> <em>Satellite imagery over Monrovia, an area that has seen a significant burden of Ebola cases.</em></p> <p>We’re looking forward to talking about OpenStreetMap, satellites, and open data; topics that are critical in the midst of the response to the Ebola outbreak in West Africa. Reach out to <a href="http://twitter.com/nas_smith">@nas_smith</a> or <a href="http://twitter.com/kamicut">@kamicut</a> on Twitter if you want to chat.</p> 2014-11-06T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/06/iccm-storms-nyc Getting to vote http://www.developmentseed.org/blog/2014/11/04/us-election <script type="text/javascript" src="https://s3.amazonaws.com/vip-voter-information-tool/app.js"></script> <div id="_vit"></div> <script type="text/javascript">vit.load({officialOnly:false});</script> <p><em>Working embed. US voters, find your polling location</em></p> <p>In the US, and around the world, it can be confusing figuring out where you should vote and which races you are eligible to vote on. In the US, going to the wrong polling station is a hassle. This hassle can be prohibitive, particularly if you are disabled or rely on public transportation. In countries coming out of conflict, going to the polling station can be a brave and risky act. You’d better be at the right place when you get there.</p> <p>Open data is helping to get voters the information they need to participate. By opening up data about voting locations and process, States can involve private actors, such as <a href="http://www.pewtrusts.org/en">The Pew Trusts</a> and Google, as partners providing accurate information about where to vote. The <a href="https://www.votinginfoproject.org/about/">VIP (Voter Information Project)</a> is an embeddable tool, based on an <a href="https://github.com/votinginfoproject/vip-embeddable-tool">open source</a> stack, that relies on an <a href="https://developers.google.com/civic-information/">open API</a>. This allows other groups to repackage and distribute this information to their audiences.</p> <p>By opening up the data, States are no longer solely responsible for getting voters to the right polling place. Voters should be tripping over this information in every Google search, Foursquare check-in, community message board, and favorite blogs.</p> <p><img src="https://farm4.staticflickr.com/3030/3002704954_94a4df58ee_o.jpg" alt="voting demonstration" /><br /> <em>“Usability and Voting” by <a href="https://www.flickr.com/photos/ericgundersen/3002704954/in/photolist-5zkUTU-5zkUhL-5zkRms-5zkTFS-5zgAm6-5zgzRV-5zgAST-5zkDz5-aqq3t9-cgCsHQ-5zgfND-5zfZMc">ericgundersen</a></em></p> 2014-11-04T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/04/us-election Hey there, designers. http://www.developmentseed.org/blog/2014/11/02/attention-designer <p>At Development Seed, design is not about pushing pixels or passing a perfect mockup to the next person. It is about truly understanding — and sometimes defining — a problem, working out a systemic solution with visual and interactive components. You will be brainstorming solutions with our strategists and turning them into sketches, websites, and data visualizations with fellow developers.</p> <p>We build new ways to help people make decisions — impacting policies and creating transparency on all levels. We are hiring a designer who is a doer and a thinker, eager to join our mission.</p> <h3 id="you-are">You are:</h3> <ul> <li>an artist; you have a favorite medium to express your ideas, be it ink, paint, vector or gif</li> <li>excited about the web, particularly how the representation of information on screens can inform people’s decisions</li> <li>eager to work with data and the patterns it leads to</li> <li>curious and hungry to learn new subjects and skills</li> </ul> <h3 id="you-know-how-to">You know how to:</h3> <ul> <li>ask good questions and get to the heart of a problem</li> <li>illustrate abstract concepts and workflows in visual forms</li> <li>use the right font and color at the right time, knowing that aesthetics is derived from your communication goal, the information, and the medium</li> </ul> <h3 id="experiences-with-any-of-the-following-will-be-a-plus">Experiences with any of the following will be a plus:</h3> <ul> <li>Web maps (such as a map made with MapBox Studio)</li> <li>Responsive web frameworks</li> <li>Git</li> <li>D3.js for visualization (bar chart counts)</li> <li>Static site generation (Jekyll, Flask, etc)</li> <li>SASS</li> </ul> <p>But don’t let any of that scare you. If your design chops are good, we will work with you to tech up on everything you need to know.</p> <h3 id="to-apply">To apply:</h3> <p>Please send your portfolio site, and links to three projects, to jobs@developmentseed.org with “Designer” in your subject.</p> 2014-11-02T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/02/attention-designer Astro Digital completing new design specs http://www.developmentseed.org/blog/2014/10/30/dauria-launch <p><img src="https://farm4.staticflickr.com/3949/15481548030_aaf1e9b01d_o.gif" alt="brazil deforestation" /></p> <p>We’re collaborating with Astro Digital now as they build out integration and visualization tools – from antenna to API, that makes image acquisition, analysis, delivery, and integration easier for both NGOs and enterprise. Astro Digital is building an API to empower developers to access fresh and historical imagery, compute needed data layers on the cloud, and harvest data in ready-to-use format. Their cloud platform can do heavy analysis and feed data directly into applications. By directly integrating with the <a href="https://www.mapbox.com/developers/api/">Mapbox API</a>, we can quickly deploy sophisticated and beautiful applications from agriculture to disaster response using the platforms that developers are already building on.</p> <p><img src="https://farm9.staticflickr.com/8591/16726324832_e6c248cc9e_b.jpg" alt="astro satellites" /></p> <p>We’d love to see more satellite providers compete on ease of integrating their data. We’ll be helping Astro Digital to review their API to make it developer friendly and we will build open source tools on top Astro Digital’s API. These tools will serve as open templates for integrating Astro Digital with tools like Mapbox to quickly build powerful, data-rich sites. This is really positive move for the industry and for users.</p> 2014-10-30T12:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/30/dauria-launch Marhaba Marc Farra http://www.developmentseed.org/blog/2014/10/28/marhaba-marc <p><a href="http://www.developmentseed.org/team/marc-farra/">Marc Farra</a> has joined the Development Seed team. Marc loves to experiment with image processing, arduino sensors, and data infrastructure. He is going to help us explore new ways to collect and process data.</p> <p>I first ran into Marc in Beirut. At the time he was running Lamba Labs, a hacker space in Beirut that was sowing the seeds of Maker culture and open data advocacy in Lebanon. A year later, he took the Afghanistan polling station locations we <a href="https://github.com/developmentseed/aodp-data">posted on Github</a> and started to build <a href="http://www.developmentseed.org/blog/2014/04/04/votehere-af/">a mobile app for Afghans to locate their nearest polling station</a>.</p> <p><img src="https://cloud.githubusercontent.com/assets/5984548/4445705/9555418c-47fb-11e4-81b5-a12a3573c74b.jpg" alt="" /></p> <p>Say “Hello”, “Salut”, or “Marhaba” to Marc on <a href="https://twitter.com/kamicut">Twitter</a> and <a href="https://github.com/kamicut">Github</a>.</p> 2014-10-28T15:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/28/marhaba-marc Getting Green into Green Energy http://www.developmentseed.org/blog/2014/10/28/climatescope <p>Reversing climate change means investing in green energy, and as the sustainable sector grows, ensuring it grows in both developing and developed countries. Today the <a href="http://www.fomin.org/">Fondo Multilateral de Inversiones</a> and <a href="http://about.bnef.com">Bloomberg New Energy Finance</a> are launching a vastly expanded <a href="http://global-climatescope.org/">Climatescope</a>, to provide open data about green energy investment in 55 countries. The data provided by Climatescope creates an information-rich environment for green energy investors. It also provides valuable data on clean energy policies for activists and policymakers.</p> <p><img src="https://farm4.staticflickr.com/3946/15025140024_eb0c81e77f_o.png" alt="climatescope" /></p> <h3 id="opening-climate-investment-data">Opening Climate Investment Data</h3> <p>We worked on the Climatescope website with <a href="http://flipside.org/">Flipside</a>, a smart, new open source technology shop based in Lisbon.</p> <p>The site takes a very thoughtful approach to opening information. All the data powering the Climatescope site is available through an open API, which you can easily integrate into your own applications. The <a href="http://global-climatescope.org/en/download/">full dataset</a> is also available for analysis. On almost every page lives a download button that provides a CSV file containing whatever you happen to be viewing.</p> <p>Most importantly, FOMIN got the licensing right. The data is licensed <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY</a>. It can be used (with attribution) by anyone, even for commercial purposes. This is critical when you want data to encourage commercial activity. Moreover the <em><a href="https://github.com/climatescope/climatescope.github.io">website itself</a></em> is also open and is licensed GPL 3.0. The entire site can be forked by other open source projects.</p> <h3 id="dynamic-static-websites">Dynamic Static Websites</h3> <p>Like many of the sites we (and Flipside) build these days, Climatescope is a fully interactive site without a database or a heavy CMS. Climatescope users can manipulate, interrogate, and download the data on any device and in low bandwidth requirements. The site uses Jekyll, Angular, and D3 (among <a href="http://demo.flipside.org/climatescope/humans.txt">other tools</a>) and is hosted on Github. <a href="http://www.developmentseed.org/blog/2012/07/27/build-cms-free-websites/">Read more on our approach to CMS-free websites</a>.</p> <h3 id="customized-weighting">Customized weighting</h3> <p>People have different priorities when evaluating the environment for clean energy. The site is designed for a range of users, from activists to journalists, politicians, environmentalists, and the curious. FOMIN is committed to giving Climatescope users full control over how much weight each metric carries. To accomodate this, we built simple, intuitive sliders. Movement in one slider spreads the difference evenly across the other three factors. You can lock any slider to make it easier to hit an exact breakdown.</p> <p><img src="http://i.imgur.com/3Soa6sf.gif" alt="" /></p> <h3 id="hacking-for-the-planet">Hacking for the Planet</h3> <p>Have some data or coding skills? Care about the planet? Consider joining an <a href="http://ecohack.org/">EcoHack</a> near you on Nov 15-16. We are hosting <a href="https://nvite.com/EcoHack/b41">the DC EcoHack</a> with <a href="http://www.wri.org/">WRI</a>. EcoHacks are also happening in Sydney, Cambridge, New York, Madrid, and San Francisco.</p> 2014-10-28T13:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/28/climatescope