Node.js - Development Seed http://www.developmentseed.org/blog/nodejs The latest developments in evented, server-side javascript. en Our new website - still Jekyll http://www.developmentseed.org/blog/2015/04/20/building-with-jekyll <p class="dropcap">We recently redesigned our website from the ground up. While the look and organization changed, the underlying technologies have not. We are still using Jekyll to power the site, just as <a href="https://developmentseed.org/blog/2011/09/09/jekyll-github-pages/">we’ve done since 2011</a>. Jekyll has come a long way and we’re taking full advantage of some of its new features.</p> <h3 id="collections">Collections</h3> <p>Collections are the <a href="http://jekyllrb.com/">Jekyll</a> way to separate content types. They allow us to create new content types and to treat them differently. We are using collections for our team and project pages. Collection items have the attributes of “posts” in Jekyll with some subtle differences, like being ordered alphabetically in ascending order or not needing to have the date on the name. With collections we can achieve a greater content separation thus making the whole site more organized.</p> <h3 id="list-views">List views</h3> <p>Jekyll still has some challenges when it comes to paginate collections or categories. We solved this problem by rendering our list views (projects and blog posts) client side.</p> <p>Jekyll outputs a JSON file that we use to render the content with ejs templates. This allows us to have greater control over content output and prevents your browser from blowing up when trying to render our 650+ blog posts at once.</p> <h3 id="wrangling-the-json-api">Wrangling the JSON API</h3> <p>Jekyll’s JSON output can lead to a familiar comma problem. When Jekyll is generating a JSON file is common to see code like this:</p> <pre><code>{% raw %}[ {% for post in site.projects %} { "title": "{{ post.title }}" } {% unless forloop.last %},{% endunless %} {% endfor %} {% endraw %}] </code></pre> <p>This will work fine when every element of the array is being printed, but if you start introducing conditions the <code>forloop.last</code> can’t be used to control the comma anymore.</p> <p>For example we’re only showing posts that are not hidden. Image we’ve only three posts of which the third one is hidden. The resulting JSON would be:</p> <pre><code>[ { "title": "Project one" } , { "title": "Project two" } , ] </code></pre> <p>See the trailing comma? That little thing would break the JSON. It’s there because our last visible project is <code>Project two</code> but that is not the last element in the array so <code>forloop.last</code> doesn’t kick in and the comma is printed anyways.</p> <p>The workaround for this situation is easy. You just have to print the comma before the element except for the first time:</p> <pre><code>{% raw %}[ {% assign comma = false %} {% for post in site.projects %} {% unless post.hidden == true %} {% if comma %},{% endif %} {% assign comma = true %} { "title": "{{ post.title | escape_once }}" } {% endunless %} {% endfor %} {% endraw %}] </code></pre> 2015-04-20T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/20/building-with-jekyll A Modern Imagery Processing Pipeline http://www.developmentseed.org/blog/2015/04/17/ad-platform-launch <p>Satellite data is a tremendously powerful resource for governments and development organizations. We built a <a href="https://developmentseed.org/blog/2015/03/10/releasing-landsat-v05/">suite</a> <a href="https://developmentseed.org/blog/2015/01/22/announcing-libra/">of</a> <a href="https://github.com/developmentseed/landsat-api">tools</a> to make open Landat data more accessible and useable. These allow our development partners to process imagery and perform analysis quicker, and that can make all the difference in rapidly evolving situations.</p> <p>Often our partners need commercial imagery with greater resolution and refresh times than what Landsat 8 offers. We have great partnerships with commercial imagery providers, to offer all sorts of imagery. Too often receiving and processing commercial imagery is a huge pain point that slows us down and makes it harder to make use of the data. As developers we know it could be better. </p> <p><a href="https://medium.com/@astrodigital/imagery-processing-pipeline-launches-3f92c30932ec">Astro Digital</a> gave us an opportunity to rebuild this workflow from the ground up. We’ve worked closely with their team to build a satellite imagery pipeline for developers and end users. We just launched a browsing and publishing platform with Astro Digital to allow anyone to discover, process, and share satellite imagery in an incredibly quick and intuitive manner. A process that previously could take days has now been cut down to minutes.</p> <h3 id="api-first">API first</h3> <p>We built an end-to-end data processing pipeline that feeds a powerful data API that unlocks possibilities for others. We broke down the fundamental goals of the Platform and built API calls around each. Those goals were to search, process, and publish. </p> <p>We built and exposed a perfomant Elasticsearch-powered endpoint, based on our previous <a href="https://github.com/developmentseed/landsat-api">landsat-api</a> work, that will allow for complex queries to find exactly the data that is needed. </p> <p>{% highlight bash %}<br /> $ curl https://api.astrodigital.com/v1/search?search=cloudCoverageFull:[0+TO+20]<br /> {% endhighlight %}</p> <p>But how to process the imagery? We extended our existing open source <a href="https://github.com/developmentseed/landsat-util">landsat-util</a> tool to handle varying band combinations and the API offers several including true color, vegation health false color and urban false color.</p> <p>{% highlight bash %}<br /> $ curl https://api.astrodigital.com/v1/methods<br /> {% endhighlight %}</p> <p>And finally, there is a simple request that can be made to process the imagery and receive a tiled map URL. This URL can be used with tools like <a href="https://mapbox.com">Mapbox</a> or <a href="http://leafletjs.com">Leaflet</a> to build upon the processed imagery in any way. Full documentation, including interactive samples, can be found at <a href="http://docs.astrodigital.com">docs.astrodigital.com</a>.</p> <p>{% highlight bash %}<br /> $ curl -X POST –data “sceneID=LC80430332014262LGN00&amp;process=urbanFalse&amp;satellite=l8” https://api.astrodigital.com/v1/publish<br /> {% endhighlight %}</p> <p>{% highlight json %}<br /> {“status”:”Image is being processed.”}<br /> {% endhighlight %}</p> <h3 id="frictionless-publishing">Frictionless publishing</h3> <p>We are using this data pipeline to power an extremely easy and visual imagery browser and publishing tool. We started with <a href="https://developmentseed.org/blog/2015/01/22/announcing-libra/">Libra</a> as a base and modified it to meet the Astro Digital specific workflow. Libra was already designed to be quick and intuitive. We added a simple publish workflow that will process and publish images and email a link to the tiled map after processing has completed. For images processed within this visual workflow, the email contains a link to an embeddable map that can be used anywhere across the web. </p> <p><img src="https://cloud.githubusercontent.com/assets/848934/7208412/996efb4a-e50e-11e4-8214-4d63c477944e.gif" alt="publish" /></p> <p>Working with Astro Digital, we built a modern publishing pipeline that we hope will push the entire industry to build more usable tools. This is good for the industry and good for users, particularly the small governments and development organizations that are the next wave of power satellite data users.</p> 2015-04-17T15:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/17/ad-platform-launch An OpenStreetMap for Government http://www.developmentseed.org/blog/2015/04/15/openstreetmap-for-government <p>The software that powers OpenStreetMap (OSM) is a fantastic open source tool for governments to manage road data. We are working with the World Bank and the Philippine government to deploy OSM software to create a collaborative tool for national and municipal authorities to manage road data.</p> <p>OSM-as-a-platform gives governments a powerful open source option for enterprise management of road data. The OpenStreetMap community benefits with more investors in the OSM ecosystem and more OSM-ready data.</p> <h3 id="osm-and-government">OSM and Government</h3> <p>OpenStreetMap is a powerful tool for collaborative mapping of critical infrastructure. Over two million OSM users have contributed 280 million road, rail, and waterway segments.</p> <p>Governments are ideal OSM users. Governments build and maintain roads and manage public transit. Governments have powerful incentives to provide road data in a manner most useful to citizens. However legal, policy, and cultural considerations prevent governments from engaging in OSM.</p> <ul> <li>Government may have a requirement for authoritative control of data for 911 purposes.</li> <li>OSM’s share-alike requirements may run foul of government publishing obligations.</li> <li>Agency IT Departments may be hesitant to authorize external services for maintaining critical infrastructure data.</li> <li>Governments may have additional data or workflow requirements that aren’t supported by OSM.</li> </ul> <h3 id="openroads---osm-for-the-philippines">OpenRoads - OSM for the Philippines</h3> <p>The National Government in the Philippines needs the ability to manage and collaboratively edit road data in much the same way as OSM, but with their own unique set of data and users. Different portions of the road network are managed by authorities at the national, provincial, municipal, or barangay. These authorities also need tools to manage data on road improvement projects, from evaluating proposed resurfacing projects to tracking the proper completion of a funded road expansion.</p> <p>The OSM software provides fantastic tools for a whole-of-government approach to road management. Working closely with the World Bank, we are building the OpenRoads Network Editor, an OSM-based system for road management across Philippines’ various road authorities. We built this using OSM-as-a-platform and utilizing tools like <a href="http://ideditor.com/">iD</a> and <a href="https://github.com/osmlab/to-fix">to-fix</a>.</p> <p><img src="https://cloud.githubusercontent.com/assets/7108211/7166919/1d3c0972-e37f-11e4-9b44-1be85d5569dd.png" alt="" /></p> <p>This tool will allow the Government to create a full map and inventory of the countries 200,000 KM of roads. Ultimately it will also provide the analytical tools to allow the Philippines to make better decisions on infrastructure investments with real-time data.</p> <p>The World Bank recognizes the value of government managing data with collaborative open source tools. This week the World Bank is introducing this platform at a summit of the Philippine’s 1490 mayors. Today we are at the World Bank for <a href="http://www.worldbank.org/en/events/2015/03/31/big-data-for-a-more-resilient-future"><em>Big Data for a More Resilient Future</em></a> to present OpenRoads and how OSM-as-a-platform can benefit other governments.</p> <h3 id="open-tools-for-open-government">Open Tools for Open Government</h3> <p>OpenRoads is part of Development Seed’s ongoing efforts to create <a href="//developmentseed.org/set/open-tools-for-open-data/">open tools to support open government</a>.</p> <p>The OSM software ecosystem provides governments with a powerful open source alternative for managing road data. The OpenStreetMap community also benefits. These deployments create additional investors in the OSM software and more OSM-ready data.</p> 2015-04-15T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/15/openstreetmap-for-government Tracking Metadata in Real-time http://www.developmentseed.org/blog/2015/04/14/tracking-osm-metadata <p>This week we’re releasing more tools to track OpenStreetMap metadata. Together with the American Red Cross, we’re launching <a href="https://github.com/osmlab/osm-meta-api">OSM Metadata API</a>, a tool to help enable analysis of OSM’s rich metadata at the user and comment level. By using hashtags in changeset comments such as <a href="http://www.missingmaps.org/">#MissingMaps</a>, API access can enable groups like the Red Cross to gain feedback from OSM deployments. </p> <p>We’re building off our <a href="https://developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata/">osm-meta-util work we released several weeks ago</a> to store and index OSM metadata. We built the API on <a href="https://www.elastic.co/">Elasticsearch</a> and <a href="https://nodejs.org/">Node.js</a>. Using Elasticsearch, we index and store metadata information such as changeset_id, user, created_at, bounding box, comment. You can filter these logs by hashtag and keyword as they come in real-time, as well as build a database of historical metadata logs. </p> <p>We’re rolling out a sample API today with the code. You can access the endpoint here: <a href="https://api.developmentseed.org/osm">https://api.developmentseed.org/osm</a>. </p> <p>You can use the full power of Apache Lucene Search to browse the data. For example, to browse for the #missingmaps hashtag between two dates:</p> <pre><code>https://api.developmentseed.org/osm?search=created_at:[2015-04-08T00:00:00Z+TO+2015-04-09T00:00:00Z]+AND+comment:"missingmaps" </code></pre> <p>In the past week, #missingmaps events have helped make 69,865 edits to OpenStreetMap in places like Haiti, Iraq, Zimbabwe, and Tanzania. </p> <p><img src="/assets/graphics/content/osm-meta-api-release-cover.png" alt="" /></p> <p>Check out the <a href="https://github.com/osmlab/osm-meta-api/wiki/API-Guide">API guide</a> for full endpoint documentation. </p> <p>All code is available via a <a href="https://github.com/osmlab/osm-meta-api">GitHub repository on OSM Lab</a>. To get started quickly, you can <a href="https://github.com/osmlab/osm-meta-api#osm-meta-api">deploy the API as a Heroku app</a>. Try it out and contribute back to the OSM community.</p> 2015-04-14T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/14/tracking-osm-metadata Howdy, Anand Thakker! http://www.developmentseed.org/blog/2015/04/02/howdy-anand <p><img src="https://farm9.staticflickr.com/8751/16827357019_7b62379f1f_b.jpg" alt="Anand looking good" /></p> <p><a href="https://developmentseed.org/team/anand-thaaker/">Anand Thakker</a> likes the messy stuff. Whether he’s using historic satellite imagery to measure electrification or finding better ways to teach high school math, Anand embraces hard problems. He brings creativity, energy, and thoughtfulness to every project.</p> <p>Anand is going to help us to build powerful tools that make a real difference. Anand has always been engineering. He built a search engine with his high school friends. At a startup right out of college, he worked on tools for analyzing and debugging XML-based web services. Then, for seven years, he devoted his energy to teaching high school math and computer science in Baltimore. There too, Anand engineered curriculum and teaching methods for improving students’ problem solving skills.</p> <p>Anand studied math and computer science at Carnegie Mellon University and has a masters in education from Harvard University. Give him a shout on <a href="https://github.com/anandthakker">github</a> or <a href="https://twitter.com/anandthakker">twitter</a>.</p> 2015-04-02T16:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/02/howdy-anand Welcome Emily Bremner! http://www.developmentseed.org/blog/2015/04/02/welcome-emily <p><img src="https://farm9.staticflickr.com/8728/16825813978_9dcf8abc28_b.jpg" alt="Emily and the tree" /></p> <p>International development is hard. <a href="https://developmentseed.org/team/emily-bremner/">Emily Bremner</a> first saw this at a school in Kenya for Somali refugees. She was 16. Over more than a decade since, Emily has pushed to improve peoples’ lives in places from El Salvador to Afghanistan. She’s been teargassed in Djibouti and battled bureaucracy back in DC.</p> <p>Emily has worked on all sides of international development and has seen its successes and its failures. Emily is going to work on our business strategy and operations. She is going to help us to better solve hard social problems and to make sure our work is impactful.</p> <p>We had the opportunity to work with Emily when she was at <a href="http://democracyinternational.com/">Democracy International</a> to open up election data in <a href="https://developmentseed.org/blog/2014/06/23/open-data-lebanon/">Lebanon</a> and <a href="https://developmentseed.org/blog/2014/05/22/tunisia-open-election-data/">Tunisia</a>. She understood then the opportunities and the limits of data and technology to make meaningful change. We look forward in working with her to expand those limits.</p> <p>Say Bienvenue or Hola to Emily <a href="https://twitter.com/emilybremner23">on twitter</a>.</p> 2015-04-02T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/04/02/welcome-emily Optimizing Landsat-util http://www.developmentseed.org/blog/2015/03/28/twice-as-fast <p>Two weeks ago we launched a <a href="https://developmentseed.org/blog/2015/03/10/releasing-landsat-v05/">new version of Landsat-util, v0.5</a>, our utility for searching and processing Landsat satellite imagery. This version is now faster than it was before. To do this we rewrote most of the internals to use flexible python frameworks.</p> <p>Below is a deep dive into how we’ve improved Landsat-util to make it faster and easier to use. </p> <h2 id="dependency-hell">Dependency Hell</h2> <p>Landsat-util downloads Landsat files, pulls out the individual bands representing wavelengths of light, corrects contrast, warps them and combines them to make a colored image you can add on a web map.</p> <p>Initially, landsat-util was written as a command line wrapper to <a href="https://www.mapbox.com/blog/processing-landsat-8/">existing pipelines</a>:</p> <ol> <li>Scale bands with <code>gdal-translate</code></li> <li>Warp with <code>gdal-warp</code> to the correct projection</li> <li>Combine bands with <code>ImageMagick</code></li> <li>Contrast correct with <code>OpenCV</code></li> <li>Pansharpen with <code>orfeoToolbox</code></li> <li>Gamma correct with <code>ImageMagick</code></li> <li>Add geographic information back with <code>gdal_edit</code></li> </ol> <p>We needed a lot of dependencies to process an image, and they’re not particularly optimized for scripting. ImageMagick, GDAL, orfeoToolbox and openCV are monolithic frameworks that don’t allow for cherry-picking functions. Installing all these frameworks can be a <a href="https://github.com/developmentseed/landsat-util/issues?utf8=%E2%9C%93&amp;q=install">painful experience</a>.</p> <p>This toolchain combination is also tedious because it creates inherent bottlenecks. To communicate between all tools, we need to write temporary files to disk and read them back in at every step.</p> <h2 id="enter-rasterio">Enter rasterio</h2> <p><a href="https://github.com/mapbox/rasterio">Rasterio</a> is a great python library written by <a href="https://twitter.com/sgillies">Sean Gillies</a> at Mapbox to work with raster data. It wraps around gdal and abstracts the band data as <a href="http://www.numpy.org/">Numpy</a> arrays. </p> <p>By standardizing the input, rasterio allows us to minimize our dependencies and use fast, in-memory, scientific libraries like <a href="http://scikit-image.org/">scikit-image</a>. </p> <p>Here’s the guts of our new process:</p> <h3 id="read-in-bands-with-rasterio">Read in bands with rasterio</h3> <p>{% highlight python %}<br /> with rasterio.drivers():<br /> with rasterio.open(‘LC82040522013123LGN01_B4.TIF’) as band4:<br /> with rasterio.open(‘LC82040522013123LGN01_B3.TIF’) as band3:<br /> with rasterio.open(‘LC82040522013123LGN01_B2.TIF’) as band2:<br /> with rasterio.open(‘LC82040522013123LGN01_B8.TIF’) as band8:<br /> band4_s = band4.read_band(1)<br /> band3_s = band3.read_band(1)<br /> band2_s = band2.read_band(1)<br /> band8_s = band8.read_band(1)<br /> {% endhighlight %}</p> <h3 id="scale-bands-with-scikit">Scale bands with <code>scikit</code></h3> <p>{% highlight python %}<br /> from skimage import transform as sktransform</p> <p>band4_s = sktransform.rescale(band4_s, 2)<br /> band3_s = sktransform.rescale(band3_s, 2)<br /> band2_s = sktransform.rescale(band2_s, 2)<br /> {% endhighlight %}</p> <h3 id="warp-with-rasterio">Warp with <code>rasterio</code></h3> <p>{% highlight python %}<br /> for color, band in zip([r,g,b,b8], [band4_s, band3_s, band2_s, band8_s]):<br /> reproject(band, color, <br /> src_transform = src.transform,<br /> src_crs = src.crs,<br /> dst_transform = dst_transform, <br /> dst_crs = dst_crs,<br /> resampling = RESAMPLING.nearest)<br /> {% endhighlight %}</p> <h3 id="pansharpen-using-numpy-operations">Pansharpen using <code>numpy</code> operations</h3> <p>{% highlight python %}<br /> m = r + b + g<br /> pan = 1/m * b8<br /> r = r * pan<br /> b = b * pan<br /> g = g * pan<br /> {% endhighlight %}</p> <h3 id="contrast-correct-and-gamma-correct-using-scikit-image">Contrast-correct and gamma correct using <code>scikit-image</code></h3> <p>{% highlight python %}<br /> # using CLAHE<br /> from skimage.exposure import equalize_adapthist<br /> for band in [r,g,b]:<br /> band = equalize_adapthist(band, clip_limit=0.02)<br /> {% endhighlight %}</p> <h3 id="write-to-disk-using-rasterio">Write to disk using <code>rasterio</code></h3> <p>{% highlight python %}<br /> with rasterio.open(<br /> tiffname,’w’, driver=’GTiff’,<br /> width=dst_shape[1],height=dst_shape[0],<br /> count=3,dtype=numpy.uint8,<br /> nodata=0,<br /> transform=dst_transform,<br /> photometric=’RGB’,<br /> crs=dst_crs) as dst:<br /> for k, arr in [(1, r), (2, g), (3, b)]:<br /> dst.write_band(k, arr)<br /> {% endhighlight %}</p> <p>The toolchain consists of only python libraries, and no other dependencies. Rasterio inherently supports GeoTiff so we don’t lose geo-information along the way.</p> <p>Landsat-util is open source, and we encourage developers to improve on our process. Fork <a href="https://github.com/developmentseed/landsat-util">our repo</a>!</p> <h2 id="how-fast">How fast?</h2> <p>By using <code>rasterio</code>, <code>numpy</code> and <code>scikit</code>, we eliminate the disk bottleneck, and we regain transparent control over the pipeline.</p> <p>We ran tests on a couple of AWS machines. Each test was conducted 5 times and the resulting times were averaged. </p> <ul> <li>R3.large: 2-core 2.5 GHz Intel Xeon (Ivy Bridge) with 15GB RAM and 32GB SSD.</li> <li>C4.2xlarge: 8-core 2.9GHz Intel Xeon (Haswell) with 15GB RAM and 32GB SSD. </li> </ul> <h3 id="results">Results</h3> <div class="table-responsive"> <table> <thead> <tr> <th>instance type</th> <th>process type</th> <th>old-landsat-util</th> <th>new-landsat-util</th> <th>speedup</th> </tr> </thead> <tbody> <tr> <td>R3.large</td> <td>non-pansharpened</td> <td>252.7s</td> <td>122.05s</td> <td><strong>2x</strong></td> </tr> <tr> <td>R3.large</td> <td>pansharpened</td> <td>846.9s</td> <td>349.17s</td> <td><strong>2.4x</strong></td> </tr> <tr> <td>C4.2xlarge</td> <td>non-pansharpened</td> <td>216.6s</td> <td>106.67s</td> <td><strong>2x</strong></td> </tr> <tr> <td>C4.2xlarge</td> <td>pansharpened</td> <td>438s</td> <td>290s</td> <td><strong>1.5x</strong></td> </tr> </tbody> </table> </div> <p>On all but the largest machine, the new landsat-util is at least twice as fast as previously.</p> <p>And it gets better: a significant amount of time (about a minute on average) is used up to decompress the NASA bundle after downloading it. Landsat-util v0.5 takes advantage of <a href="https://developmentseed.org/blog/2015/03/19/aws-landsat-archive/">AWS’s new landsat archive</a> of unzipped bands, saving even more time.</p> <p>We hope you enjoy the new landsat-util! Fork it, modify it, break it or just use it and tell us about all the ways you’re incorporating satellite data in your apps. </p> 2015-03-28T02:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/28/twice-as-fast We're in the Philippines http://www.developmentseed.org/blog/2015/03/27/in-the-philippines <p class="dropcap">This week, <a href="/team/ian-schuler/">Ian</a> and I are in the Philippines working alongside the World Bank to set up collaborative, open source tools for government to track critical infrastructure, particularly road data.</p> <p>Good road data is crucial for smart development and disaster response. To help the national and local governments manage their road infrastructure, we’ll be building on open tools such as <a href="http://ideditor.com/">iD</a>, the popular editor for <a href="http://www.openstreetmap.org/#map=18/14.08530/121.14943">OpenStreetMap</a>. Our goal is to use open-source tools to create better data infrastructure, at an enterprise level, tailored to the information needs of government.</p> <p>We are <a href="https://developmentseed.org/blog/2014/06/30/understanding-risk-forum/">confessed fans</a> of OpenStreetMap and the ecosystem that surround it. These tools are battle-tested and optimized for the quantity of data that flows daily through OSM. We think these tools can make government work better, and so far we’ve seen terrific responses from officials here.</p> <p>We’ll be in the Philippines until next Tuesday. Ping us on <a href="https://twitter.com/dereklieu">Twitter</a> if you’d like to meet and talk OSM and managing infrastructure data.</p> 2015-03-27T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/27/in-the-philippines Bem-vinda Caroline Portugal http://www.developmentseed.org/blog/2015/03/24/bem-vinda-caroline <p>Welcome Caroline! Designer and architect <a href="/team/caroline-portugal/">Caroline Portugal</a> is joining Development Seed.</p> <p><img src="/assets/graphics/content/caroline-portugal-card.jpg" alt="" /></p> <p>Caroline is going to build thoughtful, beautiful software. Caroline hails from Brazil, where she studied and practiced Architecture and Urban Planning. She worked on architecture projects in Brazil and the US ranging from <a href="http://estudioporto.com/379741/5800883/gallery/rio-2016-olympic-park-master-plan">Rio’s Olympic Park Master Plan</a> to the National Museum of African American History &amp; Culture. Caroline’s latest work has been in visual design. She recently completed a graduate degree in Graphic Design from MICA.</p> <p>Caroline’s work is moving. It is visually stunning. It is grounded in solving practical challenges with humanity, creativity and fun. Check out her work on <a href="http://estudioporto.com/379741/5686749/gallery/bike-in-park-out">bike lanes</a> and <a href="http://estudioporto.com/379741/5762373/gallery/the-future-of-contraception">the future of contraception</a> to get a sense of why we are delighted to have her on the team.</p> <p>Caroline speaks fluent Portuguese. Say bem-vinda to <a href="https://twitter.com/portugal_moura">Caroline on twitter</a>.</p> 2015-03-24T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/24/bem-vinda-caroline Powering Landsat Power Tools http://www.developmentseed.org/blog/2015/03/19/aws-landsat-archive <p class="dropcap">Amazon Web Services just opened Landsat on AWS, a publicly available archive of Landsat 8 imagery hosted on their reliable and accessible infrastructure. This investment by the AWS open data team has a big impact on our work to make satellite imagery more accessible.</p> <p><a href="https://libra.developmentseed.org/">Libra</a> is an open source Landsat imagery browser that we built with <a href="http://astrodigital.com/">Astro Digital</a>. Libra now has options on some scenes to download individual bands related to specific types of imagery analysis like NDVI, or Urban False Color. The most recent Landsat–8 images are now available for download up to two days sooner. Last week we rolled out a <a href="/blog/2015/03/10/releasing-landsat-v05/">new version of landsat-util</a>, our open source utility for processing Landsat imagery. The new version is much faster and allows you to build <a href="https://en.wikipedia.org/wiki/False_color">false color composites</a> on the fly. These improvements to Libra and landsat-util are possible because we started using Landsat on AWS, which is a publicly available archive of Landsat 8 imagery hosted on Amazon S3 that is <a href="http://aws.amazon.com/public-data-sets/landsat/">publicly available today</a>.</p> <p><a href="https://libra.developmentseed.org/"><img src="https://farm8.staticflickr.com/7636/16864977481_2e31af08b6_o.png" alt="libra-band-download" /></a></p> <p><em>Libra now allows you to download the satellite bands for advanced analysis</em></p> <p>Our newest releases of Libra and landsat-util utilize Landsat on AWS for 2015 imagery. Landsat on AWS provides 2015 imagery as unzipped individual bands. AWS makes this imagery available extremely quickly, often within hours of capture. We can pull only the data that we need and to work with it immediately.</p> <p>Landsat 8 imagery is an incredibly powerful resource. It is some of the most valuable open data produced by the US Government. Our partners rely on Landsat data for everything from evaluating droughts to tracking conflict. However, until now, individual bands of Landsat imagery has never been available via predictable download endpoints that we can integrate into our tools.</p> <p>Libra and landsat-util now allow our partners to get imagery sooner and process it faster. Speed and ease are critical to our partners who use this data to respond to natural disasters, prevent hunger, and monitor elections.</p> <p><img src="https://farm9.staticflickr.com/8683/16866164475_1b90a33b72_h.jpg" alt="gulf of guinea" /></p> <p><em>Landsat images of the Gulf of Guinea, processed with <a href="https://github.com/developmentseed/landsat-util">landsat-util</a></em></p> <p>Thanks to the AWS team and collaborators–<a href="https://twitter.com/nfwarmerdam">Frank Warmerdam</a> at <a href="https://www.planet.com/">Planet Labs</a>, <a href="https://twitter.com/vruba">Charlie Loyd</a> at <a href="https://www.mapbox.com/">Mapbox</a>, and Peter Becker and others from <a href="https://www.esri.com/">ESRI</a>–for building a Landsat archive with developers in mind.</p> <p><a href="https://github.com/AstroDigital/libra">Libra</a> and <a href="https://github.com/developmentseed/landsat-util">landsat-util</a> are open source and on Github. Go ahead and fork or contribute.</p> 2015-03-19T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/19/aws-landsat-archive Launching v0.5 of landsat-util http://www.developmentseed.org/blog/2015/03/10/releasing-landsat-v05 <p class="dropcap">We just released a new version of landsat-util, <a href="https://github.com/developmentseed/landsat-util/releases/tag/v0.5.0">version 0.5</a>. This version is lighter and has fewer dependencies. Landsat-util v0.5 downloads and processes images much faster and gives users more control.</p> <p><a href="https://pypi.python.org/pypi/landsat-util/0.5.0">Installing landsat-util</a> now is considerably easier for Mac and Linux users. We are still working to make it as easy to run on Windows.</p> <p>This new version reflects a significant change in our thinking on landsat-util. We’ve moved to using faster and simpler frameworks to optimize processing. We removed heavy dependencies, such as ImageMagick and Orfeo Toolbox, that were causing installation problems. We leveraged faster processing frameworks like <a href="https://github.com/mapbox/rasterio">Rasterio</a>, <a href="https://github.com/numpy/numpy">numpy</a>, and <a href="https://github.com/scipy/scipy">scipy</a>. These changes significantly optimize disk and memory usage resulting in faster and less error-prone processing. Landsat-util can now process images 3x faster.</p> <p><img src="{{ site.baseurl }}/assets/graphics/content/landsat-util-v05.gif" alt="Running the utility" /></p> <p>We’ll post more of the technical details on how we rebuilt landsat-util and what’s under the hood. In the meantime hit us up at FOSS4G in San Francisco all week to learn more.</p> 2015-03-10T15:30:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/10/releasing-landsat-v05 Collect and verify mobile reports http://www.developmentseed.org/blog/2015/03/10/mobile-reporting-workflow <p>A common scenario for mobile reporting looks like this:</p> <ul> <li>A group wants to collect reports from their own trusted network and also to crowdsource reports from the public.</li> <li>The group has some process to try to verify crowdsourced reports and needs to track verification.</li> <li>The group wants to publish this data using simple visuals that answer their core question and invites comparison.</li> </ul> <p>We recently worked with Pursue Lebanon to build a system for reporting breakdowns in service delivery in Lebanon’s Palestinian refugee camps. Pursue has been working with community organizations in all twelve refugee camps for the past five years. With more attention on the refugee situation in Lebanon, there are opportunities to push for improved service delivery with better, timely data. With so many tools for mobile data collection, you’d think that this should be easy to do with open source tools. It’s not. Here is our experience.</p> <h3 id="collecting-data">Collecting Data</h3> <p><a href="https://opendatakit.org/">OpenDataKit</a> is a great open source tool for data collection on Android. When you have control over the device your volunteers are using, ODK is great. <em>(Side note: If you like ODK, check out <a href="https://github.com/AmericanRedCross/openmapkit">OpenMapKit</a> an exciting project of the American Red Cross, SpatialDev, and Ona).</em></p> <p>Like many organizations, Pursue relies on reports from volunteers using their own phones and can’t expect them to report through an Android app. This is why the ODK ecosystem and tools like <a href="https://enketo.org/">Enketo</a> and <a href="https://formhub.org/">Formhub</a> are interesting. Enketo allows us to serve a simple web form on any device. Formhub gives us the ability to convert and manage Excel-based forms. While this ecosystem is great for authoring forms, we found that this system had some shortcomings:</p> <ul> <li>managing multiple datasets can be a challenge,</li> <li>API lacks some features,</li> <li>system is resource hungry, and</li> <li>difficult to deploy and maintain.</li> </ul> <p>We use these tools for what they are best at, generating web forms from Excel documents. We pull out the data via the API to use in other open source tools.</p> <p><img src="https://farm8.staticflickr.com/7651/16803250171_c8525e5500_o.png" alt="" /></p> <h3 id="verification">Verification</h3> <p>No tool that we looked at had good workflow for verification. Most had no verification or had only a simple yes/no toggle. Many organizations need to append further verification data and notes to each report. Pursue’s partners are collecting and verifying thousands of reports. To make this manageable, the system must have a workflow for quickly reviewing whether reports are verified true, verified false or are unverified.</p> <p><img src="https://farm8.staticflickr.com/7633/16181960354_f01be0bf2c.jpg" alt="" /></p> <p>We used Django to build a verification platform that is useful and usable for the data verifier who is triaging hundreds of reports a day and for the field organizer who is trying to track the status of single report.</p> <h3 id="publishing">Publishing</h3> <p>To publish and visualize the collected data, we designed a map and report interface that highlights the trends in service breakdowns between camps and within neighborhoods in camps.</p> <p>It is important to clearly show users whether information is verified. For simplicity, we use a simple checkbox to filter out all reports that are “unverified” or “verified false”, even though we distinguish between these in the verification platform. We showed the verification status prominently on all report listings.</p> <p>Reporters can select a neighborhood and choose a level of geographic precision. To protect the privacy of reporters, the system does not require reports to submit precise location information. Trusted administrators use the precise location when it is provided to verify reports. Regardless of the location precision, the public platform only shows reports to the neighborhood level.</p> <p><img src="https://farm9.staticflickr.com/8697/16182028384_eb77344140_c.jpg" alt="" /></p> <p>Join <a href="https://2015.foss4g-na.org/session/empowering-refugees-data">our talk on Wednesday morning</a> to hear more. We’ll follow up later in the day with a Birds of a Feather session discussing how the open source data collection community continues to grow.</p> 2015-03-10T14:30:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/10/mobile-reporting-workflow Hello San Francisco; Ready for FOSS4G http://www.developmentseed.org/blog/2015/03/09/in-sf-for-foss4g <p class="dropcap">We’re out in San Francisco this week for <a href="https://2015.foss4g-na.org/">FOSS4G North America</a>. Look for <a href="http://developmentseed.org/team/alireza-j/">Alireza</a> and I at the conference or at our session <a href="https://2015.foss4g-na.org/session/empowering-refugees-data">Wednesday morning at 10:30am</a>. Alireza will discuss using the OpenDataKit ecosystem to build mobile reporting and verification tools for refugee camps in Lebanon. We’ll also be at the Birds of a Feather session discussing the future of OpenDataKit.</p> <p><img src="https://farm9.staticflickr.com/8632/16581540300_e1a3d465be_c.jpg" alt="SF Landsat" /><br /> <em>Landsat 8 image over San Francisco on December 31, 2014</em></p> <p>This week we will release new versions of <a href="https://github.com/developmentseed/landsat-util">landsat-util</a> and <a href="http://libra.developmentseed.org/">Libra</a> with some powerful new features. We are also working on some tools for <a href="http://www.developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata/">managing OpenStreetMap data</a>. We look forward to collaborating with old and new friends in the open source community to improve the tools for mobile data collection, Open Geo, and satellite imagery processing. Tweet us at <a href="https://twitter.com/nas_smith">@nas_smith</a> or <a href="https://twitter.com/scisco7">@scisco7</a> to meet up. </p> 2015-03-09T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/03/09/in-sf-for-foss4g Tapping into OpenStreetMap Metadata http://www.developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata <p class="dropcap">We just launched <a href="https://github.com/osmlab/osm-meta-util">v0.1 of a new tool</a> to tap into OpenStreetMap changeset metadata. We built the tool in partnership with the American Red Cross as part of the infrastructure for tracking efforts such as <a href="http://www.missingmaps.org/">#MissingMaps</a>.</p> <p><a href="http://wiki.openstreetmap.org/wiki/Changeset">OpenStreetMap changesets</a> give us access to a wealth of metadata information that is not specifically geographic but incredibly rich. Metadata is helpful in understanding the changing nature of OSM. This is different from using geographic APIs like <a href="https://github.com/drolbr/Overpass-API">Overpass</a> because metadata contains commit text, number of edits, which editor was used, etc. With metadata, we can <a href="http://resultmaps.neis-one.org/osm-missingmaps">track hashtags</a>, analyze commit text or aggregate user metrics.</p> <p>In 2014 alone, users committed over 6 million changesets to OSM. As OpenStreetMap’s metadata grows, dealing with the sheer amount can be daunting. We built <a href="https://github.com/osmlab/osm-meta-util">osm-meta-util</a> as an experiment in making OSM metadata easier and faster to use.</p> <p><img src="{{ site.baseurl }}/images/gifs/osm-meta.gif" alt="Running the utility" /></p> <p>osm-meta-util focuses on two core functions: downloading the minutely compressed metadata files and serializing into JSON. We convert compressed OSM XML files containing multiple commits to a stream of JSON objects that can be piped to any tool or API. </p> <p>You can use the library in a Node application or as a command-line utility to download all the data between two dates: </p> <p>{% highlight javascript %}<br /> MetaUtil({<br /> ‘start’: process.argv[2],<br /> ‘end’: process.argv[3],<br /> ‘delay’: process.argv[4]<br /> }).pipe(process.stdout)<br /> {% endhighlight %}</p> <p>In combination with <a href="https://stedolan.github.io/jq/">jq</a>, to get a commit history we can simply run: </p> <p>{% highlight sh %}<br /> node app 001181708 001181721 1000 | jq -c ‘{date: .created_at, text: .comment}’<br /> {% endhighlight %}</p> <p>If you don’t give the tool any parameters, it will get the latest changesets and update every minute.</p> <p>We’re using this utility to experiment building a metadata API with the American Red Cross. But we know there are many more uses of the rich OSM metadata and want to see what others can do with the tool. Together with the American Red Cross we’ve put this on <a href="https://github.com/osmlab">OSM Lab</a>, a Github organization for OSM related projects. Follow the development of osm-meta-util on <a href="https://github.com/osmlab/osm-meta-util">Github</a>. </p> 2015-02-19T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/19/tapping-into-osm-metadata Open Data Day Garage Party http://www.developmentseed.org/blog/2015/02/13/odddc-garage-party <p class="dropcap">We love open data. And we love to talk about it over drinks with other open data lovers. Next Friday, February 20th at 7:00pm we are hosting an <a href="http://opendatadaydcparty.splashthat.com/">Open Data Day celebration</a> in the Mapbox Garage. Head over to the Garage after the first day of <a href="http://dc.opendataday.org/">Open Data Day DC</a> to talk about open tools for open data.</p> <p><a href="https://www.flickr.com/photos/developmentseed/16332412278/"><img src="https://farm9.staticflickr.com/8589/16332412278_ea7bcf4db8_c.jpg" alt="" /></a></p> <p>Even if you can’t make it to Open Data Day DC (or didn’t grab a spot on the now closed list), come over and share the #opendatalove.</p> <p>Let us know if you’re in by <a href="http://opendatadaydcparty.splashthat.com">RSVPing now</a>.</p> 2015-02-13T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/13/odddc-garage-party Data Hungry Happy Hour http://www.developmentseed.org/blog/2015/02/06/data-hungry-happy-hour <p class="dropcap">We’ll be at the <a href="http://www.tffchallenge.com/summit">Thought For Food Global Summit</a> next week working with some brilliant people on the biggest challenges in agriculture and feeding the hungry. Look for <a href="http://www.developmentseed.org/team/olaf-veerman/">Olaf</a> at the Summit.</p> <p>If you are in Lisbon for the Summit, please join us for a <a href="http://datahungryhappyhour.splashthat.com/">Data Hungry happy hour</a> on Thursday evening. We’ll be geeking out on better data, satellites, and sensors can contribute to better food policy. The Happy Hour kicks off at 18.30 in Fabulas in the center of Lisbon. The first couple of rounds are on us. You can <a href="http://datahungryhappyhour.splashthat.com/">RSVP here</a>.</p> <p><a href="http://datahungryhappyhour.splashthat.com/"><img src="https://cloud.githubusercontent.com/assets/751330/6078058/07dbb68a-aded-11e4-8d49-15a409237d70.png" alt="Data Hungry Happy Hour" /></a></p> 2015-02-06T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/06/data-hungry-happy-hour Welcome Mariano Arrien-Gomez http://www.developmentseed.org/blog/2015/02/02/welcome-mariano <p><a href="/team/mariano-arrien-gomez/">Mariano Arrien-Gomez</a> is an artist and designer. He builds <a href="https://instagram.com/mariano.ag/">beautiful graphics</a> and data visualizations that make our products more compelling, creative, and humane. Mariano directs his design skills toward the issues and topics he is passionate about, from preserving local parks, to genetically modified food, to soccer. He utilizes a range of <a href="http://marrieng.com/">visual mediums</a> including illustration, painting murals, and photography.</p> <p><img src="http://marrieng.com/images_dcbrau/dcbrau_pics_s06.jpg" alt="mariano" /></p> <p>In addition to his work at Development Seed, Mariano is active in the DC art scene. Mariano received a Bachelors of Fine Arts in Graphic Design from Virginia Commonwealth University. He is fluent in Spanish.</p> 2015-02-02T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/02/02/welcome-mariano Announcing Libra - the Landsat imagery browser you will love http://www.developmentseed.org/blog/2015/01/22/announcing-libra <p class="dropcap">We’ve been <a href="https://medium.com/@astrodigital/browsing-large-sets-of-satellite-imagery-7096db1a807f">working with Astro Digital</a> to produce <a href="http://www.developmentseed.org/blog/2015/01/15/dauria-image-search/">the most usable imagery browser</a>. Today we are releasing <a href="http://libra.developmentseed.org">Libra</a>, a fork of the Astro Digital browser for open Landsat data. Libra allows you to browse, sort, and download more than 275 Terabytes of open Landsat imagery as easily as booking an Uber.</p> <p><a href="http://libra.developmentseed.org/"><img src="https://farm9.staticflickr.com/8682/16157701429_8f417679a4_o.png" alt="libra" /></a></p> <h3 id="liberating-landsat">Liberating Landsat</h3> <p>We love open imagery. The global development organizations and developing governments that we work with use open satellite imagery for everything from evaluating disaster response, to tracking deforestation, to planning for drought. For our partners, open imagery isn’t just a matter of cost; it is a matter of licensing and distribution. They get immediate access to Landsat images and can analyze, manipulate, and distribute with almost no restrictions.</p> <p>To make Landsat data more useful, we’ve made it easier to use. We built two open source tools for working with Landsat data - <a href="https://github.com/developmentseed/landsat-util">Landsat-util</a> and <a href="https://github.com/developmentseed/landsat-api">Landsat API</a>. It used to take all day for Development Seed’s imagery specialists to turn Landsat data into imagery layers for online maps. With these two tools, any developer can do it in a matter of minutes.</p> <p>These tools gave us a huge head start in building Libra. Libra relies heavily on Landsat API to quickly query by date, geography, and cloud cover and get image URLs, scene centroids, scene boundaries, and other metadata. Using Landsat API as a backbone of Libra also encouraged us to make improvements and configuration changes on Landsat API such as changing the limits on requests and returned data and some error handling.</p> <p>Have a look at <a href="http://libra.developmentseed.org">Libra</a> and hit us with your feedback <a href="https://twitter.com/developmentseed">@developmentseed</a>.</p> <p><em>Feb 16 2015: We’ve updated this post to reflect the Dauria Geo name change to Astro Digital.</em></p> 2015-01-22T18:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/22/announcing-libra Introducing Development Seed Lisbon http://www.developmentseed.org/blog/2015/01/20/introducing-development-seed-lisbon <p class="dropcap">Last week we announced the opening of <a href="http://www.developmentseed.org/blog/2015/01/15/development-seed-opens-office-in-lisbon/">Development Seed Lisbon</a>. To kickstart our operations in Europe, we brought on our friends from Flipside, an experienced team working on meaningful open data projects for organizations around Europe. This move allows us to connect with partners and talent in the region, and also deliver quality work right out of the gate.</p> <h2 id="olaf-veerman">Olaf Veerman</h2> <p>Olaf will lead the Lisbon office, run projects, and help us establish a strong presence in Europe. He lived for many years in Latin America, working with small business networks, cooperatives and small farmer groups in Brazil, Uruguay and Venezuela. His experience in working with civil society organizatons around the world allows him to quickly understand our partners’ needs and help them use technology to increase their social and economic impact. </p> <p><img src="https://farm8.staticflickr.com/7570/16138839469_890fdabaa6_b.jpg" alt="" /></p> <p>You can connect with Olaf through <a href="http://twitter.com/obirdman">Twitter</a>.</p> <h2 id="daniel-da-silva">Daniel da Silva</h2> <p>Daniel brings solid engineering skills that he applies to anything from building light-weight frontends with well structured APIs, to deploying tools for offline/online data collection. He is a quick learner and problem solver whose technology expertise spans PHP, Node, Angular, Jekyll and Mongodb. Daniel is going to help us pick the right tool for the job and deliver quality work to our partners.</p> <p><img src="https://farm9.staticflickr.com/8613/16137670630_c7a9b6da29_b.jpg" alt="" /></p> <p>Find Daniel on <a href="https://github.com/danielfdsilva">Github</a>.</p> <h2 id="ricardo-mestre">Ricardo Mestre</h2> <p>Ricardo is a talented designer and front-end developer who pushes how modern technologies can be used to craft usable and engaging websites. He worked for some of the biggest companies in Portugal, but is most passionate about free culture and equality, which he contributes to through his music, art and <a href="http://duplos.org/">other projects</a>. Ricardo is going to have a big impact on the design and usability of our work.</p> <p><img src="https://farm8.staticflickr.com/7551/15705150643_5bb48dac7d_z.jpg" alt="" /></p> <p>You can find Ricardo on <a href="http://twitter.com/ricardoduplos">Twitter</a>.</p> 2015-01-20T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/20/introducing-development-seed-lisbon Building a Better Satellite Imagery Browser http://www.developmentseed.org/blog/2015/01/15/dauria-image-search <p class="dropcap">Satellite imagery companies spend billions of dollars acquiring pictures of the planet. However, anyone who has worked with that imagery knows that the tools for searching and accessing this imagery are painful to work with. That’s why we are excited to be working with <a href="http://astrodigital.com/products/">Astro Digital</a> to build a fast, easy, imagery browser. To do that, we are relentlessly reducing friction and creating something that feels more like Airbnb or Pinterest than traditional GIS-based imagery browsers.</p> <p>Astro Digital’s strategy is built on <a href="https://medium.com/@astrodigital/browsing-large-sets-of-satellite-imagery-7096db1a807f">making imagery easy to integrate across every industry</a>. Here is how we worked with them to bring usability to image browsing:</p> <h3 id="discover">Discover</h3> <p>A primary goal is to eliminate the steps required to start looking at imagery. Astro Digital discussed the “<a href="https://medium.com/@astrodigital/browsing-large-sets-of-satellite-imagery-7096db1a807f">bounding box problem</a>” – traditional imagery browsers require you to provide a bounding box to start a search. The user either has to outline a polygon or upload a file with the bounding box. If the bounding box is too big or returns too many results these sites often require you to start over.</p> <p>Our solution? Just zoom.</p> <p><a href="http://cl.ly/image/072m1k042a3z/dauria-just-zoom.gif"><img src="http://cl.ly/image/072m1k042a3z/dauria-just-zoom.gif" alt="just-zoom" /></a></p> <p>We will explore options for advanced search that may include boundary uploads, and those interactions will follow the same simple design concepts foundational to a good user experience. As we add those features, we will preserve the ability of users to immediately start interacting in a manner they expect from a modern website.</p> <h3 id="responsive-filter-and-sort">Responsive filter and sort</h3> <p>Once the user has identified their area of interest, the next step is sifting through all the returned images to find the best one for their needs. We’ve made this easier through visual filters and sorting tools that help you quickly get to the best image. Familiar icons allow users to filter and sort without having to learn the interface. We included visual tools like a histogram on our filters to help users intuitively understand the implications of their filtering choices. Sorting assumes “best to worst” to make sorting easy and to surface the best images. All along, we provide thumbnail previews of each image so that you can immediately see what you are getting.</p> <p><a href="http://cl.ly/image/0E3C230f370i/dauria-filter-sort.gif"><img src="http://cl.ly/image/0E3C230f370i/dauria-filter-sort.gif" alt="filter-sort" /></a></p> <h3 id="making-the-interactions-obvious">Making the interactions obvious</h3> <p>Our aim is to create interactions that seem natural without the need for a website tour. Moving around the map shows coverage areas. Selecting an area filters results to that geography. We benefited here by using Landsat imagery that has a consistent bounding area, but we believe we can keep this intuitive even with different shaped scenes. To keep user interaction from breaking, we did a lot of work under the hood to make the site fast and responsive even while loading a lot of data and images. With a streamlined request process, once you’ve found the right image, you can download or purchase in one click.</p> <p><a href="http://cl.ly/image/3Z0B2r1y1Y2m/dauria-obvious-ui.gif"><img src="http://cl.ly/image/3Z0B2r1y1Y2m/dauria-obvious-ui.gif" alt="obvious-ui" /></a></p> <p>We were able to quickly stand up a fast mapping application by building off the <a href="https://www.mapbox.com/">Mapbox</a> stack. Simple, intuitive tools make it easier for more sectors to benefit from satellite imagery. As we help global development organizations and emerging economies to use imagery products, usable tools will be critical. We think existing imagery users will also appreciate having a more enjoyable relationship with their imagery finder.</p> <p><em>Feb 16 2015: We’ve updated this post to reflect the Dauria Geo name change to Astro Digital.</em></p> 2015-01-15T11:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/15/dauria-image-search Development Seed opens office in Lisbon http://www.developmentseed.org/blog/2015/01/15/development-seed-opens-office-in-lisbon <p class="dropcap">Development Seed is opening an office in Lisbon. Our team grows by a continent today in beautiful Portugal where we will continue to build data tools and solve complex development challenges. Establishing an office in Europe puts us closer to our partners in Europe, Middle East, and Africa. It will also allow us to better connect to the talented open data hacker movement in the region.</p> <p><img src="https://farm8.staticflickr.com/7536/16100603687_40714edc2c_b.jpg" alt="" /></p> <p>To bootstrap our European team we are immediately bringing on all our talented friends at Flipside. The Flipside team have been doing fantastic work on projects ranging from opening data on clean energy, building mobile monitoring tools with Text To Change, and tracking forest fires around Portugal. The entire team joins Development Seed today. Olaf Veerman from Flipside will lead the Portugal office and help us to grow the team.</p> <p>If you are in Lisbon, come and celebrate with us tonight at our <a href="http://opendatahappyhour.splashthat.com/">Open Data Happy Hour</a>.</p> 2015-01-15T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/15/development-seed-opens-office-in-lisbon Open Data Happy Hour in Lisbon http://www.developmentseed.org/blog/2015/01/09/open-data-happy-hour <p><a href="http://opendatahappyhour.splashthat.com/"><img src="https://farm8.staticflickr.com/7473/16238390085_f53e289df1_c.jpg" alt="" /></a></p> <p class="dropcap">Next week we’ll be in Lisbon where we’re hosting an <a href="http://opendatahappyhour.splashthat.com/">Open Data Happy Hour</a> with our friends from <a href="http://flipside.org">Flipside</a>. If you’re in the neighborhood and want to talk Open Data, know more about our satellite imagery work, or geek out on sensors, make sure to drop by and have a drink with us.</p> <p>The Happy Hour will be hosted at <a href="http://liberdade229.com">Liberdade 229</a> and start around 19:00. We hope to see you there. <a href="http://opendatahappyhour.splashthat.com/">RSVP here</a>.</p> 2015-01-09T11:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/09/open-data-happy-hour Welcome Joe Flasher http://www.developmentseed.org/blog/2015/01/08/howdy-joe <p class="dropcap"><a href="https://www.developmentseed.org/team/joe-flasher">Joe Flasher</a> is joining Development Seed to help us run faster and to broaden our technology stack. Whether he’s launching Delta II rockets, growing the open data community in Mongolia, or connecting an arcade claw game to a Facebook app, Joe is always looking for better ways to bend technology to solve real problems. He is constantly seeking the solution that is more elegant, more practical, and more human.</p> <p>Joe is the perfect person to help Development Seed grow and explore. He’s going push us to build better products and to make bigger open source contributions across satellite imagery, sensors, drones, and devices.</p> <p>Give Joe a holler on <a href="https://twitter.com/joseph_flasher">Twitter</a> or <a href="https://github.com/jflasher">Github</a>.</p> <p><img src="https://farm8.staticflickr.com/7523/16043833640_bb95464a00_c.jpg" alt="" /></p> 2015-01-08T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2015/01/08/howdy-joe 2014 Web Index measures the Internet's openness and impact http://www.developmentseed.org/blog/2014/12/12/launching-web-index <p>The World Wide Web Foundation released their <a href="http://thewebindex.org/">2014 Web Index</a>, an annual accounting of how the internet has changed the economic, political, and social lives of citizens across every continent.</p> <p>This is an ambitious and complicated story. The impact of the internet on the 84 countries in the 2014 report is difficult to isolate from culture, politics, and economics. Ultimately, the report found that open and inclusive internet correlate to equitable societies. The report is rich with data and worth reading <a href="http://thewebindex.org/">in full</a>.</p> <h4 id="visualizing-liquid-data">Visualizing liquid data</h4> <p>Making useful and appropriate comparisons between 84 countries is hard. To create the Web Index, the Web Foundation pulled a range of primary and secondary data into normalized indicators, which together comprise the index. The report, produced online in collaboration with <a href="http://www.weso.es/">WESO</a>, gives a global overview in addition to nuanced subviews on topics such as internet censorship and gender equality.</p> <p>When we first began collaborating with Web Foundation to visualize this information, they were in the process of collecting and interpreting their data. They had identified key themes that they knew would be important: economic equality, neutrality, censorship and surveillance, and gender-based violence, among others.</p> <p>Using these leads, we dived into the preliminary results in search of effective, telling visuals. Quick feedback loops and frequent communication with research specialists at Web Foundation helped us to build better visuals. It also helped Web Foundation better understand their data and how to communicate it.</p> <p><img src="https://c1.staticflickr.com/9/8626/15992468205_b8979f541d_c.jpg" alt="" /></p> <h4 id="comparing-countries">Comparing Countries</h4> <p>We ran up against the challenge of showing 84 countries in interesting ways, while doing justice to the data. Although the data is country-level, not every story lends itself to a map. In our visuals, we use flags, bubbles, and names to indicate countries.</p> <p>In plotting gender-based violence, we ran into the problem of page size. The graph examines countries with stark differences in how they support victims of online gender-based violence, and how frequently they prosecute perpetrators of that violence. Each graphic had to be single page and embeddable. To make this fit, we used a <a href="http://bost.ocks.org/mike/fisheye/">fisheye</a> effect that moves with your cursor.</p> <p><img src="http://i.imgur.com/QM5sM9g.gif" alt="" /></p> <h3 id="open-data-about-the-open-internet">Open Data about the open Internet</h3> <p>The Web Index is fully open. You can download <a href="http://thewebindex.org/downloads/">all of the data here</a>. The Web Foundation and WESO built an open API with data from this year and previous years. The code for <a href="https://github.com/weso/webindex-prototype">their site</a> and <a href="https://github.com/weso/webindex-prototype/tree/master/app/visualisations">our four visualizations</a> are open as well.</p> 2014-12-12T01:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/12/12/launching-web-index A Home for Open Housing Data http://www.developmentseed.org/blog/2014/11/17/housing-in-illinois <p>The housing crisis deeply impacted millions of Americans, and today the <a href="http://www.npr.org/2014/11/18/364131391/firm-accused-of-illegal-practices-that-push-families-into-foreclosure">effects are still being felt</a>. For the many problems facing communities, from wage disparity to affordable housing, there isn’t a single fits-all solution.</p> <p>That’s why we are very excited about the work that <a href="http://www.woodstockinst.org/about">Woodstock Foundation</a> is doing to support fair housing policies. Today they launch a nuanced look at housing and income disparity in Illinois in the form of a new <a href="http://www.woodstockinst.org/content/woodstock-institute-interactive-map">map-based open data website</a>.</p> <p>The site brings together 74 datasets on the well-being of local communities. It is a good roadmap for anyone working in housing justice in Illinois. Community organizations can explore <a href="http://woodstockinst.github.io/woodstock-maps/#mortgage/census_tract/avg_loan_value/2012">the average amount of mortgage debt people take on</a> and the rate of foreclosure filings in <a href="http://woodstockinst.github.io/woodstock-maps/#chicago_foreclosure/census_tract/foreclosure_filings_per_1000/2013">the Chicago six-county region</a> and <a href="http://woodstockinst.github.io/woodstock-maps/#rest_of_illinois_foreclosure/census_tract/nos_per_1000/2013">elsewhere in Illinois</a> to inform their decisions on where to focus their work. Some <a href="http://woodstockinst.github.io/woodstock-maps/#housing/census_tract/long_term_vacancy/2008">data sets go back to 2008</a>.</p> <p><img src="https://c2.staticflickr.com/6/5606/15813741921_6b51750a2a_b.jpg" alt="high-income" /><br /> <em>Where the highest income census tracts are in the Chicago six-county region</em></p> <h3 id="serving-complex-data-through-static-json">Serving complex data through static JSON</h3> <p>Housing data is complex, and Woodstock has gathered some amazingly granular statistics about housing data in Illinois. Splitting this data into a format that we could serve over the web proved a difficult challenge. Woodstock is also a small nonprofit, and we wanted to ease as much as possible the burden of maintaining a complicated website.</p> <p>So we wrote a python library to break up their spreadsheets into <a href="http://en.wikipedia.org/wiki/JSON">JSON</a>. Every time you switch to a new facet of housing data, every time you view a different year or category of that data, your browser incrementally downloads a new JSON file. Although those files in aggregate would take ages to load, individually they are manageable. Those scripts, along with the rest of the site, are <a href="https://github.com/woodstockinst/woodstock-maps">open-source</a>.</p> <p>To further reduce load times, we use the <a href="https://github.com/mbostock/topojson">topojson</a> spec to reduce the size of geographic boundary data. This allows us to separate geographic data from numerical data, so you only download those complex census tract boundaries once. The code that runs in your browser than re-connects those boundaries to the housing or mortgage data you select on the fly.</p> <p>Using vector-based geographical boundaries has other benefits. It allowed us to use a mouse click on an overlaying geographic boundary and a <a href="http://stackoverflow.com/questions/217578/point-in-polygon-aka-hit-test">point-in-polygon</a> test to find, for example, which congressional representative is responsible for which census tract.</p> <p><img src="https://c2.staticflickr.com/6/5611/15195642794_9075b6af59_b.jpg" alt="collision-detection" /><br /> <em>Detecting congressional boundaries</em></p> <h3 id="census-tracts-over-multiple-years">Census tracts over multiple years</h3> <p>Census tracts change a lot over census years, and this can be a problem when mapping a multi-year data set that covers more than a single census geography file. Attempting to compare the data between those years that use different tract definitions can be tricky. Tract ID, or FIPS codes can either refer to a different neighborhood or disappear entirely.</p> <p>The Census releases <a href="https://www.census.gov/geo/maps-data/data/relationship.html">relationship files</a> that show where these changes and additions occur. Using this, we created a tool that overlays 2000 and 2010 census tracts, and <a href="http://www.developmentseed.org/blog/2014/04/25/making-sense-of-census-changes/">shows differences between the two years</a>.</p> <p>Currently the project covers only Illinois, but the code is available on <a href="https://github.com/developmentseed/census-tracts/tree/gh-pages">GitHub</a> and ready for you to fork and contribute your own state.</p> 2014-11-17T12:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/17/housing-in-illinois Join us at EcoHack this weekend http://www.developmentseed.org/blog/2014/11/13/hosting-ecohack <p><img src="https://farm9.staticflickr.com/8625/15781769355_8877307f0c_b.jpg" alt="" /><br /> <em>Toxic algal bloom in Lake Erie</em></p> <p>To address climate change and promote environmental justice, we need better tools to understand our changing planet. That’s why we are delighted to help host the <a href="https://nvite.com/EcoHack/b41">DC EcoHack</a> with <a href="http://www.wri.org/">WRI</a> at the Mapbox Garage.</p> <p>EcoHack is an event to bring together a diverse community of scientists, hackers, designers and others who want to tell stories and create tools to protect our environment. The event is open to people of all skill levels. As long you’re interested in using technology to improve and better understand our natural environment, we’d love to see you there.</p> <p>If you’re in DC, <a href="https://nvite.com/EcoHack/b41">register</a> and come join us at the <a href="https://www.mapbox.com/">MapBox</a> Garage in DC. Our friends at Mapbox are also hosting the <a href="https://www.mapbox.com/blog/mapbox-ecohack/">San Francisco EcoHack</a> and EcoHacks will also take place in Sydney, Cambridge, New York, and Madrid.</p> <p><a href="http://www.developmentseed.org/team/drew-bollinger/">Drew</a> and <a href="http://www.developmentseed.org/team/marc-farra/">Marc</a> will be using some of the time to work on <a href="http://www.developmentseed.org/blog/2014/08/29/landsat-util/">landsat-util</a>, an open source tool that makes it easier to work with open satellite imagery. We are also keen to help on projects using <a href="http://www.developmentseed.org/blog/2014/10/28/climatescope/">green energy investment data</a> and <a href="http://www.developmentseed.org/blog/2014/09/15/flood-mapping/">tracking natural disasters</a>.</p> <p>Hope to see you there!</p> 2014-11-13T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/13/hosting-ecohack International Conference of Crisis Mappers storms NYC http://www.developmentseed.org/blog/2014/11/06/iccm-storms-nyc <p>Today, <a href="http://www.developmentseed.org/team/marc-farra/">Marc</a> and I are heading up to New York to attend the <a href="http://crisismappers.net/">International Conference of Crisis Mappers</a>. Open crisis mapping is growing up. We are seeing greater demand for maps and data for crisis response and preparation. ICCM 2014 will be an important place to discuss how we can grow to handle this need, how we better generate real collaboration from data, and how we build infrastructure that is usable and inclusive.</p> <p><img src="https://farm4.staticflickr.com/3944/15726853685_0edd86b2c7_c.jpg" alt="" /><br /> <em>Satellite imagery over Monrovia, an area that has seen a significant burden of Ebola cases.</em></p> <p>We’re looking forward to talking about OpenStreetMap, satellites, and open data; topics that are critical in the midst of the response to the Ebola outbreak in West Africa. Reach out to <a href="http://twitter.com/nas_smith">@nas_smith</a> or <a href="http://twitter.com/kamicut">@kamicut</a> on Twitter if you want to chat.</p> 2014-11-06T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/06/iccm-storms-nyc Getting to vote http://www.developmentseed.org/blog/2014/11/04/us-election <script type="text/javascript" src="https://s3.amazonaws.com/vip-voter-information-tool/app.js"></script> <div id="_vit"></div> <script type="text/javascript">vit.load({officialOnly:false});</script> <p><em>Working embed. US voters, find your polling location</em></p> <p>In the US, and around the world, it can be confusing figuring out where you should vote and which races you are eligible to vote on. In the US, going to the wrong polling station is a hassle. This hassle can be prohibitive, particularly if you are disabled or rely on public transportation. In countries coming out of conflict, going to the polling station can be a brave and risky act. You’d better be at the right place when you get there.</p> <p>Open data is helping to get voters the information they need to participate. By opening up data about voting locations and process, States can involve private actors, such as <a href="http://www.pewtrusts.org/en">The Pew Trusts</a> and Google, as partners providing accurate information about where to vote. The <a href="https://www.votinginfoproject.org/about/">VIP (Voter Information Project)</a> is an embeddable tool, based on an <a href="https://github.com/votinginfoproject/vip-embeddable-tool">open source</a> stack, that relies on an <a href="https://developers.google.com/civic-information/">open API</a>. This allows other groups to repackage and distribute this information to their audiences.</p> <p>By opening up the data, States are no longer solely responsible for getting voters to the right polling place. Voters should be tripping over this information in every Google search, Foursquare check-in, community message board, and favorite blogs.</p> <p><img src="https://farm4.staticflickr.com/3030/3002704954_94a4df58ee_o.jpg" alt="voting demonstration" /><br /> <em>“Usability and Voting” by <a href="https://www.flickr.com/photos/ericgundersen/3002704954/in/photolist-5zkUTU-5zkUhL-5zkRms-5zkTFS-5zgAm6-5zgzRV-5zgAST-5zkDz5-aqq3t9-cgCsHQ-5zgfND-5zfZMc">ericgundersen</a></em></p> 2014-11-04T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/04/us-election Hey there, designers. http://www.developmentseed.org/blog/2014/11/02/attention-designer <p>At Development Seed, design is not about pushing pixels or passing a perfect mockup to the next person. It is about truly understanding — and sometimes defining — a problem, working out a systemic solution with visual and interactive components. You will be brainstorming solutions with our strategists and turning them into sketches, websites, and data visualizations with fellow developers.</p> <p>We build new ways to help people make decisions — impacting policies and creating transparency on all levels. We are hiring a designer who is a doer and a thinker, eager to join our mission.</p> <h3 id="you-are">You are:</h3> <ul> <li>an artist; you have a favorite medium to express your ideas, be it ink, paint, vector or gif</li> <li>excited about the web, particularly how the representation of information on screens can inform people’s decisions</li> <li>eager to work with data and the patterns it leads to</li> <li>curious and hungry to learn new subjects and skills</li> </ul> <h3 id="you-know-how-to">You know how to:</h3> <ul> <li>ask good questions and get to the heart of a problem</li> <li>illustrate abstract concepts and workflows in visual forms</li> <li>use the right font and color at the right time, knowing that aesthetics is derived from your communication goal, the information, and the medium</li> </ul> <h3 id="experiences-with-any-of-the-following-will-be-a-plus">Experiences with any of the following will be a plus:</h3> <ul> <li>Web maps (such as a map made with MapBox Studio)</li> <li>Responsive web frameworks</li> <li>Git</li> <li>D3.js for visualization (bar chart counts)</li> <li>Static site generation (Jekyll, Flask, etc)</li> <li>SASS</li> </ul> <p>But don’t let any of that scare you. If your design chops are good, we will work with you to tech up on everything you need to know.</p> <h3 id="to-apply">To apply:</h3> <p>Please send your portfolio site, and links to three projects, to jobs@developmentseed.org with “Designer” in your subject.</p> 2014-11-02T10:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/11/02/attention-designer Astro Digital completing new design specs http://www.developmentseed.org/blog/2014/10/30/dauria-launch <p><img src="https://farm4.staticflickr.com/3949/15481548030_aaf1e9b01d_o.gif" alt="brazil deforestation" /></p> <p>We’re collaborating with Astro Digital now as they build out integration and visualization tools – from antenna to API, that makes image acquisition, analysis, delivery, and integration easier for both NGOs and enterprise. Astro Digital is building an API to empower developers to access fresh and historical imagery, compute needed data layers on the cloud, and harvest data in ready-to-use format. Their cloud platform can do heavy analysis and feed data directly into applications. By directly integrating with the <a href="https://www.mapbox.com/developers/api/">Mapbox API</a>, we can quickly deploy sophisticated and beautiful applications from agriculture to disaster response using the platforms that developers are already building on.</p> <p><img src="https://farm9.staticflickr.com/8591/16726324832_e6c248cc9e_b.jpg" alt="astro satellites" /></p> <p>We’d love to see more satellite providers compete on ease of integrating their data. We’ll be helping Astro Digital to review their API to make it developer friendly and we will build open source tools on top Astro Digital’s API. These tools will serve as open templates for integrating Astro Digital with tools like Mapbox to quickly build powerful, data-rich sites. This is really positive move for the industry and for users.</p> 2014-10-30T12:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/30/dauria-launch Marhaba Marc Farra http://www.developmentseed.org/blog/2014/10/28/marhaba-marc <p><a href="http://www.developmentseed.org/team/marc-farra/">Marc Farra</a> has joined the Development Seed team. Marc loves to experiment with image processing, arduino sensors, and data infrastructure. He is going to help us explore new ways to collect and process data.</p> <p>I first ran into Marc in Beirut. At the time he was running Lamba Labs, a hacker space in Beirut that was sowing the seeds of Maker culture and open data advocacy in Lebanon. A year later, he took the Afghanistan polling station locations we <a href="https://github.com/developmentseed/aodp-data">posted on Github</a> and started to build <a href="http://www.developmentseed.org/blog/2014/04/04/votehere-af/">a mobile app for Afghans to locate their nearest polling station</a>.</p> <p><img src="https://cloud.githubusercontent.com/assets/5984548/4445705/9555418c-47fb-11e4-81b5-a12a3573c74b.jpg" alt="" /></p> <p>Say “Hello”, “Salut”, or “Marhaba” to Marc on <a href="https://twitter.com/kamicut">Twitter</a> and <a href="https://github.com/kamicut">Github</a>.</p> 2014-10-28T15:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/28/marhaba-marc Getting Green into Green Energy http://www.developmentseed.org/blog/2014/10/28/climatescope <p>Reversing climate change means investing in green energy, and as the sustainable sector grows, ensuring it grows in both developing and developed countries. Today the <a href="http://www.fomin.org/">Fondo Multilateral de Inversiones</a> and <a href="http://about.bnef.com">Bloomberg New Energy Finance</a> are launching a vastly expanded <a href="http://global-climatescope.org/">Climatescope</a>, to provide open data about green energy investment in 55 countries. The data provided by Climatescope creates an information-rich environment for green energy investors. It also provides valuable data on clean energy policies for activists and policymakers.</p> <p><img src="https://farm4.staticflickr.com/3946/15025140024_eb0c81e77f_o.png" alt="climatescope" /></p> <h3 id="opening-climate-investment-data">Opening Climate Investment Data</h3> <p>We worked on the Climatescope website with <a href="http://flipside.org/">Flipside</a>, a smart, new open source technology shop based in Lisbon.</p> <p>The site takes a very thoughtful approach to opening information. All the data powering the Climatescope site is available through an open API, which you can easily integrate into your own applications. The <a href="http://global-climatescope.org/en/download/">full dataset</a> is also available for analysis. On almost every page lives a download button that provides a CSV file containing whatever you happen to be viewing.</p> <p>Most importantly, FOMIN got the licensing right. The data is licensed <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY</a>. It can be used (with attribution) by anyone, even for commercial purposes. This is critical when you want data to encourage commercial activity. Moreover the <em><a href="https://github.com/climatescope/climatescope.github.io">website itself</a></em> is also open and is licensed GPL 3.0. The entire site can be forked by other open source projects.</p> <h3 id="dynamic-static-websites">Dynamic Static Websites</h3> <p>Like many of the sites we (and Flipside) build these days, Climatescope is a fully interactive site without a database or a heavy CMS. Climatescope users can manipulate, interrogate, and download the data on any device and in low bandwidth requirements. The site uses Jekyll, Angular, and D3 (among <a href="http://demo.flipside.org/climatescope/humans.txt">other tools</a>) and is hosted on Github. <a href="http://www.developmentseed.org/blog/2012/07/27/build-cms-free-websites/">Read more on our approach to CMS-free websites</a>.</p> <h3 id="customized-weighting">Customized weighting</h3> <p>People have different priorities when evaluating the environment for clean energy. The site is designed for a range of users, from activists to journalists, politicians, environmentalists, and the curious. FOMIN is committed to giving Climatescope users full control over how much weight each metric carries. To accomodate this, we built simple, intuitive sliders. Movement in one slider spreads the difference evenly across the other three factors. You can lock any slider to make it easier to hit an exact breakdown.</p> <p><img src="http://i.imgur.com/3Soa6sf.gif" alt="" /></p> <h3 id="hacking-for-the-planet">Hacking for the Planet</h3> <p>Have some data or coding skills? Care about the planet? Consider joining an <a href="http://ecohack.org/">EcoHack</a> near you on Nov 15-16. We are hosting <a href="https://nvite.com/EcoHack/b41">the DC EcoHack</a> with <a href="http://www.wri.org/">WRI</a>. EcoHacks are also happening in Sydney, Cambridge, New York, Madrid, and San Francisco.</p> 2014-10-28T13:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/28/climatescope Howdy Dan McCarey http://www.developmentseed.org/blog/2014/10/27/howdy-dan <p><a href="http://www.developmentseed.org/team/dan-mccarey/">Dan McCarey</a> has joined Development Seed. Dan is going to help us to turn complex data into compelling stories. Dan is an information designer and web developer. He <a href="http://roadskillmap.com/">builds</a> <a href="http://childlivesmap.com/">powerful</a> <a href="http://data.worldjusticeproject.org/">websites</a>. Dan is passionate about leaving the world better than he found it. That passion has drawn him to live and work in Nepal and Sudan.</p> <p><img src="https://farm6.staticflickr.com/5605/15459451990_7404046669_b.jpg" alt="dan" /></p> <p>Dan created the interactive “Mapping Cholera: A Tale of Two Cities”, which <a href="http://www.scientificamerican.com/article/mapping-cholera-a-tale-of-2-cities-interactive/">recently appeared in Scientific American</a>. This stunning interactive uses historical data and maps to track the spread of cholera in New York in 1832 and compares that to the spread of cholera in Haiti today.</p> <p><a href="http://choleramap.pulitzercenter.org/#fullscreen"><img src="https://farm6.staticflickr.com/5602/15642746681_895790ac3a_o.gif" alt="choleramap" /></a></p> <p>Follow Dan on <a href="twitter.com/danmccarey">Twitter</a> and <a href="github.com/dmccarey">Github</a>.</p> 2014-10-27T17:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/27/howdy-dan Giving context to open spending data http://www.developmentseed.org/blog/2014/10/02/satellite-provides-historical-context <p>Yesterday the government of Mexico launched <a href="http://datos.gob.mx">datos.gob.mx</a> to open up government data across all ministries. We <a href="http://www.developmentseed.org/blog/2014/10/01/mexico-open-data-push/">built a mapping tool</a> for ministries to quickly build rich maps from data on the site. The tool also makes it easier to combine government data with other open datasets. This provides context and meaning to complex government data.</p> <p>The first dataset that we mapped was all <a href="http://mxabierto.github.io/mapas/map/fonden-desastres-2013/">2013 funds for disaster response and reconstruction</a>. The map plots thousands of reconstruction projects across 45 natural disasters, including Hurricane Manual and Ingrid which affected two-thirds of Mexico, killing 192 people and causing $75 billion pesos in damage.</p> <p>This is an incredibily rich and complex dataset. But this data alone is not particularly helpful. We need to better understand the context to understand why government invested funds the way that it did, evaluate the effectiveness of these investments, and plan for future events.</p> <p>To help understand the context we pulled open satellite data from the days following Hurricane Ingrid to better understand the extend of flooding caused by the Hurricane. We used landsat-util to download the data, and produced a <a href="https://gist.github.com/YKCzoli/5a66f639bb79909d3934">false color composite</a> to highlight water.</p> <p><a href="https://www.flickr.com/photos/developmentseed/15421970125"><img src="https://farm4.staticflickr.com/3928/15421970125_835c2fbee3_c.jpg" alt="Datos.gob.mx" /></a></p> <p>The <a href="http://mxabierto.github.io/mapas/map/event-imagery/">resulting map</a> shows reconstruction projects in the context of the flooding.</p> <p>View the government of Mexico’s <a href="https://github.com/mxabierto/mapas">map building tool on github</a>. We used landsat-util to get the imagery and processed it with <a href="https://gist.github.com/YKCzoli/5a66f639bb79909d3934">this script</a>.</p> 2014-10-02T15:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/02/satellite-provides-historical-context Thanks for Having Us, ONA http://www.developmentseed.org/blog/2014/10/02/bye-bye-ona <p>This past weekend Jue and I presented at the <a href="http://ona14.journalists.org/">Online News Association Conference</a> in Chicago. We <a href="http://lieu.io/talks/">shared what we’ve learned</a> from creating mapping sites that toggle through a high volume of data: specifically, how we mapped tens of thousands of rows of data on a series of maps using a mostly-front-end stack.</p> <p>We built this data browser with <a href="http://backbonejs.org/">Backbone</a> and <a href="leafletjs.com/">Leaflet</a>. Given the amount of data, we incorportated data binding and drawing methods from <a href="http://d3js.org/">D3.js</a> on top of Leaflet to boost the loading and interaction speed of the site. By writing python code to output configuration and data json files, we made the site modular. We customized all the geography files into efficient topojson – some with necessary simplications to control file size, and others with customized data field to be joined with the rest of datasets in the front-end. From this experience we learned:</p> <ul> <li><strong>The good</strong>: we made a site that is hosted on Github. Even though the site handles tons of data, it will rarely go down. Topojson makes geography in the browser faster than ever.</li> <li><strong>The crazy</strong>: we spent a lot of brain cells preparing the data and its metadata in python, in order to unload the javascript from heavy calculations.</li> </ul> <p>Like other problems we encounter at Development Seed, we try to tailor our approach using the most fitting technology. We are constantly assessing our tech stacks, which prepares us for challenges that come in all shapes and sizes. Although we’ve managed the complexity of this project well, we are also excited about geographic databases such as <a href="http://postgis.net/">PostGIS</a>, which we may plug in the next time we come across a project of this size.</p> <p>I also did a hands-on session about designing maps using d3 and topojson. We provided several examples to help those starting to learn this workflow. You can <a href="https://github.com/dereklieu/talks/tree/20140926-topojson">find the talk here</a>. We’ve found this workflow to be valuable, since it allows us to serve complex sites as static files. This is consistent with our approach to <a href="http://www.developmentseed.org/blog/2012/07/27/build-cms-free-websites/">building interactive CMS-Free sites</a>.</p> <p>We hope you find our presentations useful. Let us know if you have questions or comments.</p> 2014-10-02T11:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/02/bye-bye-ona Major Open Data Push by the Mexican Government http://www.developmentseed.org/blog/2014/10/01/mexico-open-data-push <p>The Mexican Government is investing heavily in open data to directly make government more effective and the country more productive. Today, kicking off the <a href="http://condatos.org/">regonal open data gathering in Latin America - ConDatos</a>, the Government of Mexico presented <a href="http://datos.gob.mx/">datos.gob.mx</a> a massive data portal with open public data from across the Government.</p> <p><img src="https://cloud.githubusercontent.com/assets/306782/4483024/9f8ffc60-49ad-11e4-9cfa-92327a1f0ad6.png" alt="datos gob mx" /></p> <p>Data must be accessible to be useful in driving innovation and participation. Datos.gob.mx addresses accesibility in two ways. First, all data is machine-readable and searchable, and so is the metadata about those datasets. A <a href="http://ckan.org/">CKAN data portal</a> provides data in bulk download and via an API. Second, Datos.gob.mx put a heavy emphasis on stories and tools that turn raw data into insight. Storytelling tools make the data immediately accessible and understandable to both citizens and policymakers.</p> <h3 id="mapping-open-data">Mapping open data</h3> <p>We worked with the Office of the President of Mexico to build a mapping tool that integrates directly with datos.gob.mx to provide rich storytelling ability. The President’s Office worked with the Civil Protection Service to map all 2013 funds for disaster response and reconstruction. The map plots thousands of reconstruction projects across 45 natural disasters, including Hurricane Manual and Ingrid which affected two-thirds of Mexico, killing 192 people and causing $75 billion pesos in damage.</p> <p><img src="https://cloud.githubusercontent.com/assets/306782/4482853/48dfd6d4-49ac-11e4-9fe0-a196f0405287.png" alt="datos gob mx fondo de desastres naturales 2013 7" /></p> <p>The mapping tool allows ministries to quickly stand up a rich interactive map off of any dataset on datos.gob.mx through a single page of markdown. The map generation tool anticipates many of the way in which ministires will want to aggregate and display information, while also making it easy for advanced users to develop more sophisticated visualizations.</p> <p>We leaned on Jekyll for the map templating ability and mapbox for base layers. Datasets are pulled in over the CKAN API and rendered in real time. All the code for the map generation tool is open source, on github, and available to other governments interested in mapping open data.</p> 2014-10-01T15:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/10/01/mexico-open-data-push Development in the Time of Climate Change http://www.developmentseed.org/blog/2014/09/19/global-resilience-partnership-launch <p>International development is getting harder. Climate change, population strains, and conflict over resources <a href="https://twitter.com/ianschuler/status/512621211797704705">threaten to undo</a> many of the gains made toward the Millenium Development goals. Doing development right means looking outside of country and sector silos and looking to the bigger picture. Today Secretary Kerry announced the launch of the <a href="http://globalresiliencepartnership.org">Global Resilience Partnership</a>, a new partnership to address climate and population change through more coordinated and smarter action. </p> <p><img src="https://farm6.staticflickr.com/5579/15102094109_7c8faac994_o.jpg" alt="" /></p> <p>Achieving the vision of the Global Resilience Partnership will require fluid, fast, and open information that supports coordination and decsionmaking. We are proud to be working with the Global Resilience Partnership on building a data and technology infrastructure to support new ways of addressing global stresses and shocks. Whether it is building tools that connect food security workers with conflict mitigation experts, <a href="http://www.developmentseed.org/blog/2014/09/15/flood-mapping/">analyzing and opening satellite imagery after a flood</a>, or helping municipal governments analyze complex data sets, Development Seed is excited to be part of a powerful approach to solving global challenges, forged on openess and collaboration. </p> <p>The first part this effort will be a global collaborative design challenge. <a href="http://www.globalresiliencepartnership.org/">Check out the site</a> for more information on the Global Resilience Challenge. If you prefer viewing the source, you can find <a href="https://github.com/globalresiliencepartnership/globalresiliencepartnership.github.io">all the code on GitHub</a>. </p> <p><a href="http://www.globalresiliencepartnership.org/"><img src="https://farm4.staticflickr.com/3876/15265816896_4b264475bc_o.png" alt="" /></a></p> <p><em>Photo credit: Melissa Hough</em></p> 2014-09-19T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/09/19/global-resilience-partnership-launch Flood Monitoring with Satellites http://www.developmentseed.org/blog/2014/09/15/flood-mapping <p>Last week monsoon rainfall caused <a href="http://www.nytimes.com/2014/09/09/world/asia/hundreds-dead-in-flooding-in-india-and-pakistan.html?_r=1">flooding in areas across India and Pakistan</a>. <a href="http://www.nytimes.com/2014/09/11/world/asia/floods-india-pakistan.html">Srinagar</a> experienced severe flooding and relief efforts are now underway. Satellite imagery offers a first response look at the flooded area and shows when flood lines increase or recede.</p> <p>We <a href="http://www.developmentseed.org/blog/2014/08/29/landsat-util/">recently released landsat-util</a>, a tool for easily processing open Landsat imagery. Using landsat-util and some additional processing we can quickly process imagery to view flood extent from above.</p> <iframe width="100%" height="500px" frameborder="0" src="http://devseed.com/labs/Srinagar/SrinagarTC.html"></iframe> <p><em>Swipe between August 25 and September 10 Landsat images for a quick first glance at the geographic extent of the flood. <a href="https://www.mapbox.com/labs/swipe/?devseed.map-irjso2a8,devseed.SrinagarBeforeTC&amp;devseed.map-irjso2a8,devseed.Srinagar#10/34.0407/74.7070">View larger</a></em></p> <p>We used <a href="https://github.com/developmentseed/landsat-util">Landsat-util</a> to identify, download, and process Landsat imagery before and immediately after the floods. (The left image is Scene LC81490362014237LGN00 from August 25, 2014. The right image is Scene LC81490362014253LGN00 from September 10, 2014.)</p> <p>With additional processing we can get a clearer view of the floodlines. Water reflects infrared light diffirently than land. We can use this to clearly distinguish muddy water from muddy land. Working from the images we just downloaded, we created a false color composite by combining different near-infrared and mid-infrared bands (also known as a 5,6,4 band composite). The processed image clearly highlights flood lines.</p> <iframe width="100%" height="500px" frameborder="0" src="http://devseed.com/labs/Srinagar/SrinagarFC.html"></iframe> <p><em>A 5,6,4 false color image clearly distinguishes waters from land to derive a flood line. <a href="https://www.mapbox.com/labs/swipe/?devseed.map-irjso2a8,devseed.SrinagarFCBefore&amp;devseed.map-irjso2a8,devseed.SrinagarAfterFC#10/34.0407/74.7070">View larger</a></em></p> <p><strong>First response</strong></p> <p>Timely response is vital in disaster scenarios. When ground information is limited, satellite imagery can provide first responders with a clear picture of an area. With landsat-util we hope to make landsat imagery accessable to more organizations. For more on image processing, see <a href="https://www.mapbox.com/tilemill/docs/guides/landsat-8-imagery/">this great tutorial</a> by the Mapbox satellite team and view the code for the false color comparison <a href="https://gist.github.com/YKCzoli/5a66f639bb79909d3934">here</a>.</p> 2014-09-15T14:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/09/15/flood-mapping Power tools for Satellite Imagery http://www.developmentseed.org/blog/2014/08/29/landsat-util <p>Our love affair with Landsat <a href="https://farm9.staticflickr.com/8508/8466078950_0884854e1d_b.jpg">is</a> <a href="http://www.developmentseed.org/blog/2014/06/26/processing-iraq-landsat/">well</a> <a href="https://twitter.com/nasa_landsat/status/504716315979165698">documented</a>. Today we are sharing the Landsat love with <a href="https://github.com/developmentseed/landsat-util">landsat-util</a>, a command line utility that makes it easy to search, download, and process Landsat imagery. We hope these tools help NGOs, small government agencies, and researchers to benefit from open satellite data.</p> <p><img src="https://farm4.staticflickr.com/3901/14889713977_401169a3cb_o.png" alt="" /></p> <p>The Landsat Program has provided continuous imagery of the earth to the public since 1972. The newest Landsat satellite, Landsat-8, has sophisticated sensors like thermal infrared, which we use to <a href="http://www.developmentseed.org/blog/2014/06/25/iraq-oil-fire/">detect fires</a>, and near infrared, which we use to <a href="http://www.developmentseed.org/blog/2014/07/08/NK_NDVI/">measure vegetation health</a>. Landsat-8 has collected nearly two petabytes of open imagery data. This is an incredibly powerful data source for NGOs, researchers, municipal governments, and government agencies in developing countries. It is useful for everything from urban planning to detecting the effects of climate change.</p> <p>Landsat data is still difficult and time consuming to work with. The same NGOs and small government agencies that stand to benefit most from Landsat data often lack the specialized technical expertise to process it. Over the past few months we’ve built tools to automate our own work with satellite imagery. It once took us all day to collect, georeference, composite, color correct, and pan-sharpen imagery. Now, we can do it in a matter of minutes. We’ve packaged our processing scripts into a command line utility. <a href="https://github.com/developmentseed/landsat-util">Landsat-util</a> makes it easier for other developers and organizations to work with open satellite imagery.</p> <p>Landsat-util does three things well:<br /> - It searches loads of Landsat metadata,<br /> - It makes downloading easier,<br /> - It processes the data, with natural color-correction and pansharpening, and gets it ready for use in Mapbox Studio or your tool of choice.</p> <h3 id="searching">Searching</h3> <p>Using our <a href="https://github.com/developmentseed/landsat-api">landsat-api</a>, you can search all Landsat-8 metadata and find the images you are looking for. You can limit your search to specific date ranges, filter by cloud coverage, and look within specific rows and paths.</p> <p>Landsat-util also makes it easier to find the imagery for a specific area. You can point it to a local shapefile and landsat-util selects all images that cover your shapefile. If you give a country, landsat-util selects all images that cover that country.</p> <h3 id="downloading">Downloading</h3> <p>Landsat-util uses imagery from Google Storage to download results faster than <a href="http://earthexplorer.usgs.gov/">USGS Earth Explorer</a>. Google, in partnership with USGS and NASA, stores Landsat imagery on its <a href="https://earthengine.google.org">Google Earth Engine</a> servers and offers them to the public for free. Landsat-util automatically downloads all of the SceneIDs that fit your search.</p> <h3 id="processing">Processing</h3> <p>Landsat-util can do much of the processing required to make Landsat images useful in your project. It generates natural color images that are ready to be used on mapping tools such as TileMill and Mapbox Studio. All images are adjusted for quality, color, and contrast, and have incredible details (pansharpening increases pixel resolution 2x). They are WGS84 Web-Mercator (EPSG: 3857) georeferenced and can easily be added as a layer to web-based maps.</p> <h3 id="the-power-of-the-command-line">The power of the command line</h3> <p>If you know exactly what you are looking for you can search, download, color-correct, and pansharpen all with one command.</p> <pre><code>landsat search --download --imageprocess --pansharpen --cloud 4 --start "january 1 2014" --end "january 10 2014" pr 009 045 </code></pre> <iframe width="100%" height="600px" frameborder="0" src="https://a.tiles.mapbox.com/v4/devseed.map-irjso2a8,devseed.TCI.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#10/21.606/-71.959"></iframe> <p><em><a href="https://a.tiles.mapbox.com/v4/devseed.map-irjso2a8,devseed.TCI.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#10/21.606/-71.959">Turks &amp; Caicos Islands, British West Indies</a></em></p> <p>You can preview images before you download. Search commands provide a link to a thumbnail for each image.</p> <pre><code>landsat search --cloud 4 --start "August 1 2013" --end "August 25 2014" country 'Vatican' </code></pre> <p>Using the <code>--pansharpen</code> flag will take longer to process but will produce clearer images.</p> <pre><code>landsat search --download --imageprocess --pansharpen --cloud 4 --start "august 11 2013" --end "august 13 2013" pr 191 031 </code></pre> <iframe width="100%" height="600px" frameborder="0" src="https://www.mapbox.com/labs/swipe/?devseed.map-irjso2a8,devseed.raster_pan&amp;devseed.map-irjso2a8,devseed.Rome_no_pan#13/41.8304/12.4717"></iframe> <p><em><a href="https://www.mapbox.com/labs/swipe/?devseed.map-irjso2a8,devseed.raster_pan&amp;devseed.map-irjso2a8,devseed.Rome_no_pan#13/41.8304/12.4717">Vatican/Rome, Italy - slide between images to check out pansharpening in action</a></em></p> <p>You can also perform all processing on images that you previously downloaded.</p> <pre><code>landsat download LC81050682014217LGN00 landsat process --pansharpen /your/path/LC81050682014217LGN00.tar.bz </code></pre> <iframe width="100%" height="600px" frameborder="0" src="https://a.tiles.mapbox.com/v4/devseed.map-irjso2a8,devseed.gurig_national_park.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#9/-11.6078/132.2328"></iframe> <p><em><a href="https://a.tiles.mapbox.com/v4/devseed.map-irjso2a8,devseed.gurig_national_park.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#9/-11.6078/132.2328">Gurig National Park, Australia</a></em></p> <pre><code>landsat download LC82310622014187LGN00 landsat process --pansharpen your/path/landsat/zip/LC82310622014187LGN00.tar.bz </code></pre> <iframe width="100%" height="600px" frameborder="0" src="https://a.tiles.mapbox.com/v4/devseed.map-irjso2a8,devseed.manaus.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#10/-3.071/-60.316"></iframe> <p><em><a href="https://a.tiles.mapbox.com/v4/devseed.map-irjso2a8,devseed.manaus.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#10/-3.071/-60.316">Manaus, Brazil</a></em></p> <p>For more possibilities, <a href="https://github.com/developmentseed/landsat-util/blob/master/README.rst">check out the documentation.</a></p> <h3 id="limitations">Limitations</h3> <p>Landsat-util uses a number of image processing tools that are very powerful but also very resource hungry. The image processing functions consume a good amount of memory (RAM) and it might not work on computers that have less than 6GB of ram. Using landsat-util also requires some other applications and libraries such as GDAL, ImageMagick and Orfeo Toolbox.</p> <p>Mac users can install landsat-util and all dependencies through a simple brew command. We have provided a walkthrough for Ubuntu users. For other systems we provide a list of required dependencies.</p> <h3 id="open-source">Open Source</h3> <p>Landsat-util helps us in our own satellite imagery work and we believe could help others run smarter, faster and better analysis and research using satellite products. Let us know what you think and contribute to the repo.</p> 2014-08-29T16:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/08/29/landsat-util Exploring space faster http://www.developmentseed.org/blog/2014/08/21/sneak-peek-landsat-util <p>Exploring events from space is going to get a little easier. Below is a sneak peek of a simple and smart utility we’re working on to save ourselves some time searching, downloading, and processing Landsat imagery. </p> <p><img src="https://farm6.staticflickr.com/5556/14806387597_ae0241c1f6_c.jpg" alt="" /><br /> <em>Turks &amp; Caicos (path: 09, row: 045, id: <a href="http://api.developmentseed.com:8000/landsat?search=sceneID:LC80090452014008LGN00">LC80090452014008LGN00</a>)</em></p> <p>We regularly use satellite imagery to better understand global events. Landsat-8 is a satellite we love because it is open data that is regularly updated. We think this tool will help other developers and organizations to work with open satellite data. Check out below a few of our other favorite spots we’ve processed through the tool. </p> <p><img src="https://farm6.staticflickr.com/5596/14989636351_e70d6b036b_b.jpg" alt="" /><br /> <em>Palestine and Israel (path: 174, row: 038, id: <a href="http://api.developmentseed.com:8000/landsat?search=sceneID:LC81740382014188LGN00">LC81740382014188LGN00</a>)</em></p> <p><img src="https://farm6.staticflickr.com/5559/14992358502_e0068eab59_b.jpg" alt="" /><br /> <em>Yamal Peninsula, Siberia (path: 168, row: 010, id: <a href="http://api.developmentseed.com:8000/landsat?search=sceneID:LC81680102014178LGN00">LC81680102014178LGN00</a>)</em></p> <p><img src="https://farm6.staticflickr.com/5563/14806188247_0b8207d748_c.jpg" alt="" /><br /> <em>Amazon Delta, Brazil (path: 225, row: 060, id: <a href="http://api.developmentseed.com:8000/landsat?search=sceneID:LC82250602013174LGN00">LC82250602013174LGN00</a>)</em></p> <p><img src="https://farm4.staticflickr.com/3840/14989636271_16fa6d6f91_c.jpg" alt="" /><br /> <em>Karachi, Pakistan (path: 153, row: 43, id: <a href="http://api.developmentseed.com:8000/landsat?search=sceneID:LC81520432014018LGN00">LC81520432014018LGN00</a>)</em></p> 2014-08-21T16:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2014/08/21/sneak-peek-landsat-util