Drupal - Development Seed http://www.developmentseed.org Posts related to Drupal. en SatSummit Highlights http://www.developmentseed.org/blog/2017/02/21/SatSummit-Highlights <p class="dropcap">At SatSummit, 231 global development and satellite industry experts joined us to map how we can do smarter development work with satellite data. The major satellite data providers (including NASA, ESA, DigitalGlobe and Planet) joined big data gurus (from AWS, IBM, and Orbital Insights) and dozens of global development organizations (including USAID, World Bank, Humanitarian OpenStreetMap Team, and World Resources Institute) for a truly inspiring day of discussion.</p> <h2 id="product-launches">Product Launches</h2> <p>SatSummit has several noteworthy launches:</p> <ul> <li>DigitalGlobe launched a new <a href="https://www.digitalglobe.com/opendata">Open Data Program</a> focused on data for disaster response. Planet also debuted their <a href="https://www.planet.com/disasterdata/">Disaster Data</a> open data platform.</li> <li>Astro Digital, Amazon Web Services (AWS), and Development Seed launched <a href="https://www.planet.com/disasterdata/">MODIS on AWS</a> providing daily snapshots of the earth.</li> <li>The Humanitarian OpenStreetMap Team (HOT) launched the official (non-beta) version of <a href="https://openaerialmap.org/">OpenAerialMap</a>.</li> <li><a href="https://www.satellogic.com/">Satellogic</a> committed to provide free global 30-meter resolution hyperspectal imagery for scientific and humanitarian purposes.</li> <li>SatSummit was a great coming out party for Open Imagery Network (OIN), a new effort to support the use of open imagery by the development community. OIN emerged from research around last SatSummit, and is supported by the Gates and Omidyar Foundations.</li> </ul> <p>The SatSummit website has <a href="https://satsummit.io/insights/2017/02/03/launches/">a full list of launches and commitments</a> from the Summit.</p> <h2 id="deep-thoughts">Deep Thoughts</h2> <p>I left SatSummit with many new ideas and some new concepts. The stickiest concepts permeated the discussion throughout the day and many conversations since.</p> <h3 id="geodiversity">Geodiversity</h3> <p><a href="https://twitter.com/ericg">Eric Gundersen</a> kicked off the day reminding us that this diversity is our strength. An extremely diverse set of people are behind our satellite launches, data analysis, and bringing this knowledge to global development. Satellite and remote sensing is a human, global endeavor and does not discriminate. We need to respect and defend our diversity and be vocal in the face of challenges to our diversity.</p> <p>#Geodiversity took on a life through the Summit from celebrating our diverse teams, to diverse imaging approaches, to using different sensors work in combination to give us deeper insights and flexible operation.</p> <h3 id="fat-data">Fat data</h3> <p>Satellite data isn’t just big, it’s fat. <a href="https://twitter.com/bronwynagrios">Bronwyn Agrios</a> argued that imagery files are bulky and big in ways that create unique challenges for people who archive that data, run analysis across that data, or distribute imagery on the web.</p> <h3 id="data-access">Data Access</h3> <p>Your pixels don’t matter if they aren’t in front of people who need them, when they need them, and how they need them. The Distribution Panel flipped the question of distribution to one of Access and recognized it as a human design problem as much as a technical challenge.</p> <h3 id="apis-over-portals">APIs over Portals</h3> <p>Powering the data revolution means distributing data to be accessed by computers. We need to move away from portals toward APIs and data in predictable file structures. As Bronwyn Agrios emphasized, if you love your data then you’ve got to “put an API on it”. This gives you the flexibility to build the UI that your users will love.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/satsummit-browyn.jpg" /> <figcaption>Bronwyn Agrios telling folks to "put an API on it."</figcaption> </figure> <p>What did I miss? I wasn’t able to get to every session. If you have other reflections to share, please send them to info@satsummit.io and we will post them on <a href="https://satsummit.io/insights/">SatSummit Insights</a>.</p> 2017-02-21T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/02/21/SatSummit-Highlights DC's Bold Family Leave Policy http://www.developmentseed.org/blog/2017/02/17/paid-family-leave <figure class="align-center"> <img alt="DJ Thomas at the controls" src="/assets/graphics/content/dj-thomas-card.jpg" /> </figure> <p class="dropcap">We are delighted that <a href="http://dcist.com/2017/02/paid_family_leave_passes.php">a bold family leave policy has moved forward in DC</a>. The policy makes it easier for Development Seed to do right by our team and all the great folks we work with in DC, from our suppliers and vendors, to the people who treat us to heavy pours at happy hour.</p> <p>As a small business, sharing our responsibility to our employees with other DC businesses lets us offer more aggressive leave policies to our team. A flat tax allows us to do that. There is still work to be done. Congress has 30 days to challenge the law, and there are implementation details to work out. In the meantime, we are proud of DC for taking this big step.</p> 2017-02-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/02/17/paid-family-leave Fed Mapping Happy Hour http://www.developmentseed.org/blog/2017/02/05/open-mapping-happy-hour <p class="dropcap">This upcoming Monday we’ll be hosting a Fed Mapping Happy Hour, our way of saying thank you to the hard working geographers and data folk in our Federal Government.</p> <p>Come anytime between 6:00 to 8:00pm for the free drinks and snacks, stay a while for a quick round of lightning talks from our team and our open source friends in town.</p> <p><a href="https://openmappinghappyhour17.splashthat.com/">RSVP here</a> so we have enough brews on hand.</p> <p><a href="https://openmappinghappyhour17.splashthat.com/"><img src="/assets/graphics/content/prose/OpenMappingHappyHour.jpg" alt="" /></a></p> 2017-02-05T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/02/05/open-mapping-happy-hour Open Source Machine Learning http://www.developmentseed.org/blog/2017/01/30/machine-learning-learnings <p class="dropcap">Tomorrow, I will be at <a href="https://satsummit.io/">SatSummit</a> talking about Development Seed’s machine learning work and what we’ve learned over 18 months of building open source tools for imagery analysis. We are building <a href="https://github.com/developmentseed/skynet-train">open source machine learning tools</a> and open algorithms that can be shared and improved by academics and implementers without expensive software or license fees.</p> <p>Machine learning has tremendous potential to improve the work of development organizations. We use machine learning to create smart algorithms to analyze satellite imagery to determine land use or to identify features like roads and buildings. Automating imagery analysis allows us to more quickly map unmapped areas or to monitor vast crop or forest areas and alert us to anomalies.</p> <p>But what does it require to take advantage of machine learning in your work? Here are a few things we’ve learned about using machine learning to identify features like roads and buildings.</p> <h3 id="for-road-detection-use-one-meter-resolution-imagery-or-better-multispectral-imagery-can-improve-results">For road detection use one meter resolution imagery or better. Multispectral imagery can improve results.</h3> <p>We’ve done land use classification and vegetation analysis on all manner of data, from 30 meter to less than one-meter resolution. For many of these applications, higher resolution imagery doesn’t significantly improve results, and can even add noise in some cases. For road and building detection, we’ve had greatest success with imagery that is one-meter resolution or better.</p> <p>Bringing in infrared bands can better distinguish plants from built up areas. It’s helpful for identifying roads in certain contexts, particularly in suburban neighborhoods with lots of trees.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-nir-true-annotate.png" alt="NIR and True Color Imagery, Machine Learning Output" /> <figcaption>left column: input images (infrared, true color), &copy; <a href="https://www.digitalglobe.com/" target="_blank">DigitalGlobe</a>; middle: <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; right: our model prediction</figcaption> </figure> <h3 id="you-need-at-least-100-square-kilometers-of-high-quality-training-data-for-feature-extraction">You need at least 100 square kilometers of high quality training data for feature extraction</h3> <p>Machine learning requires a lot of good training data to produce accurate results. Groups often ask, “How much training data is enough?” We ran some experiments to determine that.</p> <p>For the road detection work that <a href="https://developmentseed.org/blog/2016/08/17/sotm-skynet/">we demonstrated at State of the Map</a> we used 559 square kilometers of training data. We experimented with using gradually less data (372km<sup>2</sup>, 186km<sup>2</sup>, 93km<sup>2</sup>, 47km<sup>2</sup>) to determine at what point the accuracy deteriorated. For less dense mapping across a consistent area, sometimes ~100km<sup>2</sup> of high quality training data produced accurate results on roads. (That’s about the size of Sacramento. Or a little less than Washington DC’s NW and NE quadrants combined)</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-buckets-1.png" alt="Seattle Streets, Machine Learning Output" /> <figcaption>left-to-right: input image, &copy; <a href="https://www.mapbox.com/" target="_blank">Mapbox Satellite</a>; <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; columns 3-6: our model predictions with varying amounts of input data</figcaption> </figure> <p>For denser areas, or when trying to create a model which can make predictions over a larger variety of areas, more input data helps.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-buckets-2.png" alt="Seattle Streets, Machine Learning Output" /> <figcaption>left-to-right: input image, &copy; <a href="https://www.mapbox.com/" target="_blank">Mapbox Satellite</a>; <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; columns 3-6: our model predictions with varying amounts of input data</figcaption> </figure> <p>For buildings, we started with the same 559km<sup>2</sup> baseline for training data and saw promising results but it was tough to get clearly defined edges. We tried to limit our inputs to places where we felt very confident about the training data. Because this limitation reduced the input data to under 100km<sup>2</sup>, we didn’t see good improvement. Jump in <a href="https://github.com/developmentseed/skynet-train">the repo</a> if you have some ideas for how to improve!</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-buildings-1.png" alt="Dar es Salaam Buildings, Machine Learning Output" /> <figcaption>left-to-right: input image, &copy; <a href="https://www.mapbox.com/" target="_blank">Mapbox Satellite</a>; <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; our updated model with less input data; our first attempt </figcaption> </figure> <h3 id="training-a-model-can-take-a-few-days-but-once-its-trained-you-can-apply-it-in-real-time">Training a model can take a few days. But once its trained you can apply it in real time.</h3> <p>We’ve had most success running about 100 thousand iterations of our model. With one AWS <code>g2.2xlarge</code> instance that takes around 4 days. This is mostly something that can run in the background, but occasionally you’ll need to kick the server.</p> <p>Once the machine learning process has produced a algorithm, you can apply it to additional imagery in real time. We have set up apps that conduct analysis of imagery in real time as you browse.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-tile-server.gif" alt="Manilla Roads, Machine Learning Live Prediction" /> <figcaption>Live prediction of roads in Manilla, Philippines</figcaption> </figure> <p>While the results aren’t always perfect, they are pretty good. They can be used to give an immediate idea of where to spot features or to run a quick calculation on approximately how many buildings or kilometers of road fall into an administrative region or disaster area.</p> <h3 id="open-works-well-for-development-work">Open works well for development work</h3> <p>We’ve benefited greatly from using open tools for our work and by sharing our imagery processing pipeline with others. We love collaborating with people with interesting problems to solve or ingenious solutions we haven’t considered. You can check out the <a href="https://github.com/developmentseed/skynet-train">code on Github</a> or find me at SatSummit to chat.</p> 2017-01-30T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/30/machine-learning-learnings All of NASA's Earth Science Data on the Cloud http://www.developmentseed.org/blog/2017/01/30/NASA-on-the-cloud <p class="dropcap">No one processes and provides more earth observation data than NASA. NASA provides access to more than 17.5 petabytes of earth observation data across more than 11,000 unique data products from its satellite and aerial missions. <a href="https://earthdata.nasa.gov/nasa-data-policy">In FY 2016, NASA delivered more than 1.51 billion earth science data products to more than 3 million data users around the world</a>. Managing this data is a huge task. We are working with NASA’s Earth Science Data System (ESDS) Program to build Cumulus, a cloud-based prototype to ingest, process, catalog, archive, and distribute NASA’s Earth Data streams.</p> <p>Cumulus is one of a suite of tools that NASA is developing to better leverage cloud computing for data storage, processing, and access. We are working closely with NASA’s ESDS team as well as several of the Distributed Active Archive Centers (<a href="https://earthdata.nasa.gov/about/daacs">DAACs</a>) that ensure Earth Data missions are available and accessible to users. We are working with these teams to prototype and refine a system that can be used to process, archive, and publish data as varied as lightning strike datasets and MODIS scenes.</p> <p>Cumulus will be open source. We are working with NASA to transfer ownership of Cumulus to NASA and to provide the code and documentation through their platforms. We will keep you posted when the code is available. We hope that Cumulus will be useful to other organizations building imagery publishing pipelines.</p> 2017-01-30T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/30/NASA-on-the-cloud A imagery pipeline for disaster response http://www.developmentseed.org/blog/2017/01/27/imagery-pipeline <p class="dropcap">We are working with the <a href="http://hotosm.org/">Humanitarian OpenStreetMap Team</a> (HOT) to get the best satellite and drone imagery immediately into the hands of disaster responders. Partnering with HOT, we are building an imagery coordination pipeline. In the event of a disaster, this workflow will track the most pressing needs on the ground, and connect with satellite companies to task their satellites and drone operators to prioritize flights where they are most useful. We also built upload tools to OpenAerialMap so that drone operators can quickly distribute their images.</p> <h2 id="coordinating-imagery-needs">Coordinating Imagery Needs</h2> <p>We’re working together with HOT to better plan and coordinate imagery needs in times of response. Through HOT’s Imagery Coordination Working Group, we’re testing ideas and workflows to make it easier for disaster responders to flip through available pre- and post-disaster imagery and, when nothing meets their need, coordinate imagery acquisition with the wider community and imagery providers. The tool will be open in order to make it easier for local drone operators and GIS units to participate in defining needs and sharing data. Below is a sneak peak of comparing imagery within the tool.</p> <p>HOT will share more at <a href="https://satsummit.io">SatSummit</a> on the ongoing conversations to improve coordination among imagery providers.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/big--imagery-requests-search.gif" /> <figcaption>Humanitarian responders, satellite companies, and drone operators can search for imagery to meet their needs</figcaption> </figure> <h2 id="drone-data">Drone Data</h2> <p>Local drone operators can share their imagery through a <a href="https://upload.openaerialmap.org/">simple user interface</a>. OpenAerialMap’s upload tool is integrated directly with Dropbox and Google Drive. Often the drone operators nearest to a disaster area aren’t experts in processing imagery for sharing on the web. The upload tool makes it quick and easy to share some of the first surveys of a disaster.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/oam-uploader-form.png" /> </figure> <p>This is a very important part of the OAM ecosystem as it enables more people and organizations to submit imagery to the ever-growing catalog of OAM. It allows any surveyor to quickly share fresh imagery with the world under the <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY 4.0</a> making it available to whomever needs to use it. It’s now easier than ever to share new images and help with disaster preparedness.</p> 2017-01-27T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/27/imagery-pipeline SatSummit: Solving Global Issues Through Data http://www.developmentseed.org/blog/2017/01/26/satsummit <p class="dropcap">Next week, Development Seed and Mapbox will host <a href="https://satsummit.io/">SatSummit</a> in Washington, D.C. Satellite industry and global development experts from around the globe will come nerd-out on solving social and environmental challenges through practical applications of satellite data. The big focus this year is turning data into action, with deep dives into machine learning, maintaining climate data streams, famine and malaria mapping, and imagery for humanitarian response.</p> <p>Look for us throughout the day:</p> <ul> <li> <p><a href="https://developmentseed.org/team/ian-schuler/">Ian</a> will moderate a discussion with “The Geographers” from The World Bank and USAID. They’ll tackle tough questions relating to how organizations incorporate remote sesning data into daily decision-making.</p> </li> <li> <p><a href="https://developmentseed.org/team/drew-bollinger/">Drew</a> will share our progress with developing a suite of machine learning tools built entirely in “open”. He’ll show what <a href="https://github.com/developmentseed/skynet-data">Skynet</a> is capable of and our pipeline to use OpenStreetMap as a deep source of training data for deep learning.</p> </li> <li> <p>Behind the camera <a href="https://developmentseed.org/team/dylan-moriarty/">Dylan</a> will capture big moments, while <a href="https://developmentseed.org/team/anna-scalamogna/">Anna</a> and <a href="https://developmentseed.org/team/derek-lieu/">Derek</a> live tweet the event sharing key insights that are discussed. Our friendly face at registration <a href="https://developmentseed.org/team/miles-watkins/">Miles</a> will greet people as they arrive, while <a href="https://developmentseed.org/team/olaf-veerman/">Olaf</a> and <a href="https://developmentseed.org/team/alireza-j/">Alireza</a> roam around to chat up new and old friends.</p> </li> <li> <p>I’ll be there as well making sure things run smoothly.</p> </li> </ul> <p>Our good friends will be there from groups like DigitalGlobe, The World Bank, AWS, Planet, and USAID Global Development Lab. We are also excited to meet new folks and hear new approaches to tackling hard problems. SatSummit is a real opportunity to form unexpected partnerships and think outside the box, and we’re looking forward to what comes out of it.</p> <p>The event is unfortunately booked up. If you haven’t signed up and want to attend, add your name to the waitling list <a href="here">here</a>. We’re in the process of confirming attendees, and tickets may become available. For those unable to join in person, follow along <a href="https://twitter.com/sat_summit">@sat_summit</a> on Twitter.</p> 2017-01-26T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/26/satsummit Development Seed, Annual Report http://www.developmentseed.org/blog/2017/01/05/annual-report <p class="dropcap">Development Seed builds products that draw insight from complex data, helping people make better decisions. We work with non-profits and civic for-profits addressing the worlds biggest challenges. We use open technology, modern cloud infrastructure and thoughtful design practices to quickly build and scale our meaningful data driven tools.</p> <p>In 2016, we built bigger and better products; we reached our biggest audience; and we worked with the largest sets of open data we’ve ever touched. We pioneered new approaches to machine learning and monitoring earth from space. We traveled the world working with governments to track agriculture, electricity and road projects.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/AR-egypt1.jpg" alt="Ali in Egypt" width="1400" height="365" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/AR-India-derek.jpg" alt="Derek in front of Taj Mahal" width="1400" height="491" /> </figure> <p>Development Seed operates as a non-profit. Our work with transparency and development partners supports our open source work, e.g. <a href="https://github.com/sat-utils">sat-utils</a>, <a href="http://prose.io/">prose</a>, <a href="http://devseed.com/dirty-reprojectors-app/">dirty reprojections</a>, and <a href="https://github.com/developmentseed/skynet-train">skynet</a>.</p> <h2 id="blowing-the-roof-off">Blowing the roof off</h2> <figure class="align-center"> <a href="https://www.washingtonpost.com/2016-election-results/us-presidential-race/?utm_term=.f0e61c3a004e"><img src="/assets/graphics/content/prose/AR-wapo.jpg" alt="Washington Post election webpage" width="675" height="540" /></a> </figure> <p>Our <a href="https://developmentseed.org/blog/2016/11/08/post-eday/">project with The Washington Post, building live election maps</a> was our most visible project in 2016. Tens of millions of people have interacted with our maps, along with more than 100 other syndicates in The Washington Post network. On election night, The Washington Post broke records for the number of people on the website. The maps not only withstood the load, but were consistently a minute or more faster than competitor sites. Our maps allowed users to track up-to-the-moment election results, through an easy-to-use interface that worked on all devices, and also gave The Washington Post team the flexibility to highlight the most important coverage as results came in.</p> <figure class="align-center"> <a href="https://www.nasa.gov/"><img src="/assets/graphics/content/prose/AR-NASA_logo.svg" alt="NASA logo" width="294" height="236" /></a> </figure> <p>We are working with NASA (Yes, <em>that</em> NASA :swoon:) to build Cumulus, a cloud-based tool for processing science data from satellites, planes, and ground sensors. NASA currently maintains over 6,100 data streams from sensors in space and on the ground. Cumulus is still in the prototype phase, and we are working closely with NASA and USGS data operators to streamline and centralize processing of data streams from near realtime hurricane data to climate data.</p> <figure class="align-center"> <a href="https://openaerialmap.org/"><img src="/assets/graphics/content/prose/AR-oam.jpg" alt="Open Aerial Map" width="675" height="540" /></a> </figure> <p>We’re helping the development community use satellite data to track and understand our Planet. Our <a href="https://github.com/developmentseed/skynet-train">Skynet</a> tools bring modern machine learning methods to automatically extract features from satellite imagery–such as roads or damaged buildings. We built a powerful earth monitoring platform for Astro Digital, that automatically processes new satellite imagery from NASA and the European Space Agency (ESA) and delivers it straight to your app or inbox. We also expanded <a href="https://openaerialmap.org/">OpenAerialMap</a>, enabling drone operators to more easily contribute data, helping them quickly collect drone imagery after a natural disaster.</p> <figure class="align-center"> <a href="http://oci.carnegieendowment.org/"><img src="/assets/graphics/content/prose/AR-oci.jpg" alt="Oil Climate Index" width="675" height="540" /></a> </figure> <p>We built practical tools to address climate change. We partnered with OpenAQ’s effort to <a href="https://openaq.org/">end air inequality</a>, by collecting and organizing the world’s air quality data. The OpenAQ network, collects critical data to inform research and policy decisions. As of this morning, 34.5 million measurements, from 4,706 locations, in 43 countries have been collected. We also worked with the Carnegie Endowment to build the <a href="http://oci.carnegieendowment.org/">Oil Climate Index</a>, a tool to monitor the climate impact of different oils around the world. OCI allows investors, policymakers, industry, and the public, to see how we can reduce greenhouse gas emissions, <em>without changing consumption levels</em>, by making better decisions about the types of oils we use.</p> <p>And much more. Read about some of <a href="https://developmentseed.org/projects/">our other projects</a>.</p> <h2 id="growing-the-family">Growing the Family</h2> <p>I’m so incredibly proud to be part of this team and happy to announce that this truly impressive group grew by <a href="https://developmentseed.org/blog/2016/01/21/welcome-matt/">three</a> <a href="https://developmentseed.org/blog/2016/02/18/welcome-ali/">team</a> <a href="https://developmentseed.org/blog/2016/11/10/welcome-laura/">members</a> this year.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/AR-biketour.jpg" alt="DS on our brewery bike tour" width="738" height="416" /> <figcaption>Devseed brewery bike tour</figcaption> </figure> <p>I’m also incredibly grateful to the wonderful partners, collaborators, and rabble rousers that we work with. I can’t wait to show off what we are cooking up for 2017.</p> <p>See you at <a href="https://satsummit.io/">SatSummit</a>!</p> 2017-01-05T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/05/annual-report Vote for OpenAQ http://www.developmentseed.org/blog/2016/12/20/vote-for-openaq <p class="dropcap">OpenAQ, a project we helped build to end air inequality, needs your help to continue growing its data. The project is currently one of the six finalists for the <a href="https://www.openscienceprize.org/">Open Science Prize</a>. Winning this prize would allow OpenAQ to expand the geographic coverage of official data sources and architect the integration of low-cost sensor and satellite data. The latter would be an important step toward bridging the data gap that affects places with poor air quality most.</p> <h2 id="air-inequality">Air inequality</h2> <p>Poor air quality is responsible for 1 out of 8 deaths in the world, more than malaria and HIV/AIDS combined. Access to air quality data is fundamental for scientists and policy makers to gain a deeper understanding of the issue, but is often hard to come by.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/openaq-compare.jpg" /> <figcaption>OpenAQ makes it easy to compare air quality around the globe</figcaption> </figure> <p><a href="https://openaq.org">OpenAQ</a> addresses this issue by providing open access to historical and real time air quality data around the globe. The platform currently provides access to over 32 million air quality measurements, and this number continues to grow at the rate of 150 000 a day. Beyond building an open data platform, the OpenAQ team has done a tremendous job building a global community of scientists, journalists and software developers. A community capable of harnessing the data to influence policy and empower the general public.</p> <p>Show you support for OpenAQ <a href="http://event.capconcorp.com/wp/osp/vote-now/">by voting</a> by January 6th.</p> 2016-12-20T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/12/20/vote-for-openaq Dirty Reprojectors http://www.developmentseed.org/blog/2016/12/15/dirty-reprojectors <script src="https://api.tiles.mapbox.com/mapbox-gl-js/v0.28.0/mapbox-gl.js"></script> <link href="https://api.tiles.mapbox.com/mapbox-gl-js/v0.28.0/mapbox-gl.css" rel="stylesheet" /> <style> #map-virtue { height: 600px; margin-bottom: 2rem; } .mapboxgl-ctrl-top-right .mapboxgl-ctrl { margin: 10px 2rem 0 0; } </style> <div class="map-container bleed-full"> <div id="map-virtue"></div> </div> <p><em>The Robinson projection in Mapbox GL</em></p> <p class="dropcap">We’re open sourcing a way to pre-process geometries for use in Mapbox GL, to produce maps that render something besides the standard Web Mercator projection.</p> <p>Almost all web mapping libraries render maps using Web Mercator, making an assumption that you generally can’t change out-of-the-box. This has <a href="https://gist.github.com/tmcw/a3828827c8490f7e13a9">advantages</a>, but it posed a real challenge for us when we set out to build the <a href="https://developmentseed.org/blog/2016/11/08/post-eday/">Washington Post’s live election results map</a>, where using the <a href="http://bl.ocks.org/mbostock/5545680">Albers USA</a> projection was an important requirement. To meet that requirement, we built a pipeline to pre-process geometries.</p> <p>We started with our standard WGS 84 (longitude/latitude) coordinate geojson files and projected those files into our desired projection. Next, we scaled the result to fit within the Web Mercator coordinate system.</p> <figure class="align-center"><img src="/assets/graphics/content/prose/projection-part-1.png" alt="projection" /></figure> <p>We took the scaled files and reverse-projected them back into WGS 84, before converting them to mbtiles and uploading the whole thing to Mapbox Studio.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/projection-part-2.png" alt="projection" /> </figure> <p>We used Mike Bostock’s <a href="https://github.com/d3/d3-geo">d3-geo</a> and <a href="https://github.com/d3/d3-geo-projection">d3-geo-projection</a> libraries to do the heavy lifting on the rather tricky projection math.</p> <p>To show some of the projections you can achieve with this method, we built the <a href="https://www.developmentseed.org/dirty-reprojectors-app/">Dirty Reprojectors App</a>. The site includes the long list of projections d3 comes bundled with, and will also accept and project your own geojson files.</p> <figure class="align-center"> <div class="map-container bleed-full"> <a href="https://devseed.com/dirty-reprojectors-app/" target="_blank"><img src="/assets/graphics/content/prose/projection-examples.gif" alt="projection" /></a> </div> </figure> <p>For greater control over projection variables like parallels and center points, use the <a href="https://github.com/developmentseed/dirty-reprojectors">Dirty Reprojectors CLI</a> instead. There you can tweak the projection code as needed.</p> <h3 id="caveats">Caveats</h3> <p>After reprojecting geometries, the actual longitude/latitude coordinates will not be accurate. This complicates tasks like reverse geocoding.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/projection-part-3.png" alt="Actual coordinates are off" /> </figure> <p>Bounds are also tricky. Depending on the projection, a bounding box (rectangle) in longitutde/latitude coordinates may no longer look like one once it’s reprojected. Functions like map.fitBounds() where the map fills the space of its bounds can take more work to get right. Our work-around was to calculate and save the post-projection bounding box for each US state, instead of using the “real” bounding box coordinates.</p> <p>We’re always looking for novel ways to solve problems. If this helps you solve a problem, we’d love to hear about it. You can always reach out to us on <a href="https://twitter.com/developmentseed">Twitter</a> or <a href="https://github.com/developmentseed">Github</a>.</p> <script> mapboxgl.accessToken = 'pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q'; var map = new mapboxgl.Map({ container: 'map-virtue', style: 'mapbox://styles/devseed/civlnnvyf000y2imxc4xqimcx', center: [0, -10], zoom: 1, minZoom: 1 }); map.addControl(new mapboxgl.NavigationControl()); map.scrollZoom.disable(); var hoverSource = 'composite:hover'; // duplicate the source as a hover source function onStyleLoad (e) { var style = map.getStyle() if (style.sources && style.sources.composite) { map.addSource(hoverSource, Object.assign({}, style.sources.composite)); map.off('style.load', onStyleLoad); } } // once the newly duped hover source is loaded, // reset the map style but alter the hover layer to point to new source // then assign the mousemove listeners function onSourceLoad (e) { if (e.source.id === hoverSource) { var style = map.getStyle() var hover = style.layers.find(function (d) { return d.id === 'Country-Fill-Hover'; }); hover.source = 'composite:hover'; map.setStyle(style); map.on('mousemove', function(e) { var features = map.queryRenderedFeatures(e.point, { layers: ['Country-Fill'] }); if (features.length) { map.setFilter('Country-Fill-Hover', ['==', 'admin', features[0].properties.admin]); } else { map.setFilter('Country-Fill-Hover', ['==', 'admin', '']); } }); map.on('mouseout', function() { map.setFilter('Country-Fill-Hover', ['==', 'admin', '']); }); map.off('source.load', onSourceLoad); } } map.on('style.load', onStyleLoad); map.on('source.load', onSourceLoad); </script> 2016-12-15T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/12/15/dirty-reprojectors Mappers Gather at Development Seed http://www.developmentseed.org/blog/2016/11/18/ds-mapathon <p class="dropcap">The <a href="http://www.redcross.org/">American Red Cross</a>, <a href="http://mapgive.state.gov/">MapGive</a>, and <a href="https://www.usaid.gov/">USAID/OFDA</a> joined our team on Monday night to kickoff GeoWeek in DC. We started the night off with presenters showcasing their work for the OpenStreetMap community. These institutions apply open source and open data sharing for humanitarian response and economic development.</p> <p>Rachel Levine from the American Red Cross began by explaining her role in the community, and used examples of projects people are contributing to around the world. She then spoke about the missing maps project in Malawi.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mapathon.jpg" alt="mapathon" /> </figure> <p>We then heard from Keifer Gonzalez of USAID/OFDA who discussed <a href="https://volcanomapping.github.io/">decreasing volcanic risks in Indonesia</a>, and shared explosive photos of volcanoes. Andrew McKenna from the State Department’s MapGive program talked about U.S. President’s Emergency Plan for AIDS Relief. PEPFAR requests help to inventory buildings and roads for HIV/AIDS prevention and treatment programs in Tanzania.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mapathon-2.jpg" alt="mapathon" /> </figure> <p>We had mappers with all levels of experience join us: active mappers chose the project that resonated with them most and began mapping after presentations concluded; Tom Gertin (MapGive) walked beginners through the ins and outs of contributing to the OSM Community.</p> <p>If you’d like to contribute to mapping these projects, check out the <a href="http://tasks.hotosm.org/">HOT tasking manager</a> for tasks <a href="http://tasks.hotosm.org/project/750">#750</a>, <a href="http://tasks.hotosm.org/project/2296">#2296</a>, <a href="http://tasks.hotosm.org/project/2301">#2301</a>. Happy mapping!</p> 2016-11-18T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/18/ds-mapathon Mapathon at Development Seed http://www.developmentseed.org/blog/2016/11/11/osm-geoweek <figure class="align-center"> <img src="/assets/graphics/content/prose/osmgeoweek.jpg" alt="osm-geoweek" /> </figure> <p class="dropcap">Monday evening (11/14) we are hosting a Mapathon as part of <a href="http://osmgeoweek.org/">OSM GeoWeek</a> at our office. Join Development Seed, USAID, the American Red Cross and MapGive to improve the map for humanitarian efforts around the world. We’ll work on tasks that support these organizations and their focus to decrease volcanic risks in Indonesia, eliminate Measles and Rubella in Malawi, and gain better insight into energy access in Sub-Saharan Africa.</p> <p>You don’t need to be an expert mapper to participate. Just bring your laptop and we’ll help get you set up and equipped to map. <a href="https://www.eventbrite.com/e/osm-geoweek-mapathon-hosted-by-development-seed-tickets-29088088255">Reserve your spot now!</a></p> 2016-11-11T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/11/osm-geoweek Bienvenida Laura! http://www.developmentseed.org/blog/2016/11/10/welcome-laura <p class="dropcap">We are thrilled to welcome Laura Gillen to the Development Seed team! Laura will run Mission Control, helping us to improve our operations, grow Development Seed’s community presence, and run big events like <a href="https://satsummit.io">SatSummit</a>.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/laura-welcome.jpg" alt="Laura on a boat at sea" /> <figcaption></figcaption> </figure> <p>Laura has a fantastic mix of civic and business experiences. She has worked at Women for Women International and worked on the communications team for Senator Al Franken’s 2014 re-elect campaign. Most recently, Laura was the first employee of Laurel Strategies and helped to grow the strategic communications firm and global business advisory. While at Laurel, she worked with clients that span multiple sectors, including software, e-commerce, and telecomm, helping them with the development and execution of strategy, maximizing positioning and media relationships, and refining communications.</p> <p>When she’s not at the office, you can find the native Minnesotan camping at a DC coffee shop, serving aces on the tennis court, jogging to carefully curated playlists, or boarding a plane (or boat) for her next adventure.</p> <p>Welcome Laura! We can’t tell you how excited we are that you are here.</p> 2016-11-10T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/10/welcome-laura It’s Election Day in the USA: Where will you watch the results? http://www.developmentseed.org/blog/2016/11/08/post-eday <p class="dropcap">Wherever you end up watching the election returns tonight, check out The Washington Post live election maps. We worked closely with The Post to build lightning fast maps that provide up-to-the-second data presented in a manner that is easy to understand.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/wapo-preview.gif" alt="election-maps" /> </figure> <p>We iterated and improved our approach over 24 primary nights and we have introduced new features for the today’s election. Here is some of what you can expect tonight:</p> <h3 id="clearinformative-coverage">Clear/informative coverage</h3> <p>The Post’s live election maps take data from 532 races and organize it into an interface that is understandable at a glance but also allows users to explore the context and detail of this election. We include a live cartogram map, to overcome <a href="https://www.washingtonpost.com/graphics/politics/2016-election/how-election-maps-lie/?tid=sm_tw">the disadvantages of traditional election maps</a> and visually represent the electoral weight of each state. Also, after the states are called a wind map will show how the vote has shifted since the last election in each county for every race.</p> <h3 id="updates-and-callouts-to-catch-important-moments">Updates and callouts to catch important moments</h3> <p>The interface is designed to make sure that you don’t miss important updates, even if you look away from your screen. We pull in new data across roughly 3,000 counties, every 4 seconds, to show the freshest data possible. We’ve redesigned our <a href="https://developmentseed.org/blog/2016/03/15/tracking-live-events/">“freshies”</a> – callouts on the map that notify users of tightening races or new vote leaders.</p> <p>Live maps are deployed throughout The Washington Post website. The Post’s reporters and analysts will add additional news and analysis in their features and the <a href="https://www.washingtonpost.com/politics/2016/live-updates/general-election/real-time-updates-on-the-2016-election-voting-and-race-results/">Washington Post Live Blog</a>, which you can read without taking your eyes off the results.</p> <h3 id="coverage-no-matter-where-you-are">Coverage no matter where you are</h3> <p>Many people will track the election returns on the road or on multiple screens. The Post’s maps are just as usable and fast on mobile. No matter where you end up spending this election night, be sure to take a look at <a href="https://www.washingtonpost.com/">washingtonpost.com</a> to keep up-to-date with tonight’s gripping election results.</p> 2016-11-08T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/08/post-eday Thanks Maptime! http://www.developmentseed.org/blog/2016/10/06/Maptime <p class="dropcap"><a href="http://maptime.io/">Maptime</a> is a great meetup for people interested in learning modern mapping techniques &amp; applications. We recently hosted a training <a href="http://www.slideshare.net/DylanMoriarty/maps-by-hands">on hand-drawn maps</a> along with Maptime DC. We were delighted to have 40 people come by.</p> <p>When you let people loose to create their own unique map through the good ol’ technique of “pen to paper,” (and in some instances, crayon) you can see some good creativity. By the end of the night we had a variety of maps that reflected the diverse group of attendees.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_3.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_4.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_2.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_5.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_1.jpg" alt="maptime photo" /> <figcaption>Photos by Brian Davidson &amp; Dylan Moriarty.</figcaption> </figure> <p>Hopefully we’ll see you at the next <a href="http://www.meetup.com/Maptime-DC/">Maptime</a>. In the meantime, let us know if there are other geo or tech events you’d like to see or to help us host!</p> 2016-10-06T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/10/06/Maptime OpenAerialMap Improved http://www.developmentseed.org/blog/2016/09/22/oam-improvements <p class="dropcap">It’s been more than a year since we first launched <a href="https://developmentseed.org/blog/2015/05/27/introducing-openaerialmap/">OpenAerialMap</a>, and ever since, we’ve continued to invest in the site and make improvements. We truly believe that this service helps users to share, find and use aerial imagery in as simple and efficient of a way as possible. Below I’ve outlined a few of the ways we’ve improved the site to make for a seamless experience.</p> <p>Some of the latest changes include:</p> <ul> <li>A brand new <a href="http://openaerialmap.org/">homepage</a> that introduces users to the project.</li> <li>Improvements to the <a href="https://upload.openaerialmap.org/">upload form</a> by adding integration with Dropbox, and Google Drive (not yet deployed, but expect to see this soon). This aims to lower the contribution barrier.</li> <li>Added a feedback form to the <a href="https://beta.openaerialmap.org/">browser</a> (allows users to easily report problems with the imagery).</li> <li>Added options for imagery preview, including TMS.</li> <li>Under the hood improvements.</li> </ul> <p>I’ll be at <a href="http://2016.stateofthemap.org/">State of the Map</a> in Brussels this week to talk about the OAM project, specifically these improvements and the changes to come. If you’re there be sure to come listen – you can find me in <a href="http://2016.stateofthemap.org/2016/open-imagery-matters-remote-mapping-with-openaerialmap/">Auditorium C, Friday at 12:30pm</a>.</p> 2016-09-22T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/09/22/oam-improvements Preview The Washington Post's election maps http://www.developmentseed.org/blog/2016/09/08/wapo-preview <p class="dropcap">This week we finally got the chance to show some of the tools that The Washington Post will use to cover election results in November, live on The Post’s website.</p> <p>On Tuesday, The Washington Post and SurveyMonkey released a comprehensive presidential poll of all 50 states. Alongside this ambitious polling effort, The Post also rolled out preview pages detailing the <a href="https://www.washingtonpost.com/2016-election-results/us-presidential-race/">Presidential</a>, <a href="https://www.washingtonpost.com/pb/2016-election-results/us-senate/">Senate</a>, <a href="https://www.washingtonpost.com/pb/2016-election-results/house-of-representatives/">House</a>, and <a href="https://www.washingtonpost.com/pb/2016-election-results/governors-state-race/">Gubernatorial</a> races across all 50 states and Washington DC. These pages show the current state of the race and offer predictions and historic results.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/wapo-preview.gif" alt="election-maps" /> </figure> <p>The maps on these pages lean on much of the technology and workflows that we are developing for Election Day. To render maps, we make heavy use of <a href="https://github.com/mapbox/mapbox-gl-js">Mapbox GL JS</a>, an open-source mapping library that uses your graphics processor to draw fast, highly detailed maps—borrowing from technologies that render detailed video game universes.</p> <p>This framework provides cutting-edge hardware acceleration to draw beautiful maps that effortlessly accommodate highly interactive input from Washington Post readers. However, making WebGL maps work for live reporting an election on a major national news site presented a number of interesting challenges:</p> <ul> <li><strong>Albers US projection</strong> To make the coverage work with US-centric elections, we employed a method called* “Lying to Leaflet,” inserting the more appropriate <a href="http://bl.ocks.org/mbostock/2869946">Albers USA projection</a>, in place of the default <a href="https://en.wikipedia.org/wiki/Web_Mercator">Web Mercator</a>. You can use this method to show other projections, for example <a href="https://www.mapbox.com/blog/mapping-arctic-ice-polar-projection/">the polar projection to map artic ice</a>.</li> <li><strong>Dynamic sources</strong> Although not required for preview pages, the real-time election data on Election Day will require us to join data to vector tiles “on the fly” before feeding these to Mapbox GL to render. We are doing this for preview pages today in order to test and optimize our data pipeline in advance of November.</li> <li><strong>Reactive updates</strong> We employ a <a href="https://gist.github.com/staltz/868e7e9bc2a7b8c1f754">reactive programming design pattern</a> to build the code around Mapbox GL. This allows us to keep program state in one central location and only respond to necessary changes, including using a “style diff” to update the maps. <a href="https://www.mapbox.com/blog/mapbox-gl-js-reactive/">A recent Mapbox blog post further describes this approach.</a></li> <li><strong>Shared rendering resources</strong> The Post’s expansive coverage requires us to load several maps concurrently on the same webpage, in main views and in small multiples. By using a shared pool of web workers, we minimized the overhead for faster loading.</li> <li><strong>Independent components with a central driver</strong> At the same time, we often need maps to behave independently. Components must be maximally flexible, embeddable on pages we don’t control, and usable in a number of configurations. To do this, we created a central, application-wide, reducer-based store. This allows the Post to create pages with an arbitrary number of components without too much orchestration.</li> <li><strong>Image-based fallbacks</strong> Though <a href="http://caniuse.com/#feat=webgl">WebGL</a> support is already high and climbing, a small percentage of users on older browsers don’t have access. To serve these users, we are building a workflow that generates static images directly from our Mapbox GL pipeline. This ensures that all users can see the most recent data no matter their browser.</li> </ul> <p>As this election season progresses, The Post will provide contextual, historical, and demographic data while updating its predictions. In parallel, we will refine and test our approach to make sure it provides powerful data on what will certainly be an interesting election.</p> <figure class="align-center"> <a href="https://www.washingtonpost.com/2016-election-results/us-presidential-race/"><img src="/assets/graphics/content/prose/wapo-small-mult.jpg" alt="demographic-maps" /></a> <figcaption>Small multiples in a box.</figcaption> </figure> <p><em>*h/t <a href="https://twitter.com/mojodna">Seth Fitzsimmons</a> of Stamen Design.</em></p> 2016-09-08T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/09/08/wapo-preview Machine Learning at State of the Map US http://www.developmentseed.org/blog/2016/08/17/sotm-skynet <p class="dropcap">A couple weeks ago, I talked at <a href="http://stateofthemap.us/">State of the Map US</a> about our “skynet” experiments, using deep learning with satellite imagery and OpenStreetMap data to develop algorithms for automated mapping. Thanks to everyone who came to my talk–I’m really grateful for the positive feedback and excited by the great ideas that you shared!</p> <p>The <a href="https://www.youtube.com/watch?v=BftllmqXSpA&amp;list=PLqjPa29lMiE3eR-gK80irr3xdUiRbIMeg&amp;index=2">video</a> and <a href="http://anand.codes/talks/sotmus-2016/">slides</a> are available, but here’s the “tl;dr”:</p> <div class="callout"> <p>Can deep learning get us to highly accurate and complete automated road detection?</p> <p><strong>Almost certainly.</strong></p> <p>Do you need to have hundreds of thousands of dollars and a dedicated research lab full of experts to do it?</p> <p><strong>Definitely not!</strong></p> </div> <figure class="align-center"> <a href="https://www.youtube.com/watch?v=BftllmqXSpA&amp;list=PLqjPa29lMiE3eR-gK80irr3xdUiRbIMeg&amp;index=2"><img src="/assets/graphics/content/prose/sotm-skynet.jpg" alt="skynet" style="max-height: 300px" /></a> <figcaption>Here is the somewhat longer version.</figcaption> </figure> <p>Conversations at State of the Map and afterwards made it clear that there is a lot of interest and activity right now around using computer vision and machine learning for mapping:</p> <ul> <li>The <a href="http://deeposm.org/">DeepOSM</a> team is using a lean, fast neural net to estimate the likelihood of “road registration errors” in OSM.</li> <li>Facebook gave us some good insight into their <a href="http://forum.openstreetmap.org/viewtopic.php?id=55220">internal efforts at automated mapping</a>.</li> <li><a href="https://twitter.com/calimapnerd">Dale Kunce</a> suggested that Dar es Salaam has been sufficiently mapped by the Missing Maps effort to provide good training data. As soon as I got home I started working with this and have already been getting promising results in East Africa: <img src="/assets/graphics/content/prose/skynet-dar.png" alt="Skynet results in Dar es Salaam" /> <em>(left: input image, © <a href="https://www.mapbox.com/">Mapbox Satellite</a>; middle: <a href="https://www.openstreetmap.org/">OpenStreetMap</a> data; right: our model)</em>.</li> <li>We had some great conversations with the World Bank on how to apply this to their own efforts.</li> <li>There were thoughtful (and mildly controversial) reflections in blog posts from <a href="http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.html">Mike Migurski</a> and <a href="https://tomlee.wtf/2016/07/31/introductory-openstreetmap-politics/">Tom Lee</a> about the broader implications of “robot mapping.”</li> </ul> <p>There’s much more to do here, but it finally feels like deep learning is starting to become accessible to us as a community interested in real, practically useful results. Within the next few weeks, a few people working on machine learning for road mapping are going to jump on a hangout. If you are interested to join, please <a href="https://twitter.com/anandthakker">ping me on twitter</a>. I’m excited to see what we can build.</p> <p>Meanwhile, we’ve been working to clean up the code and docs from our experiments, in the hopes that others in the community can replicate our results and take them farther. Check out the readmes for <a href="https://github.com/developmentseed/skynet-data">skynet-data</a> and <a href="https://github.com/developmentseed/skynet-train/tree/master/segnet">skynet-train</a>. If we did it right, you should be able to start training your own neural net to extract features from satellite imagery with little more than an AWS account, and an hour of setup, and 🐳<code>docker run ...</code>. If not, drop us a line, or open an issue or PR!</p> <p><em><a href="https://www.flickr.com/photos/openstreetmapus/27930399144/in/album-72157671548583845/">cover</a> and <a href="https://www.flickr.com/photos/openstreetmapus/28441015702/in/album-72157671548583845/">card</a> photos by <a href="https://twitter.com/tatsvc">@tatsvc</a></em></p> 2016-08-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/08/17/sotm-skynet Get the best open satellite imagery from one interface http://www.developmentseed.org/blog/2016/08/04/ad-sentinel2 <p class="dropcap">Sentinel-2 data is now available in the <a href="https://fetch.astrodigital.com/">Astro Digital Imagery Browser</a>. Now you can quickly search for, publish, and download the best open satellite imagery from NASA and ESA in one interface. The interface is powered by the <a href="https://api.astrodigital.com/">Astro Digital API</a> which recognizes that users just want the best data and don’t always care what satellite it comes from.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ad-sentinel.png" alt="Sentinel in Astro Digital Imagery Browser" /> </figure> <p>Like our previous work with Landsat, this is built on top of powerful, <a href="https://github.com/sat-utils/">open source processing tools</a> we’re creating for working with Sentinel-2 imagery.</p> <p>Truly open truly works. This was only possible because AWS made <a href="http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/">Sentinel-2 data programmatically accessible</a>.</p> 2016-08-04T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/08/04/ad-sentinel2 Maintain a common user experience across websites with Design Systems http://www.developmentseed.org/blog/2016/07/14/reusable-design-components <p class="dropcap">We use Design Systems to help us more efficiently organize and reuse design assets and code across sites that share elements.</p> <p>We often create and tie together multiple websites, building what feels like a single site to its end user. Doing this makes our projects easier to maintain, allows us to develop in an extremely agile manner, and gives us the freedom to use the technology that best suits each task.</p> <p>For instance, we built a number separate sites for <a href="http://openaerialmap.org">OpenAerialMap</a>, a tool that provides humanitarians with satellite and drone imagery. We built a separate site for the <a href="http://beta.openaerialmap.org">imagery browser</a>, the <a href="https://upload.openaerialmap.org">data uploader form</a>, and the <a href="http://docs.openaerialmap.org/">documentation</a>, just to name a few. Building a site for each of these facets allowed us not only to select the tech stack that best fit the needs of the project, but also prevented us from building a monolith app whose maintenance would be complicated. However, maintaining a consistent design language and interactions across all of them became a real challenge.</p> <p>We found ourselves copying and pasting code between sites, creating a codebase that was hard to maintain. To cope with this we built Design System, a centralized module to contain all shared elements across sites.</p> <figure class="align-center desktop"> <div class="desktop-group-trio"> <div class="media-frame"> <img src="{{site.baseurl}}/assets/graphics/content/prose/reusable-design-components--uploader.png" alt="OAM Uploader Form" width="1920" height="960" /> </div> <div class="media-frame"> <img src="{{site.baseurl}}/assets/graphics/content/prose/reusable-design-components--docs.png" alt="OAM Docs" width="1920" height="960" /> </div> <div class="media-frame"> <img src="{{site.baseurl}}/assets/graphics/content/prose/reusable-design-components--browser.png" alt="OAM Browser" width="1920" height="960" /> </div> </div> <figcaption>These are all separate websites, but we manage the design and user interactions in one place.</figcaption> </figure> <h2 id="design-system">Design System</h2> <p>Building on the approach we took for <a href="http://collecticons.io/">Collecticons</a>, and drawing further inspiration from <a href="http://foundation.zurb.com/sites/docs/">Foundation</a>, <a href="http://v4-alpha.getbootstrap.com/getting-started/introduction/">Bootstrap</a>, and <a href="https://medium.com/salesforce-ux/designing-products-that-scale-c8f3001f709b">this great article by Salesforce</a>, we used methods for distributing code through a package manager, e.g. node package manager (NPM) or Ruby Gems. This allowed us to take all of our shared elements across sites, and turn them into an installable node module that can be included in each website that makes up a project. The three main elements that make up the Design System are:</p> <ul> <li><code>scripts</code> that can be included like any node module;</li> <li><code>styles</code> that can be loaded as a whole or as individual components and;</li> <li><code>images</code> which are common throughout the ecosystem and include logos and other branding elements.</li> </ul> <p>Instead of copying and pasting code across websites, using <a href="https://www.npmjs.com/">node packages</a> let us store all shared elements in one place and make those elements easily accessible and readily available to each and every site. For example, if we wanted to update the logo, we’d only have to change it in one place for it to be automatically updated on all other sites the next time they build.</p> <p>Finally, having one repository for all shared elements, allows codebases to be more easily maintained and standardized, helping developers get started on new products, which is especially important for open source projects.</p> <p>Although we’re still testing out this approach, using OAM, we think having a centralized design system, like the one we’ve shown here, is a great solution for complex and/or open source projects. Want to help? Give the <a href="https://github.com/hotosm/oam-design-system">Design System</a> a try and send any suggestions or recommendations our way. We’re always looking to improve our workflow. A quick look at the docs is all that’s needed to start building. For more tech-savvy users, an example of its usage can be found in the <a href="https://github.com/hotosm/oam-docs/blob/master/gulpfile.js">OAM Docs gulpfile</a>.</p> 2016-07-14T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/07/14/reusable-design-components Development Seed Summer BBQ! http://www.developmentseed.org/blog/2016/07/07/bbq-gathering <figure class="align-center"> <a href="https://devseedbbq.splashthat.com"><img src="/assets/graphics/content/prose/devseedbbq.png" alt="RSVP" style="max-height: 300px" /></a> </figure> <p class="dropcap">Summer is here, so it must be time for a Development Seed summer BBQ. The whole team is in town and we are going to celebrate with our good friends, half smokes, and the finest blue ribbon beverages. Come by our new(ish) <a href="https://a.tiles.mapbox.com/v4/devseed.l73g0j47/attribution,zoompan.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#17/38.90635/-77.02437">home in Blagden Alley</a> on Wednesday, July 20.</p> <p>We’ll kick things off around 5:30. Bring friends and loved ones. Furry friends are always welcome!</p> <p>We’ll keep the food and beverages flowing. Help us out by <a href="https://devseedbbq.splashthat.com">letting us know if you plan to join the fun!</a></p> 2016-07-07T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/07/07/bbq-gathering Join Development Seed as an engineer http://www.developmentseed.org/blog/2016/06/21/engineer <p class="dropcap">We are looking for an engineer who wants to help organizations better understand and distribute their data. You’ll join a dynamic group of engineers and designers in DC and Lisbon tackling big global challenges with open data and open technology. You’ll collaborate with everyone on the team to produce products and systems with real impact.</p> <p>We’ll want you to design and develop APIs, tackling the problems that come with serving data at scale. You’ll create data pipelines that transform messy, real-world data into structured information. This could mean <a href="projects/astro-digital/">satellite imagery</a>, <a href="/projects/openaq/">air quality data</a>, or even <a href="/projects/wapo/">real-time election reporting</a>. For any project, you’ll get a chance to experiment with new technologies to best meet our client’s needs.</p> <p>Let’s talk about you. You are a great communicator and collaborator. You learn quickly and love trying new things. You know when to use a quick fix and when to invest more time refactoring. And you are looking to use technology to make a real difference with other likeminded people.</p> <p>Check out the <a href="/about/engineer/">job post</a> for more details.</p> 2016-06-21T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/06/21/engineer Run Mission Control at Development Seed http://www.developmentseed.org/blog/2016/06/17/mission-control <p class="dropcap">We are looking for someone to run <a href="/about/mission-control/">Mission Control</a> Development Seed. We are a dynamic group of engineers and designers in DC and Lisbon tackling big global challenges with open data and open technology. Mission Control will run all aspects of our business operations. You will work directly with me to help Development Seed grow, scale and maximize our impact.</p> <p>In Mission Control you will lead out all of our operational streams. You will manage our DC office, run events, streamline our business processes, and help the team to run faster. Every day will be different. On one day you might coordinate team travel; review a million dollar contract; buy a flock of drones; and help us to launch a new product.</p> <p>A little bit about you. You run fast and learn quickly. You don’t need to be a technologist, but you must embrace the open technology community and a humane startup culture. You are a great communicator. You instinctively distill commanders intent. You thrive in a flat organization. You take responsibility for your own time and get things done through orchestration rather than authority. You constantly look for ways to make things better. You are good person and you want to make a real difference with other fun, decent humans.</p> <p>If this sounds like you, take a gander at the <a href="/about/mission-control/">Mission Control job post</a>.</p> <figure class="align-center mono-spacing"> <img src="/assets/graphics/content/prose/datamapsgraphic.svg" width="718" height="425" alt="Data map graphic" /> </figure> 2016-06-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/06/17/mission-control Beautiful badges for OSM heroes http://www.developmentseed.org/blog/2016/06/09/mm-badges <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-mapathon.png" alt="Image" style="max-height: 300px" /> </figure> <p class="dropcap">Good illustrations carry a lot of weight. They communicate complex concepts and capture complex emotions. When you are use an illustration as a reward, it is all the more important to be thoughtful about the feelings that illustration inspires.</p> <p>We recently launched <a href="https://developmentseed.org/blog/2016/03/01/osm-stats-launched/">real-time analytics for Missing Maps</a>, a project that grows the community of humanitarian mappers around <a href="https://www.openstreetmap.org/">OpenStreetMap (OSM)</a>. As a part of the projects, we created a badge reward system that highlights the achievements of <a href="http://www.missingmaps.org/">Missing Maps</a> contributors. We created a style for these badges that would recognize the contributions of mappers, while also acknowledging that badges aren’t the primary reason that they participate. We also developed a style that is guided by collaborative design principles and is ready for others to contribute.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badge-heros.png" alt="Image" /> </figure> <h3 id="rewarding-heroes">Rewarding heroes</h3> <p>Our reward system was designed to recognize the contributions of humanitarian mappers, while being clear that we know they aren’t doing it for the badge. We needed to focus the badges on the real-world impact and contributions these mappers were having. We designed it to be fun and encouraging, while also recognizing that mapping would sometimes be done in the aftermath of disasters with devastating consequences.</p> <h4 id="friendly-unique-and-engaging">Friendly, unique, and engaging</h4> <p>The Missing Maps badge system provides a showcase for the work a person has made in contributing to OSM. A badge can provide a small milestone for what they can map next and can be an encouragement to map more. This can help cut down how repetitive, seemingly endless, and often thankless mapping out the world in OSM can be. Badges can help break up a long night of disaster mapping, or can let people know that it is time to take a break.</p> <p>We felt the badges should have a welcoming quality because MissingMaps relies heavily on volunteer effort. MissingMaps encourages contributions by making Mapathons fun, communal events (often with pizza). The importance of the work is the initial hook, but it’s the welcoming community that keeps people coming back, and the badges should reflect that.</p> <p>To make the badges feel like rewards we felt they needed to look interesting, unique, and beautifully designed. Something substantial that can feel more like a reward than a colored circle with text on top might. To this end, the badges were drawn in a modern style but with a fluid, non-gridded style to emphasize that each badge is a unique and thoughtful thing, created lovingly by hand.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badge-working.png" alt="Image" /> </figure> <h3 id="principles-for-missing-maps-badges">Principles for Missing Maps badges</h3> <p>Designing for future contributors is like designing for an internal team where you don’t know who teammates are going to be. We created guidelines, a template for getting started, and examples that serve as blueprints. We created a style that fits with the Missing Maps brand while capturing the goals of any reward system - make it friendly, make it feel substantial, and tell a story to keep the user engaged. And to keep it accessible for future contributors, we made it straightforward and open.</p> <h4 id="consistency--simplicity">Consistency &amp; Simplicity</h4> <p>We used a consistent and simple style for the Missing Maps user page badges to ease the burden on a contributor to make graphics that fit within the system. The elements that make up this style can be anything from line widths, to patterns, or other visual cues. For Missing Maps badges we use a color and the visual cues to create the sense of unity.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badgestyle.png" alt="Image" /> </figure> <p>Most of the badges are tiered, providing a natural flow to tell a story between badges of the same category. For example, our users are rewarded the “mapathoner badge” for attending mapathons. There are three levels for this badge, each rewarded for attending 5, 20, and 50 mapathons respectively. The badges for these levels each features a common character that starts out in a slight jog, then a sweating sprint, and finally reaches a finish line. This provokes the user to get to the next level in order to find out what might happen next, and also makes the badges feel like part of a set.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badge-story.png" alt="Image" /> </figure> <h3 id="open-source">Open Source</h3> <p>By making the badges entirely in digital tools and in the svg format we ensured most artists who would want to contribute to the project start out on a level playing field. There aren’t any fancy filters, proprietary fonts, or program-exclusive styling on these graphics.</p> <p>We’ve documented rules and guidelines for creating future badges and provided a template file for getting started. Everything is marked out on Github and available for anyone to contribute. Watch the repository for upcoming opportunities to work on new badges.</p> 2016-06-09T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/06/09/mm-badges OSM-in-a-box http://www.developmentseed.org/blog/2016/05/16/macrocosm-easy <p class="dropcap">We’ve been thinking about how to make <a href="https://github.com/developmentseed/macrocosm">Macrocosm</a>, our fork of the OpenStreetMap API, easier to use. Our vision is that one day you’ll be able to type <code>npm install openstreetmap</code> and be off to the races.</p> <p>We’re not there yet, but to that end, we’ve started to bundle <a href="http://ideditor.com/">iD</a> as a submodule in the Macrocosm repo. Now you can <a href="https://github.com/developmentseed/macrocosm#installation">set up your database, API, and editor all from the same repo</a>, using simple commands.</p> <p>We think there’s a lot of use for this deployable OSM. Testing, creating backups, and developing OSM applications all come to mind. Heck, you could even set up an awesome collaborative editing environment for you and your friends. You can kick the tires on <a href="https://github.com/developmentseed/macrocosm#installation">Macrocosm</a> or <a href="https://github.com/developmentseed/macrocosm/issues">contribute</a>, to get us closer to a full OSM-in-a-box.</p> 2016-05-16T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/16/macrocosm-easy The Value of User Research http://www.developmentseed.org/blog/2016/05/16/fast-valuable-ux-research <p class="dropcap">To build a successful product, you must deeply understand your users and their needs. Unfortunately, when timelines are tight, user research is often what gets dropped or simply forgotten about, right when it is needed the most. Doing even a little bit research up front can avoid costly delays later in the process, saving time and resources, resulting in a better product.</p> <p>We recently put this theory to the test while working with the <a href="https://hotosm.org/">Humanitarian OpenStreetMap Team (HOT)</a> on their <a href="https://hotosm.org/projects/openstreetmap_analytics">OSM Analytics prototype</a>. HOT came to us to help them better understand what their users really wanted and needed, to inform what they should build, from the beginning. Because of their short timeframe, we condensed our research process, conducting user research, iterating and then testing to verify assumptions, within a weeks time.</p> <p>For our initial UX research phase, we spent an intensive day with the team at HOT helping them think through their <a href="https://en.wikipedia.org/wiki/SMART_criteria">measurable goals</a> for the product, the kind of people they hoped would use their tool, and finally, the kind of content that would achieve their goals as well as those of their users. Doing this type of analysis up front, not only ensured that we were all driving towards the same goal, but also focused us on features and design decisions that were right for the product, instead of suggesting whatever is new and flashy. The result by the end of the day—a simple, testable prototype.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/OSM-analytics-1.jpg" alt="Image" width="1440" height="360" /> </figure> <p>The second step of the process was to get our prototype in front of real users. Testing is invaluable because it provides the project team with fast feedback, allowing them to make more informed changes early on. For OSM Analytics, we tested our prototype and verified our assumptions by using our preferred method, <a href="https://www.nngroup.com/articles/remote-usability-tests/">moderated usability testing</a>. We asked a number of relevant users to perform tasks on the prototype, to ensure that what we built matched their expectations. Their responses quickly showed us what worked and what didn’t within the product, allowing us to fix big issues and more confidently continue building. Also, hearing directly from users how they’d like to use the tool, gave us and the HOT team insight into what future phases of the product should look like.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/OSM-analytics-2.jpg" alt="Image" width="1440" height="360" style="border: 1px solid #ddd;" /> </figure> <p>Within the week we had for this project, we not only gained an understanding of HOT’s users wants and needs, but also learned what worked for the product and what didn’t. This all allowed us to iterate our way to a more user centered design showing the value of doing UX research and testing, no matter the timeline.</p> <p>If you haven’t tried OSM Analytics yet, <a href="http://osm-analytics.org/">check it now</a>. Join in or watch further development of the tool <a href="https://github.com/hotosm/osm-analytics">on GitHub</a>.</p> 2016-05-16T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/16/fast-valuable-ux-research Ecuador: Post-Earthquake drone imagery now on OAM http://www.developmentseed.org/blog/2016/05/06/ecuador-drones <p class="dropcap">Post-earthquake drone imagery <a href="http://beta.openaerialmap.org/#/-80.33477783203125,-0.5273363048115043,8/2100011013?resolution=high">is now available on OpenAerialMap</a>. These images were collected by local drone operators coordinated by the <a href="http://uaviators.org/">UAViator network</a>, mostly by the Instituto Geográfico Militar, the Universidad de Cuenca, and the <a href="https://twitter.com/uav_ecu">Voluntarios UAV-ECU</a>. Thanks to <a href="https://twitter.com/fruizc">Francisco Ruiz</a> for helping coordinate. All of the imagery is very-high resolution, and available for tracing in OSM.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ecuador-drone.jpg" alt="Image" /> </figure> <p>Drone pilots around the world stand ready to help respond to disasters. OpenAerialMap is a platform for drone and satellite operators to easily share openly licensed imagery. OpenAerialMap is a managed in partnership by the Humanitarian OpenStreetMap Team, Development Seed, Azavea, Stamen, and other humanitarian tech partners.</p> <p>You can contribute by <a href="http://beta.openaerialmap.org/#/-80.33477783203125,-0.5273363048115043,8/2100011013?resolution=high">mapping now</a>. If you are new to OSM mapping, get started with these <a href="http://www.missingmaps.org/contribute/">great training videos from our friends at Missing Maps</a>.</p> <figure class="bleed-full"> <a href="http://beta.openaerialmap.org/#/-80.33477783203125,-0.5273363048115043,8/2100011013?resolution=high"><img src="/assets/graphics/content/prose/oam-ecuador-drone.jpg" /></a> <figcaption>Start mapping now</figcaption> </figure> 2016-05-06T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/06/ecuador-drones Modeling Solutions to Energy Access Problems http://www.developmentseed.org/blog/2016/05/04/mit-rem-demo <p>MIT-Comillas’ Reference Electrification Model (REM) determines the most cost-effective strategy to provide energy access in areas that lack it. We are collaborating with the <a href="http://universalaccess.mit.edu">MIT-Comillas Universal Energy Access Group</a> <sup><a href="#footnote-1" title="View footnote">[1]</a></sup> to build modern mapping platforms that put this data in the hands of policymakers, energy planners, and entrepreneurs to better connect rural communities.</p> <p>The basic input data for REM are fuel costs, solar irradiance, grid extent and reliability, consumer demand and infrastructure costs. It then uses a deep learning-based computer vision system that the MIT-Comillas team developed to locate buildings in satellite imagery. With this infrastructure information and geodata, REM generates detailed engineering designs, providing recommendations on the best ways to connect rural communities through a combination of grid extensions, microgrids, and stand-alone systems.</p> <p>We are working with the MIT-Comillas team to make these unique and valuable insights more accessible. Lightweight web maps can easily get these localized plans to regional power authorities and development organizations in the field. Further opening the data can show where off-grid approaches are viable intermediary solutions for electrification, motivate essential policy decisions, and encourage off-grid energy providers to work in the places they can have the greatest impact.</p> <figure class="bleed-full"> <section id="rem-demo" style="height: 640px;"></section> <script src="https://devseed.com/rem-web-demo/dist/demo.js"></script> <figcaption>Example REM outputs for the Vaishali district in Bihar, India.</figcaption> </figure> <p>REM enables users to perform sensitivity analyses to external factors like demand level, grid reliability, fuel and technology cost, and cost of non-served-energy. In this example, you can see how the model projections change with the price of diesel. As diesel prices rise, it becomes more and more cost efficient to connect people to the existing grid (this is because many of the solar-and battery-powered microgrids also use diesel for backup power). But even with high diesel prices, there are some areas that are better served through off-grid power models. We built this map using <a href="https://www.mapbox.com/mapbox-gl-js/api/">Mapbox GL</a>. Mapbox GL uses your computer or phone’s powerful graphics card to quickly process huge amounts of data. This enables us to take complex model results and render them on the fly into a highly detailed, interactive web map based on input from the user. The extremely efficient vector tiles that Mapbox GL uses for geodata makes the tool usable even on slower Internet connections.</p> <p><a href="http://www.worldbank.org/en/news/feature/2015/05/18/where-does-the-world-stand-in-reaching-sustainable-energy-objectives">1.1 billion people still lack access to electricity</a> and frequently use dirty kerosene lamps for lighting. Nearly 3 billion rely on harmful biomass fuels for cooking, and these activities contribute to air pollution that <a href="https://medium.com/@openaq/filling-the-air-quality-data-gap-10e0a494517#.p6zfcf2xe">kills millions of people per year</a>. Expedited electrification will decrease risks of respiratory disease, improve income generation prospects, and enable children to study longer.</p> <p>Immediately extending the existing grid isn’t always the right answer. Depending on complex geographic and economic factors, it may be more cost-effective and sustainable to build microgrids and stand-alone systems instead. If the grid’s reach eventually extends to these areas, connecting grid-compatible off-grid solutions may encourage infrastructure reuse and promote a cleaner generation mix as well. Satellite imagery, machine learning, powerful modeling tools, and modern mapping technology can empower infrastructure planners to design off-grid electricity solutions that are viable, cost-effective, and can better serve communities.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Ongoing funding for REM development is provided by the <a href="http://tatacenter.mit.edu/">Tata Center for Technology and Design and MIT</a>.</p> <ol class="footnotes-list"> <li id="footnote-1">The MIT-Comillas Universal Energy Access Research group currently works with policymakers, NGOs, and private companies in India, Rwanda, and Uganda to explore paths towards universal electricity access. Given the multivariate objectives inherent in sustainable development, the group is investigating how regulation and advanced planning tools have the potential to enable game-changing technologies and business models for electrification.</li> </ol> 2016-05-04T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/04/mit-rem-demo Announcing Gippy v1.0 http://www.developmentseed.org/blog/2016/05/03/gippy <p class="dropcap">We’re happy to announce the beta release of <a href="https://github.com/gipit/gippy">Gippy v1.0.0b1</a>. Gippy is a python library to process geospatial raster data (like satellite sensor data) fast and efficiently, regardless of image size, or the size of your machine. We’ve had a lot of success building <a href="https://developmentseed.org/projects/landsat-util/">tools for easily processing Landsat imagery</a>. Gippy gives us the processing engine that will allow us to quickly expand these capabilities to process other open datasets, such as Sentinel-2 and MODIS.</p> <p>Building upon the previous 0.3 version, we’ve added documentation, automated testing, and additional functionality to create a 1.0.0 release. Gippy is open source and released under an Apache license.</p> <h3 id="open-source-powered-by-open-source">Open-source, powered by open-source</h3> <p>Gippy automatically handles several issues common to geospatial raster data, such as handling of nodata values and managing very large datasets by breaking up processing into pieces. Gippy doesn’t reinvent the wheel, instead encapsulating functionality from <a href="http://www.gdal.org/">GDAL</a> for reading and writing to files, and <a href="http://cimg.eu/">CImg</a> for doing image processing. It is also reasonably lightweight, requiring only GDAL system libraries and a reasonably modern C++ compiler.</p> <h3 id="features">Features</h3> <p>Gippy is built for processing remote sensing data. It intelligently manages band numbers and handles <code>nodata</code> values in a smart way. It also supports process chaining and piecewise processing. It comes with a range of operations and algorithms out of the box, including:</p> <ul> <li>arithmetic, logical, exponential and other operations</li> <li>dynamic creation and applying of masks</li> <li>algorithms for mosaicing multiple images in any combination of projections</li> <li>statistics, spectral correlation and covariance</li> <li>cloud detection algorithms ACCA and F-mask for Landsat7</li> <li>multiple indices (NDVI, EVI, LSWI, NDSI, NDWI, SATVI, MSAVI-2)</li> <li>arbitrary linear transforms</li> <li>RX Detector, a multispectral anomaly detection algorithm</li> </ul> <h4 id="referencing-bands">Referencing bands</h4> <p>With Gippy there’s no more having to worry about trying to identify band numbers in your data. Gippy allows algorithms to be written to target specific types of bands (e.g., red, nir, lair). When a GeoImage, the main class that is used, is opened or created bandnames can be assigned by setting the bandnames keyword to open. Bands can be referenced via band index or the band name and can be iterated over as a collection.</p> <pre><code>from gippy import GeoImage geoimg = GeoImage.open(filename, bandnames=(['red', 'green', 'blue'])) red_arr = geoimg['red'].read() </code></pre> <h4 id="theres-nodata-around-these-parts">There’s nodata around these parts</h4> <p>To make identification and customization even easier in Gippy, nodata results can even be set for each raster band, or entire image. Gippy also processes nodata values so that they stay as nodata pixels, even if the actual nodata value is changed when written to an output file. Given that “no data” values are extremely common in geospatial data and are often used to mask out invalid pixels or highlight missing data due to sensor issues, like broken scan line corrector mirrors, it is helpful that they are taken care of automatically.</p> <h4 id="process-chains-and-image-chunking">Process chains and image chunking</h4> <p>In order to efficiently do piecewise processing, Gippy chains together operations (e.g., +, -, log, abs) and applied the chain at one time, when the data is requested (upon a read or save operation).</p> <p>For example, to convert a landsat-7E TM+ image from radiance to top of the atmosphere reflectance, where theta is the solar zenith angle and sundist is the earth-sun distance:</p> <pre><code>green_toa = img['green'] * (1.0 / ((1812.0 * numpy.cos(theta)) / (numpy.pi * sundist * sundist))) </code></pre> <p>The 1812.0 is the exoatmospheric solar irradiance for the green band, as given in the Landsat handbook. This green_toa band can then be further processed, but none of the calculations anywhere in the chain will be performed until a read is requested through the read() or save() functions.</p> <pre><code># get a numpy array of the byte scaled green TOA reflectance green = green_toa.autoscale(1, 255).read().astype('byte') </code></pre> <h3 id="next-steps">Next steps</h3> <p>Improving Gippy is just the start. We are working on developing a suite of building blocks, we’re calling sat-utils, to provide easier ways to query, download, and process imagery into a variety of both visual and scientific products. Look out for more on this in the following weeks.</p> <p>In the meantime, check out the Gippy <a href="https://github.com/gipit/gippy">repo</a> and <a href="http://gippy.readthedocs.io">documentation</a>, and report any suggestions or problems. If you are at FOSS4G-NA, stop my <a href="https://2016.foss4g-na.org/session/gippy-high-performance-geospatial-image-processing-library-python">my talk on Gippy today</a> and say hi to Alireza and me.</p> 2016-05-03T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/03/gippy See you at FOSS4GNA http://www.developmentseed.org/blog/2016/05/02/foss4g-na <p class="dropcap">We’re headed down to Raleigh for <a href="https://2016.foss4g-na.org/">FOSS4G NA</a> - North America’s largest annual conference on Open Source Geospatial Software. We are big supporters of open geo, so we can’t wait for a week of stellar presentations and fascinating hallway conversations with hundreds of other open geo geeks.</p> <p>You can see some of what we’ve been working at two sessions, both on Tuesday at 5:40pm.</p> <ul> <li> <p><a href="https://developmentseed.org/team/anand-thaaker/">Anand</a> will demonstrate tools that we’ve developed for geographic analysis on vector tiles. This allows users to interact with the underlying data in vector tiles and perform sophisticated analytics instantly in their browser. (<a href="https://2016.foss4g-na.org/session/browser-scale-independent-geo-data-analysis-using-vector-tiles">302C - Tuesday @ 17:40</a>)</p> </li> <li> <p><a href="https://developmentseed.org/team/matt-hanson/">Matt</a> will present the latest release of GIPPY, a python toolkit for fast, distributed processing of large remote sensing data sets. (<a href="https://2016.foss4g-na.org/session/gippy-high-performance-geospatial-image-processing-library-python">304 - Tuesday @ 17:40</a>)</p> </li> </ul> <p>Also look for <a href="https://developmentseed.org/team/drew-bollinger/">Drew</a>, <a href="https://developmentseed.org/team/marc-farra/">Marc</a>, and <a href="https://developmentseed.org/team/alireza-j/">myself</a>. Hit any of us on twitter if you want to meet up.</p> <p>NB: We had a serious team discussion on whether to decline to attend FOSS4G NA in order to register our outrage over the new North Carolina law that encourages discrimination. The law runs counter to our values and isn’t conducive to the type of events we want to attend. The FOSS4G NA conference organizers <a href="https://2016.foss4g-na.org/ncga_hb2">have handled the situation remarkably well</a> as has <a href="http://www.raleighnc.gov/home/news/content/CorNews/Articles/MayorStatementHB2.html">the City of Raleigh</a>, which in the end, influenced our decision to attend.</p> 2016-05-02T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/02/foss4g-na Learning to Ask the Right Questions http://www.developmentseed.org/blog/2016/04/28/ask-the-right-questions <p class="dropcap">Building a successful product means knowing your users and building your product with them in mind. Unfortunately, many times this is overlooked. As a result, product teams make assumptions and their products end up unintentionally being geared toward them, instead of the end user. This problem can easily be solved though by simply by integrating <a href="https://www.nngroup.com/articles/definition-user-experience/">user experience (UX)</a> practices such as personas, and user testing. To demystify and simplify these integral processes and encourage others to integrate them into their work, we’ll dive into why they’re important, what they are and how to you can implement them, here and within future blog posts.</p> <p>If you don’t have an understanding of your users and their lives, how do you know what technology they have, what kind of internet may or may not be available or even if they have a need for the tool you’re building? People’s lives, wants and needs are all so different that knowing exactly what will serve those needs is incredibly hard without asking your audiences the right questions.</p> <p>Being able to ask and get meaningful answers to these questions doesn’t have to be hard or time consuming: it just involves integrating research, care and empathy into each and every project. Here at Development Seed, we do this by building purposeful UX design practices into our projects up front, so that we’re thinking about and building for our users throughout the life of a project. Because we have a great understanding of our users from the beginning, it saves us time from potentially having to rebuild large parts of the product down the road. Also, knowing what our users want allows us to make informed design decisions, making this process easier and more staightforward.</p> <p>Of course every project that we do is different but the two UX practices we feel are the most integral, and would like to cover, are personas and user testing.</p> <p><strong>Personas</strong> - <a href="https://en.wikipedia.org/wiki/Persona_(user_experience)">Personas</a> are archetypes of who the users of your platform will be. There are many different methods for figuring out who your users are and/or who they should be, from <a href="https://www.google.com/analytics/">google analytics</a> to <a href="https://hbr.org/2009/03/ethnographic-research-a-key-to-strategy">ethnographic research</a>. Whichever you choose, building personas allows you to more deeply consider the lives of your users, be empathic and think about things from their perspective, making your product better geared toward them.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/user-personas.jpg" style="border: 1px solid #D8D8D8;" alt="Image" width="1440" height="360" /> </figure> <p><strong>User Testing</strong> - <a href="https://en.wikipedia.org/wiki/Usability_testing">User testing</a> is asking a relevant user to do tasks on a product and seeing how they perform. Whether you do it through a platform like <a href="https://www.usertesting.com/">usertesting.com</a> or use other methods like <a href="https://www.nngroup.com/articles/remote-usability-tests/">moderated usablity testing</a> as we do, any testing is good testing. Not only are you able to collect both qualitative (preferential) and quantitative (raw numerical) data, giving you the best insight into what needs to be changed, but also seeing first-hand how users interact and react to your product is invaluable.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/usertesting-screen.jpg" style="border: 1px solid #D8D8D8;" alt="Image" width="1440" height="360" /> </figure> <p>Over the next month watch for our posts that will dive deeper into these processes, using our own Development Seed site as an example. Want to join the process? We’ll be looking for people to participate for a half hour sometime during the month of May. Just fill out the following survey, <a href="http://ds.io/uxing-devseed">http://ds.io/uxing-devseed</a>, so that we can learn a little about you before we dive into the specifics.</p> 2016-04-28T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/04/28/ask-the-right-questions Alley Lyfe http://www.developmentseed.org/blog/2016/04/27/alley-lyfe <p class="dropcap">Even though a few months have passsed since <a href="https://developmentseed.org/blog/2015/10/16/a-new-alley/">our move to Blagden Alley</a>, we’ve only just finally unpacked the last of the boxes. I’m sure you can relate.</p> <p>We miss the garage, but our new place really feels like home. Come on by for a visit. If you come late in the day you might catch us on the deck enjoying the grill that we got as an office warming gift from <a href="https://astrodigital.com/">one of our favorite clients</a>. Here are a few photos of our new home, as only the talented <a href="https://www.instagram.com/zatiana/">Tatiana</a> can capture.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos1.jpg" alt="Image" width="1440" height="360" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos2.jpg" alt="Image" width="1440" height="360" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos3.jpg" alt="Image" width="1440" height="360" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos4.jpg" alt="Image" width="1440" height="360" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos5.jpg" alt="Image" width="1440" height="360" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos6.jpg" alt="Image" width="1440" height="360" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/office-photos7.jpg" alt="Image" width="1440" height="360" /> </figure> 2016-04-27T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/04/27/alley-lyfe Auditing India's Power Grid with Satellites http://www.developmentseed.org/blog/2016/04/04/india-electrification <p class="dropcap">Two weeks ago I visited Delhi with a team from the World Bank to launch <a href="http://nightlights.io/">India Lights</a>, a tool that uses nighttime satellite imagery to track electrification in India’s rural villages. India Lights visualizes light output for over 600,000 villages, using nightly captures over a period of 20 years. Using freely available data, we aim to provide the tools to shine more light on the country’s electrification efforts.</p> <p>Electrification is a top priority for the Indian government. In his Independence Day Speech a year ago, Prime Minister Narendra Modi committed to electrifying 18,452 villages within 1000 days. By the government’s own accounting, 7,696 villages have been connected in the past year. However, <a href="http://www.thehindu.com/opinion/op-ed/on-paper-electrified-villages-in-reality-darkness/article8397038.ece">some report the actual number is lower</a>, due to clerical errors, lack of followup, and poorly implemented electrification. Although the Ministry of Power tracks their efforts, their reliance on site visits and reports from affected villages makes monitoring their large territory understandably challenging.</p> <p>On the other hand, to a satellite, remote villages are as accessible as urban centers. Using nearly 6 billion individual light readings from the National Oceanic and Atmospheric Administration’s (NOAA) weather satellites, we derive a unique light output curve for each of India’s states, districts, and 600,000 villages. This lets users track the light output of individual villages and how it changed, and see which villages participated in rural electrification projects and how they’re doing. All of this is possible without humans in the field.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/india-lights-compare.png" alt="Examining nighttime lights on indialights.io" width="1440" height="360" /> </figure> <p><em>Compare two different points in time at <a href="http://nightlights.io/">nightlights.io</a>.</em></p> <p>Though satellite cover a lot of ground, there are issues. Cloud cover, especially during the summer monsoon season, can wash out days or weeks of recordings, and small amounts of indoor light may not show up. Accordingly, a tool like nightlights works best when supplemented (and calibrated by) field-based methods, making for better, smarter monitoring. To learn more about the challenges we faced working with India Lights data, <a href="http://india.nightlights.io/#/about">see our full post</a>.</p> <p>What’s next? We’re currently working to add data beyond 2013, and incorporate imagery from newer, more precise satellites. Also, we’ll be working with the Ministry of Power to see how they might incorporate daily nighttime images into their electrification monitoring system.</p> <p>In the meantime, check out the <a href="http://nightlights.io/">new UI on India Lights</a>, stay tuned for updates as we start pulling in more recent data, and check out the <a href="http://api.nightlights.io/">open API</a> that powers the platform.</p> 2016-04-04T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/04/04/india-electrification Government of Vanuatu opens drone imagery on OpenAerialMap http://www.developmentseed.org/blog/2016/03/31/vanuatu-imagery-on-oam <p class="dropcap">The Government of Vanuatu has just released new high-resolution aerial imagery into <a href="http://beta.openaerialmap.org/">OpenAerialMap</a>. After Cyclone Pam, a Category 5 storm that caused major damage across Vanuatu in March 2015, assessment teams needed access to imagery to understand the extent of the damage. The Vanuatu government worked with the World Bank and the <a href="http://uaviators.org/">Humanitarian UAV Network (UAViators)</a> to use humanitarian drones to collect imagery in over 30 areas of the islands. We’ve worked with the teams over the last week to start to import the imagery and <a href="http://beta.openaerialmap.org/#/169.8541259765625,-17.88465917954279,7">make it available on OAM</a>.</p> <figure class="bleed-full"> <iframe width="100%" height="500px" frameborder="0" src="https://devseed.com/labs/vanuatu/"></iframe> <figcaption>Some of the hardest hit areas that were surveyed.</figcaption> </figure> <p>Tools like OAM to search and access drone imagery matter when conducting assessments. The damage assessment teams in Vanuatu needed quick access to imagery to understand the extent of the damage. OAM is now providing the tools to search and find this imagery to make it easier for future assessment teams to focus more on the analysis and less on the challenges of sharing large datasets.</p> <p>We’ve just started the import so expect to see more imagery online soon. Thanks to <a href="https://twitter.com/PatrickMeier">Patrick Meier</a> and Keiko Saito for helping import the imagery into OAM and also to the UAV teams, Heliwest and X-Craft, for carrying out the UAV flights. See all the <a href="http://beta.openaerialmap.org/#/169.27459716796875,-18.81271785640776,8">new images across the islands of Vanuatu on OAM.</a>.</p> 2016-03-31T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/03/31/vanuatu-imagery-on-oam Building an Open-Source Community http://www.developmentseed.org/blog/2016/03/30/open-source-community <p class="dropcap">Building a community around an open-source project is critically important to its success so that it can grow and stay relevant over time. However, figuring out how to start laying the groundwork for a vibrant community is not always easy or straightforward and is often the last thing a developer thinks about. When you get it right though, the benefits to the overall project quality, usage and impact can be immense. Here at Development Seed this is a challenge we’ve faced many times with projects like <a href="https://openaq.org">OpenAQ</a>, <a href="http://prose.io/">Prose</a>, <a href="http://openaerialmap.org/">OpenAerialMap</a> and <a href="https://github.com/developmentseed/landsat-util">landsat-util</a>. We’ll show you how to include people interested in helping your project progress and also how to deal with common issues like promoting community involvement and handling project management.</p> <p>So how do you get people to contribute; growing your community and your project?</p> <h4 id="give-people-a-problem-to-solve">Give people a problem to solve</h4> <p>The single biggest way to get people involved is to motivate them with an interesting problem or by providing them with access to tools/data that were previously unavailable. Hopefully, you have an interesting problem that you’re trying to solve and will help a lot of people, if not, why are you doing it? That’s all to say, people should be interested in what your project is trying to achieve.</p> <h4 id="show-people-what-needs-done">Show people what needs done</h4> <p>Once people are invested in the idea of your project, give them obvious ways to contribute. Have a maintained issues queue (e.g., <a href="https://github.com/">GitHub</a>, <a href="https://trello.com/">Trello</a>, <a href="https://asana.com/">Asana</a>) where visitors can see what they can help out on and potentially a project timeline so everyone knows the plan for upcoming work. Also, remember on many projects, there are often different types of tasks needed. Not everyone is a highly-capable JS/Python/Ruby developer, but they might be an awesome designer and may be able to provide a better design for your landing page. Or they might be willing to talk about your project via blog posts; or focus on building out your test coverage; or upgrade your documentation and examples. On most projects, there are different types of tasks and good project management can let everyone contribute.</p> <h4 id="make-it-easy-to-get-started-and-show-people-how-to-contribute">Make it easy to get started and show people how to contribute</h4> <p>Contributors will be coming from different backgrounds, coding styles and companies, so tools that help enforce standards are very beneficial to the overall continuity in the project and lead to a better experience for contributors. Make your system easy to stand up, develop on and contribute to. This means including things like:</p> <ul> <li> <p>Contributing guides<br /> Contributing guides are very useful in letting visitors know <em>how</em> they are expected to contribute. Do you require tests to be written or test coverage to increase, before code can be committed? Do you require every code change to be tied to an issue in a queue? Because all projects are slightly different, contributing guides can help remove any mystery. As examples, check out our contributing guides for <a href="https://github.com/prose/prose/blob/gh-pages/CONTRIBUTING.md">Prose</a> and <a href="https://github.com/openaq/openaq-fetch/blob/develop/CONTRIBUTING.md">OpenAQ</a>.</p> </li> <li> <p>Tests<br /> Good tests are indispensable in helping contributors learn about functionality and make sure they’re not breaking anything when they suggest changes. This gives everyone more confidence in the code that is being submitted for review. In many cases, reading over the tests that have been written for a system are a great way for contributors to learn about its intended functionality. A full overview of the value of testing is outside the scope of this post, but take a look at <a href="https://github.com/developmentseed/landsat-util/tree/develop/tests">landsat-util’s tests</a> as an example of a well-tested open-source project.</p> </li> <li> <p>Linters<br /> Linters are small programs that check code for stylistic or programming errors and help keep it internally consistent. With them, you can enforce syntactical rules across your project like spacing, semicolon usage or variable naming. Also, linters are great for uncovering lots of easy to miss, common programming errors. If you’re using JavaScript, <a href="http://eslint.org/">eslint</a> is a good place to start.</p> </li> <li> <p>Style guides<br /> A style guide is a reference for the visual components of your site. If someone wants to contribute a new page with a form, how should that form look? How are the labels and tooltips handled? What are the primary colors? Ideally, there is one place where everyone can look and see what styles should be used across the site. For an example of an awesome style guide, check out 18F’s work on <a href="https://playbook.cio.gov/designstandards/">U.S. Web Design Standards</a>.</p> </li> </ul> <h4 id="make-everyone-feel-a-part-of-the-community">Make everyone feel a part of the community</h4> <p>When trying to build an active community, it’s important to think about fostering long-term engagement from the very start by involving people early. One of the main ways to do this is through open communication channels. For the <a href="https://openaq.org">OpenAQ</a> project, we set up a dedicated <a href="http://openaq-slackin.herokuapp.com">Slack channel</a> where everyone can talk about general air quality issues, as well as ask specific questions about the platform. It’s been very helpful for consolidating communications and introducing newcomers to the existing community. Whenever gathering contributors, either virtually or physically, it’s a good idea to explicitly set community standards with a code of conduct. This is becoming more common for open-source projects and a few popular ones are from <a href="https://www.djangoproject.com/conduct/reporting/">the Django project</a>, <a href="https://www.python.org/community/diversity/">Python</a> and <a href="http://www.ubuntu.com/about/about-ubuntu/conduct">Ubuntu</a>.</p> <p>Gathering contributors in person can be helpful to either just meet each other or to do mini-sprints to work on certain features. However, this is often difficult due to the geographically distributed nature of most teams. For both OpenAerialMap and OpenAQ we haven’t been able to meet in person frequently, but have been able to get people together at events community members were already attending, like State of the Map or the American Geophysical Union conference.</p> <h4 id="be-honest">Be honest</h4> <p>For most of us, working on an open-source project isn’t our full-time job. When issues arise (and oh they’ll arise), try to be honest with people about how and when you can go about fixing them, even if it’s going to be a month before you can. It’s better to set reasonable expectations, than leave someone wondering why their contributions are being ignored.</p> <hr /> <p>By giving people a problem to solve, showing what needs done and how to help and being inclusive and honest, you’ll be on the right path to getting your community off the ground and headed in the right direction. All of this can be difficult at times, but building an active and diverse community is incredibly rewarding and will make you see your project in new ways. From getting feedback from users you never even knew would be interested in the project, to being alerted to flaws you had never considered. Ultimately, all of this will greatly raise the potential for your project to have an impact, which is the real goal.</p> <p>Want to learn more about how to build an active and diverse community? There were a number of topics we lightly touched on here. Watch out for future posts that talk about them in more detail.</p> 2016-03-30T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/03/30/open-source-community Designing for Live Election Data http://www.developmentseed.org/blog/2016/03/15/tracking-live-events <p>Real-time data is great. Building real-time interfaces is hard.</p> <p>We are working with The Washington Post to build tools for the company’s live election coverage of the US primary elections. The Washington Post wanted to create a experience that was informative and highly visual and aimed at readers following election coverage on a mobile phone or tablet device and often while watching another screen. Our challenge was to build a real-time data app that engages users and make it easy to track the most important information. Inspired by interactions and visual presentation from fantasy football apps, we designed tools for live election tracking that make it clear when the data is changing and provide contextual information to make these changes understandable.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/wapo-rep-super-tuesday.png" alt="Image" width="1440" height="360" /> </figure> <h3 id="reporting-live-events-through-data">Reporting live events through data</h3> <p>We noticed similarities between the football games were were tracking and the primaries we were building for. There is a winner. The results unfold over 2-4 hours. There are close games and blowouts. These situations can translate into interesting events during a primary race. For example, when the lead frequently flips or when a candidate outperforms expectations.</p> <p>With any time based event, what piques a user’s interest isn’t necessary what happens during one point in time but how the data or information changes moment-by-moment. Sports apps address these challenges on a daily basis. We formed a small fantasy football league, to practice tracking multiple games simultaneously through apps and websites. We evaluate how these apps informed users of real-time updates. A regular NFL Sunday became our preparation for Super Tuesday. We were frustrated by the apps that made it hard to track multiple games simultaneously. We loved the apps that showed us what had happened since the last update. The best apps had a clear hierarchy of information. They showed an overview of all the games but allowed you to easily dig in and flip between games. The best apps were selective about what data to show at what level. They let you know where the ball is, how much time is left, who is in ‘the red zone’, and what games are worth watching.</p> <p>We translated our findings from football tracking to election tracking. Election data can be overwhelming when results come in all at once from thousands of counties, across multiple states, for a plethora of candidates. We started with a clean interface that was selective about what data to show at the national and state level.</p> <h3 id="highlighting-new-information">Highlighting New Information</h3> <figure class="align-center"> <img src="/assets/graphics/content/prose/super-tuesday.gif" alt="Image" width="1440" height="360" /> </figure> <p>To make sure users are always up to date on the most important action, just like in the fantasy football apps, we introduced a design tool to highlight counties where something meaningful was taking place. If the leading candidates are within 1% of each other, a candidate has just overtaken another candidate, or the race has been called, that county receives a highlight on the map and an annotation in the tooltip. This was a way to highlight the ‘red zone’ in an election race. It offers a clean, non-intrusive way to help the Post’s readers make a little more sense out of the results coming in and to not miss any of the action.</p> <h3 id="context-layers">Context layers</h3> <p>The main map view shows who is winning in each county. This has the advantage of being extremely easy to understand and track over time. The disadvantage of this kind of map is that it can misrepresent the importance of large, rural counties with few votes. To compensate for this we added a population layer to give users more context for results. We used a light population stippling to give users a clean visual cues of which counties would have more votes. We continuously adjusted color palette, opacity, and shade of the data layer and the population stripling to make sure this layer stood out without mudding the map. We started with more saturated colors but ended up going with much lighter tones by Super Tuesday to allow the population layer to really shine through.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mass.png" alt="Image" width="1440" height="360" /> </figure> <p>With many republican candidates and a tight democratic race, precise maps and a focused narrative are more important than ever. We hope the population layer and real-time curated updates give users the right visual cues to give them a better understanding of election data coming in night by night. You can try the app tonight <a href="https://www.washingtonpost.com/2016-election-results/us-primaries/">on The Washington Post website</a> while you track results from Florida, Illinois, Missouri, North Carolina, and Ohio.</p> 2016-03-15T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/03/15/tracking-live-events AWS Lambda Functions http://www.developmentseed.org/blog/2016/03/08/aws-lambda-functions <p class="dropcap">When people talk about “the cloud”, Amazon Web Services (AWS) is often a big part of what they mean. AWS allows you to run complex web applications on a vast array of cloud computers around the world. Most web developers are familiar with AWS’s S3 buckets for file storage and EC2 instances for running applications. However, there is more beyond S3 and EC2–higher level services that can greatly simplify cloud services, at a cheaper cost, and requiring little maintenance.</p> <p>We recently worked with the <a href="http://www.missingmaps.org/">Missing Maps</a> OpenStreetMap Project to create <a href="https://developmentseed.org/blog/2016/03/01/osm-stats-launched/">real-time user pages, badges, and leaderboards</a> for all Missing Maps contributions to OSM. We built <a href="https://github.com/AmericanRedCross/osm-stats">software to track all OSM contributions</a> and an infrastructure on AWS to process and aggregate this data in real time. Most of the contributions for the Missing Maps project occur during mapathons where hundreds of volunteers submit edits and additions over a couple of hours. This means that the system needs to handle large spikes of activity when thousands of edits are added. Outside of mapathons there is far less activity. Because of these spikes and lulls, we wanted an alternative to constantly running multiple EC2 instances. Lambda functions proved to be the optimal solution to solve this problem.</p> <p>Here we will show you the benefits of using Lambda and how to implement and deploy Lambda functions on AWS.</p> <h3 id="arent-lambda-functions-a-insert-most-any-language-here-thing">Aren’t lambda functions a [insert most any language here] thing?</h3> <p>The term originates from the lambda calculus, a logic system developed in 1936 by Alonzo Church, a mathematician and <a href="https://en.wikipedia.org/wiki/Alan_Turing">Alan Turing</a>’s Doctoral advisor. Developers will likely be familiar with the term lambda function since it has become a feature of most major programming languages. A lambda function is an anonymous function (that is, it has no name), that takes in a single variable input. They are used when a function is made for a one-time or limited use and are able to be described with a more concise syntax than a regular function.</p> <figure class="align-center"> <a href="https://en.wikipedia.org/wiki/Alonzo_Church"><img src="/assets/graphics/content/prose/Alonzo_Church.jpg" alt="Who is the real Alonzo Church?" /></a> <figcaption>Alonzo Church, explaining the <a href="https://en.wikipedia.org/wiki/Entscheidungsproblem">Entscheidungsproblem</a></figcaption> </figure> <h3 id="aws-lambda-functions">AWS Lambda functions</h3> <p>An AWS Lambda function is a collection of code with a single entry point, or handler, and can be written in either Node.js (0.10.26), Python (2.7), or Java (8). While AWS Lambda functions do take a single input, they are of course not anonymous functions or else there would be no way to invoke them from the outside world. A Lambda function, generally, can be thought of as a simple function taking a single input and performing some transformation. Yet they are not restricted to simple operations. They can be quite complex, since libraries can be uploaded to the function. The Lambda function works by being configured to fire in response to some event, such as data added to a Kinesis stream, or files uploaded to an S3 bucket. Behind the scenes, Amazon stores the Lambda function code and configuration on S3 and when an event fires, it creates (or possibly reuses) a container, and passes the event data to the function handler.</p> <p>With this architecture AWS handles the management of the function and gives users three main advantages:</p> <h5 id="serverless-simple">Serverless (simple)</h5> <p>Lambda functions don’t require the maintenance of a server. The resources a lambda function uses is set only by specifying it’s memory usage, from 128 MB to 1536 MB. CPU speed is scaled as memory goes up and the memory allocation can be changed at any time.</p> <h5 id="auto-scaling-smart">Auto-scaling (smart)</h5> <p>There is no scaling to enable or to configure. Lambda functions spawn as necessary to keep up with the pace of events, although bandwidth may be restricted depending on other services accessed (e.g., bandwidth to a Kinesis stream is based on number of shards in the stream).</p> <h5 id="cost-effective-cheap">Cost-effective (cheap)</h5> <p>Compared to an EC2 instance which is up 24/7 and incurring costs, you are only charged based on how much the function runs, with no costs when it is not running. Furthermore, a function that has been allocated 512 GB of memory only costs 3 cents for every hour of computation. For a function that takes 1 second, those 3 cents can buy you over 3000 invocations of your function. Even for large scale operations, that are performing millions of functions a month, the cost will still typically be less than $20.</p> <h3 id="deploying-lambda-functions-with-python-and-boto3">Deploying Lambda functions with Python and boto3</h3> <p>The <a href="https://console.aws.amazon.com/lambda/home">AWS console</a> is a great way to get started on Lambda functions. It steps you through the process of creating one and includes templates for different languages. However the AWS console lacks an automated way to add new functions and update existing ones. The <a href="https://aws.amazon.com/cli/">AWS CLI</a> (command line interface) can be automated, however when the Lambda functions are complex (for instance, when they require event source mappings and IAM roles and policies) doing these actions in Python scripts using <a href="https://github.com/boto/boto3">boto3</a> is easier.</p> <p>Using a simple example, I’ll demonstrate how to use boto3 to create a function with an associated role and policies, and how to update that code via Python scripts. We’ll start with a simplified version of one of the AWS Lambda example templates in the file lambda.py.</p> <p>{% highlight python %}<br /> import base64<br /> import json</p> <p>def lambda_handler(event, context):<br /> print(“Received event: “ + json.dumps(event))<br /> for record in event[‘Records’]:<br /> # Kinesis data is base64 encoded so decode here<br /> payload = base64.b64decode(record[‘kinesis’][‘data’])<br /> print(“Decoded payload: “ + payload)<br /> return ‘Successfully processed {} records.’.format(len(event[‘Records’]))<br /> {% endhighlight %}</p> <p>Next, create a zip file containing the code including any additional dependencies needed in the same directory (boto3 is already included).<br /> <code> $ zip lambda.zip lambda.py </code></p> <p>We’ll then use a python script to create a Kinesis stream, an IAM access role, create a Lambda function with the zip file, then finally map the stream to the Lambda function.</p> <p>{% highlight python %}<br /> #!/usr/bin/env python</p> <p>import time<br /> import json<br /> import boto3</p> <p>kinesis = boto3.client(‘kinesis’)<br /> iam = boto3.client(‘iam’)<br /> l = boto3.client(‘lambda’)</p> <p>def create_stream(name):<br /> “”” Create kinesis stream, and wait until it is active “””<br /> if name not in [f for f in kinesis.list_streams()[‘StreamNames’]]:<br /> print ‘Creating Kinesis stream %s’ % (name)<br /> kinesis.create_stream(StreamName=name, ShardCount=1)<br /> else:<br /> print ‘Kinesis stream %s exists’ % (name)<br /> while kinesis.describe_stream(StreamName=name)[‘StreamDescription’][‘StreamStatus’] == ‘CREATING’:<br /> time.sleep(2)<br /> return kinesis.describe_stream(StreamName=name)[‘StreamDescription’]</p> <p>def create_role(name, policies=None):<br /> “”” Create a role with an optional inline policy “””<br /> policydoc = {<br /> “Version”: “2012-10-17”,<br /> “Statement”: [<br /> {“Effect”: “Allow”, “Principal”: {“Service”: [“lambda.amazonaws.com”]}, “Action”: [“sts:AssumeRole”]},<br /> ]<br /> }<br /> roles = [r[‘RoleName’] for r in iam.list_roles()[‘Roles’]]<br /> if name in roles:<br /> print ‘IAM role %s exists’ % (name)<br /> role = iam.get_role(RoleName=name)[‘Role’]<br /> else:<br /> print ‘Creating IAM role %s’ % (name)<br /> role = iam.create_role(RoleName=name, AssumeRolePolicyDocument=json.dumps(policydoc))[‘Role’]</p> <pre><code># attach managed policy if policies is not None: for p in policies: iam.attach_role_policy(RoleName=role['RoleName'], PolicyArn=p) return role </code></pre> <p>def create_function(name, zfile, lsize=512, timeout=10, update=False):<br /> “”” Create, or update if exists, lambda function “””<br /> role = create_role(name + ‘_lambda’, policies=[‘arn:aws:iam::aws:policy/service-role/AWSLambdaKinesisExecutionRole’])<br /> with open(zfile, ‘rb’) as zipfile:<br /> if name in [f[‘FunctionName’] for f in l.list_functions()[‘Functions’]]:<br /> if update:<br /> print ‘Updating %s lambda function code’ % (name)<br /> return l.update_function_code(FunctionName=name, ZipFile=zipfile.read())<br /> else:<br /> print ‘Lambda function %s exists’ % (name)<br /> for f in funcs:<br /> if f[‘FunctionName’] == name:<br /> lfunc = f<br /> else:<br /> print ‘Creating %s lambda function’ % (name)<br /> lfunc = l.create_function(<br /> FunctionName=name,<br /> Runtime=’python2.7’,<br /> Role=role[‘Arn’],<br /> Handler=’lambda.lambda_handler’,<br /> Description=’Example lambda function to ingest a Kinesis stream’,<br /> Timeout=timeout,<br /> MemorySize=lsize,<br /> Publish=True,<br /> Code={‘ZipFile’: zipfile.read()},<br /> )<br /> lfunc[‘Role’] = role<br /> return lfunc</p> <p>def create_mapping(name, stream):<br /> “”” add a mapping to a stream “””<br /> sources = l.list_event_source_mappings(FunctionName=name,<br /> EventSourceArn=stream[‘StreamARN’])[‘EventSourceMappings’]<br /> if stream[‘StreamARN’] not in [s[‘EventSourceArn’] for s in sources]:<br /> source = l.create_event_source_mapping(FunctionName=name, EventSourceArn=stream[‘StreamARN’],<br /> StartingPosition=’TRIM_HORIZON’)<br /> else:<br /> for s in sources:<br /> source = s<br /> return source</p> <p>name = ‘alonzo’</p> <h1 id="create-kinesis-stream">create kinesis stream</h1> <p>stream = create_stream(name)</p> <h1 id="create-a-lambda-function">Create a lambda function</h1> <p>lfunc = create_function(name, ‘lambda.zip’, update=True)</p> <h1 id="add-mapping-to-kinesis-stream">add mapping to kinesis stream</h1> <p>create_mapping(name, stream)</p> <p>{% endhighlight %}</p> <p>If it runs without error, you should have an <a href="https://console.aws.amazon.com/iam/home">IAM role</a> named alonzo_lambda, and a <a href="https://console.aws.amazon.com/kinesis/home">Kinesis stream</a> and <a href="https://console.aws.amazon.com/lambda/home">lambda function</a> named alonzo.</p> <p>Now, to test it. Use the AWS CLI to add a JSON record to the stream.<br /> <code> $ aws kinesis put-record --stream-name alonzo --data "{'0': 'the', '1': 'lambda', '2': 'calculus'}" </code></p> <p>You should get back JSON indicating the ShardId. Take a look at the logs in <a href="https://console.aws.amazon.com/cloudwatch/home">CloudWatch</a> to see if the function worked. Under Logs there should be an “/aws/lambda/alonzo” log group. Within that are different log streams. New log streams will be created periodically, but a single log stream may hold the logs from more than a single invocation of the function. You should see the output of the event and the decoded JSON we went into the stream.</p> <figure class="align-center"> <a href="https://console.aws.amazon.com/cloudwatch/home"><img src="/assets/graphics/content/prose/lambda_logs.png" alt="the lambda logs" /></a> <figcaption>Memory usage and duration to help plan memory allocation and expected costs</figcaption> </figure> <p>caveat emptor: While Lambda functions cost nothing when not running, it is typically used in a system where there are other resources which do cost something. Kinesis streams are charged per hour per shard. This example only creates a single shard so your alonzo stream will cost you $0.36 a day. When your done playing, make sure you delete the resource through the <a href="https://console.aws.amazon.com/console/home">AWS management console</a>.</p> <h5 id="security-considerations">Security Considerations</h5> <p>In order for Lambda functions to access other AWS services, such as S3, a Kinesis stream, or a Virtual Private Cloud (VPC), they must be granted access through policies using the Identify Access Management (IAM) service. A role should be created specifically for your Lambda function, and then policies added to that role to grant access to the services mentioned above.</p> <p>If your lambda function requires access to an EC2, RDS or other resource inside a VPC and those have restricted access, more work is required to access them with the Lambda function. Just granting a policy to allow VPC configuration is not enough (although that too is required). The Lambda function must be assigned at least one security group (as shown below), and then you must add an Inbound Rule to the EC2 or RDS security group to allow access to the Lambda function security group. To keep organized, it is recommended to create a security group specifically for each service (e.g., Lambda, EC2 instance) and give it the same name as your service name (e.g., alonzo_lambda).</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/lambda_vpc.png" alt="allow lambda VPC access" /> <figcaption>Assign a VPC, subnets, and at least one security group to the lambda function</figcaption> </figure> <p>A drawback when doing this is once ‘placed’ within your VPC, the function will be unable to access any other resource outside of your VPC (as if it is within the internal network). To gain access to the outside a NAT gateway must be set up to route responses back to the Lambda function.</p> <h3 id="conclusion">Conclusion</h3> <p>The example above uses a pull model where it pulls data from a Kinesis stream. Other services, such as S3, can invoke the Lambda function directly, which is the push model. See the Lambda documentation for a full list of <a href="http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html#intro-core-components-event-sources">event sources</a>. An example of how the push model works is when monitoring additions to an S3 bucket. To do this an event is added to the bucket directly and the Lambda function is invoked when the criteria is met. In addition to this, you could have the Lambda function automatically update metadata in a database, process the new data, or even deploy newly uploaded code to additional Lambda functions. The possibilities are endless.</p> <p>Breaking down problems into a series of functions that each operates on discrete variables is exactly what the lambda calculus was all about. Even if AWS Lambda functions aren’t lambda functions in the truest sense, no doubt Alonzo Church would find it in the spirit of the lambda calculus.</p> 2016-03-08T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/03/08/aws-lambda-functions Sentinel-2 imagery live for comparison http://www.developmentseed.org/blog/2016/03/03/sentinel-2-on-landscape <p class="dropcap">Sentinel-2 is now live on the <a href="http://landscape.satsummit.io/">Satellites in Global Development Landscape Report</a>. Sentinel-2 is the European Space Agency’s newest satellite providing open imagery data. Its 10m resolution imagery and 13 spectral bands make it particularly useful for tracking climate change, land monitoring, and disaster response. The <a href="http://landscape.satsummit.io/">Landscape Report</a> is a living document we’ve helped to build about the uses of satellite data for global development.</p> <figure class="bleed-full"> <a href="http://landscape.satsummit.io/"><img src="/assets/graphics/content/prose/sentinel-2-landscape.png" alt="Sentinel-2 on landscape report" /></a> </figure> 2016-03-03T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/03/03/sentinel-2-on-landscape OpenStreetMap Badges and Leaderboards http://www.developmentseed.org/blog/2016/03/01/osm-stats-launched <p><a href="https://www.openstreetmap.org/">OpenStreetMap</a> now has real-time user analytics. We’ve rolled <a href="http://www.missingmaps.org/users/">user profiles</a> and <a href="http://www.missingmaps.org/leaderboards/">leaderboards</a> into <a href="http://www.missingmaps.org/">Missing Maps</a>, a project to map the most vulnerable places in the developing world on OSM.</p> <h3 id="get-your-badges">Get your badges!</h3> <p>Badges reward users for different types of contributions, such as the number of buildings created or kilometers of road added. Every OSM user who contributes to a Missing Maps project now has badges for progress tracking.</p> <figure class="align-center"> <a href="http://www.missingmaps.org/users/"><img src="/assets/graphics/content/prose/mm-user.gif" alt="Check out Missing Maps user profiles" /></a> </figure> <h3 id="track-your-mapathons-with-leaderboards">Track your mapathons with leaderboards</h3> <p>Live leaderboards allow you to track contributions in real-time. They are great for powering a mapathon or tracking an activation. OSM users can now compare their contributions on live leaderboards.</p> <figure class="align-center"> <a href="http://www.missingmaps.org/leaderboards/"><img src="/assets/graphics/content/prose/mm-leaderboard.gif" alt="Check out Missing Maps user profiles" /></a> </figure> <p>Working with the <a href="https://github.com/AmericanRedCross">American Red Cross</a> we developed the infrastructure to make real-time analytics work. Since 2014, Missing Maps volunteers have made more than 22 million edits. Every contribution with a #missingmaps hashtag is now tracked in real-time. The infrastructure that powers Missing Maps has the potential for application at scale across OSM.</p> <p>The implementation code is all open source <a href="https://github.com/AmericanRedCross/osm-stats/tree/develop">on GitHub</a> and builds on <a href="https://developmentseed.org/blog/2015/09/28/whats-trending-osm/">our previous work</a> on real-time OSM tracking. We hope to roll out badges and leaderboards to more corners of OSM soon.</p> 2016-03-01T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/03/01/osm-stats-launched Ballot box stuffing in Uganda Elections http://www.developmentseed.org/blog/2016/02/29/uganda-elections <p class="dropcap">Uganda held general elections on February 18th, that were <a href="http://www.nytimes.com/2016/02/21/world/africa/yoweri-museveni-ugandas-president-wins-a-widely-criticized-election.html">widely criticized</a> by <a href="http://www.theguardian.com/global-development/2016/feb/22/ugandan-elections-polls-fraud-yoweri-museveni">media</a> and <a href="http://news.yahoo.com/ugandan-election-commission-lacks-independence-eu-observers-111925859.html">independent observers</a>. On Thursday, the Ugandan Electoral Commission released the <a href="http://www.ec.or.ug/sites/default/files/docs/01-Summary_PRESIDENT_1_Final22-FEB-2016.pdf">results in PDF</a>. Many concerns about the electoral process will not be evident in the results alone, however, analysis of the data can still be valuable in detecting some types of fraud.</p> <h2 id="evidence-of-ballot-box-stuffing">Evidence of ballot box stuffing</h2> <p>Our preliminary analysis found evidence of blatant ballot box stuffing in concentrated areas. We used <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3478593/">“election fingerprinting”</a> to quickly look at the distribution of election returns and identify anomalies.</p> <blockquote> <p>Election fingerprints are two-dimensional histograms of the number of units for a given voter turnout (x axis) and the percentage of votes (y axis) for the winning party (or candidate)</p> </blockquote> <figure class="align-center"> <img src="/assets/graphics/content/prose/uganda-fingerprint.png" alt="Ugandan election fingerprint" /> </figure> <p>Results in the upper right corner (very high turnout, very high percentage of votes for the winner) are clear signs of ballot stuffing. Bi-modal distributions can arise naturally in certain countries but generally don’t differ by average turnout and are often a sign of incremental fraud.</p> <p>Of the 28,010 voting stations, 130 of them had 100% voter turnout, 113 of which voted 90% or more for the eventual winner and incumbent, Yoweri Museveni (42,768 votes for him in these stations). 105 of these highly suspicious stations occurred in just 4 districts:</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/uganda-districts.png" alt="Ugandan districts" width="400px" /> <figcaption> Mbarara, Sembabule, Kiruhura, and Nakaseke</figcaption> </figure> <h2 id="open-election-data">Open Election data</h2> <p>This analysis was possible because the election data was provided at the polling station level by the Ugandan Electoral Commission. To support further analysis of the data, we parsed this PDF and converted it to a <a href="https://github.com/developmentseed/open-elections/tree/master/Uganda">machine-readable format</a> (CSV). Because of the size of the file, it was difficult to parse using <a href="http://tabula.technology/">Tabula</a> so we wrote custom scripts relying heavily on <a href="https://github.com/euske/pdfminer/">pdfminer</a>.</p> <p>Truly open elections mean providing timely, machine-readable, and easily accessible results. We look forward to seeing what you do with the data. Find it on <a href="https://github.com/developmentseed/open-elections/tree/master/Uganda">GitHub</a>.</p> 2016-02-29T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/02/29/uganda-elections