Drupal - Development Seed http://www.developmentseed.org Posts related to Drupal. en Empowering volunteer mappers with machine learning http://www.developmentseed.org/blog/2017/09/15/power-mapping-with-machine-learning <p class="dropcap">I’ll be at the <a href="https://www.hotosm.org/">Humanitarian OpenStreetMap Team</a> (HOT) Summit today, facilitating a discussion on integrating machine learning into their workflow.</p> <p><a href="http://www.openstreetmap.org/">OpenStreetMap</a> (OSM) is ripe for machine learning. As a community built map, its most valuable resource is the time and engagement of its volunteers. If OSM mappers were even 7% more efficient, they could map another Australia every year. [1]</p> <p>HOT’s work involves rapidly building the map around disaster events, such as its amazing efforts over the past two weeks mapping flood areas in South Asia, earthquake-affected Mexico, and the entire paths of Irma and Harvey. But HOT volunteers also spend long monotonous hours searching through open forests to map small villages in support of Malaria eradication projects or power lines in support of electrification projects.</p> <p>Machine learning can help in both of these cases, by directing volunteers attention to where it is most valuable or by directing them to tasks suited to their skill level and the time they have to contribute.</p> <h2 id="machine-learning-in-the-hot-workflow">Machine Learning in the HOT workflow</h2> <p>As a thought exercise, here is an example of what it could look like to augment HOT’s workflow with machine learning.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/HOT-ML1.jpg" /> </figure> <p>A HOT manager defines a new task and selects an area of interest. As she creates the new task she has the option of adding some context layers. These invoke a computer vision algorithm created using machine learning to quickly scan the imagery and create a layer that provides volunteers with more information about where to start mapping. For instance, this context layer might show an estimate of the number of buildings, the number of buildings that appear to be damaged, or the locations for settlements.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/HOT-ML2.jpg" /> </figure> <p>These context layers can be used to prioritize parts of the map or to help users select a square that fits their skills and available time. Mapping rural areas can involve monotonous time scanning through miles of forest looking for a settlement. Preprocessing can help direct attention where it matters.</p> <p><a href="https://twitter.com/blakegirardot">Blake</a> has an even better idea. The algorithm could try to assess the difficulty of mapping each square. Difficulty could be determined through a known heuristic, such as how close buildings are to each other. (Beginning mappers do better with detached homes.) Or data from the OSM validation process could be used to teach a computer what an easy cell or a hard cell looks like.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/HOT-ML3.jpg" /> </figure> <p>A volunteer mapper selects a high priority grid square and starts mapping. In order to save time, he selects a button that says “Suggest buildings”.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/HOT-ML4.jpg" /> </figure> <p>A machine learning trained process instantly provides suggestions of building outlines along with appropriate building tags. The user reviews this data; he adds, removes, and edits the suggestions as needed. When he is happy he saves the changes and pushes the changeset to OSM.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/HOT-ML5.jpg" /> </figure> <p>As mappers submit their changesets, a machine learning process automatically scans them looking for evidence of errors or vandalism. This allows a QA team, using a tool like OSMCha, to have smarter flagging and ordering of tasks in order to spend QA time on areas of greatest need. As the QA team reviews changesets, their actions help the process get smarter about what to flag and prioritize. In this way new “flavors” of mistakes can be quickly noticed and addressed.</p> <h2 id="what-do-you-think">What do you think?</h2> <p>If you are interested in the possibilities for machine learning to help HOT, join us today at 10 at the HOT Summit. There is a great group of people at the Summit who are already working on applying machine learning at scale. It’s going to be a great conversation.</p> <p>If you aren’t at the HOT Summit you can drop your thoughts in the <a href="https://docs.google.com/document/d/15eBrDgH5Mph86undSJ00u5bKdULEvwUWFuRucl0QNxE/">discussion notes</a> or on twitter using #HOTSummit.</p> <ol class="footnotes-list"> <li>Some quick numbers from <a href="https://twitter.com/kamicut">@kamicut</a>: Over the past year, OSM mappers have added 568,633,723 nodes and 65,193,364 ways. All of Australia currently mapped on OSM currently comprises around 40 million nodes and 3 million ways.</li> </ol> 2017-09-15T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/09/15/power-mapping-with-machine-learning Where Machine Learning Fits in HOT http://www.developmentseed.org/blog/2017/09/14/hot-summit-welcome <p class="dropcap">We’ll be at the Humanitarian OpenStreetMap Team (HOT) <a href="http://summit.hotosm.org/">Summit</a> to discuss how machine learning techniques can help improve the mapping and field data collection work that HOT provides to disaster aid workers.</p> <p>There’s tremendous potential for this. In the peak moments after a disaster, speed and accuracy are both crucial elements of a response. This is why we’ve worked on tools to <a href="https://developmentseed.org/projects/skynet/">automatically map regions</a> using open data and machine learning, and why we’re working with the World Bank to build a more user-friendly, offline-ready field data collection app.</p> <p>On Friday at 10am ET, <a href="https://developmentseed.org/team/ian-schuler/">Ian Schuler</a> will facilitate a community discussion on how we can <a href="http://summit.hotosm.org/program/#NewTech&amp;MachineLearninginHOTWorkflows">integrate similar machine learning approaches into the HOT workflow</a>.</p> <p>We’ll also be participating in panels and discussions on providing a source of <a href="https://developmentseed.org/projects/openaerialmap/">open drone and satellite imagery</a> to first responders, and improving the design and interface of the HOT tasking manager, a crucial tool that allows the organization to quickly and collaboratively map affected areas.</p> <p>We’ll be posting thoughts and reflections on <a href="https://twitter.com/developmentseed">Twitter</a>. If you’re attending, ping <a href="https://twitter.com/ianschuler">@ianschuler</a>, <a href="https://twitter.com/ascalamogna">@ascalamogna</a>, and <a href="https://twitter.com/felskia">@felskia</a>, to talk about machine learning, field data collection, and humanitarian response. Or just to grab a coffee.</p> 2017-09-14T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/09/14/hot-summit-welcome Zhuangfang NaNa Yi joins Development Seed http://www.developmentseed.org/blog/2017/08/25/welcome-zhuangfang-nana-yi <p class="dropcap">Join us in welcoming <a href="https://developmentseed.org/team/zhuangfang-nana-yi/">Zhuangfang NaNa Yi</a> to Development Seed! Zhuangfang will help refine our machine learning algorithms and distributed computing abilities. She finds insights in complex datasets using machine learning and multi-layer neural networks.</p> <p>Zhuangfang will begin contributing immediately on <a href="https://developmentseed.org/projects/skynet/">Skynet</a>, our machine learning platform for detecting features from satellite imagery and open data. She will be exploring new frameworks and approaches, and fine-tuning our current methods for better results.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/nana-blog.jpg" /> <figcaption> Zhuangfang singing karaoke!</figcaption> </figure> <p>Before joining Development Seed, Zhuangfang researched environmental science at the World Agroforestry Centre. Most recently, she was a data science fellow at The Data Incubator. She has <a href="https://scholar.google.com/citations?user=di3K1igAAAAJ">21 research publications</a> to date. She primarily codes in Python and R</p> <p>Fun fact: Zhaungfang was a traditional Dai (or Tai Lue) dancer. She also enjoys karaoke, running, and rock climbing in her spare time.</p> 2017-08-25T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/08/25/welcome-zhuangfang-nana-yi Introducing Geolambda http://www.developmentseed.org/blog/2017/08/17/geolambdas <p class="dropcap">Geolambda greatly simplifies the process to develop and deploy code that uses standard geospatial libraries. It takes the guesswork out of bundling native binaries with your AWS Lambda functions, so you can focus on building. We run Geolambda in production on projects for <a href="https://developmentseed.org/projects/nasa/">NASA</a> and <a href="https://developmentseed.org/projects/libra/">Astro Digital</a>.</p> <p>Geospatial processing code often depends on a small pool of standard libraries: GDAL, Proj.4, image format libraries, etc. Bundling these libraries as dependencies is often not well documented for cloud computing environments, leading to the trial and error of uploading new code and testing to get it right.</p> <p>This is a hassle. So we built Geolambda. At its core, Geolambda is a Docker image pre-loaded with standard geospatial libraries, and scripts to package them into a zip file that can be uploaded directly to AWS.</p> <p>We built Geolambda with <a href="https://developmentseed.org/blog/2016/03/08/aws-lambda-functions/">AWS Lambda</a> functions in mind. Essentially on-demand cloud computing, Lambda functions have proven capable of transparent and automatic scaling, terrific flexibility, and ease of maintenance, all while being cheaper than always-running alternatives. That said, Geolambda is for generating a portable code package, and you can deploy it to anywhere running Amazon-flavored Linux.</p> <p>I’m going to run through the step-by-step of how to use Geolambda. You can also browse the code on <a href="https://github.com/developmentseed/geolambda">Github</a>.</p> <h2 id="lets-make-a-geolambda">Let’s make a Geolambda</h2> <p>You can create a very simple Geolambda with just a few files, which I’ll detail below. A starter version of these files are included in <a href="https://github.com/developmentseed/geolambda/tree/master/geolambda-seed">the Github repository in a directory called geolambda-seed</a>, so you can follow along.</p> <h4 id="dockerfile">Dockerfile</h4> <p>First, you need a <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile</a> that <a href="https://github.com/developmentseed/geolambda/blob/master/geolambda-seed/Dockerfile">specifies the Geolambda image to use</a>:</p> <pre><code>FROM developmentseed/geolambda:full WORKDIR /home/geolambda </code></pre> <p>Geolambda Docker images are available on Docker Hub. There are <a href="https://github.com/developmentseed/geolambda#available-image-tags">a couple of tags you can choose from</a>, corresponding to how much of a pre-built environment you’re looking for. If you’re just trying this out or not sure what to use, use <code>developmentseed/geolambda:full</code>.</p> <h4 id="lambdalambdahandlerpy">lambda/lambda_handler.py</h4> <p>This is the code responsible for handling a Lambda event. The example below sets up a logger for each event, then calculates statistics for a file stored on AWS S3, given it’s S3 url.</p> <pre><code>import os import sys import logging from osgeo import gdal # add path to included Python packages path = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(0, os.path.join(path, 'lib/python2.7/site-packages')) # set up logger logger = logging.getLogger(__file__) logger.setLevel(logging.DEBUG) def handler(event, context): """ Lambda handler """ logger.debug(event) # read filename from event payload and get image statistics fname = event['filename'].replace('s3://', '/vsis3/') # open and return metadata ds = gdal.Open(fname) band = ds.GetRasterBand(1) stats = band.GetStatistics(0, 1) return stats </code></pre> <h4 id="testtestlambdapy">test/test_lambda.py</h4> <p>Testing Lambda functions can help you avoid <a href="https://sourcebox.be/blog/2017/08/07/serverless-a-lesson-learned-the-hard-way/">hairy situations</a>. The example file structure in geolambda-seed includes a directory for unit and integration tests, and starts you off with a <a href="https://github.com/developmentseed/geolambda/blob/master/geolambda-seed/test/test_lambda.py">dummy test</a>. As your handler grows, use this file as a starting point for future tests.</p> <h4 id="docker-composeyml">docker-compose.yml</h4> <p>With a Dockerfile, a handler, and tests, all that remains is to build the image and create a deployment package. This is easy to do with docker-compose. A <code>docker-compose.yml</code> file contains the recipe for building and testing your deployment.</p> <p>You’ll notice that <a href="https://github.com/developmentseed/geolambda/blob/master/geolambda-seed/docker-compose.yml">our example file</a> contains a couple different services. We’ll go over these later.</p> <h4 id="env">.env</h4> <p>The <code>.env</code> file contains any environment variables that may be used in your handler. Our example uses AWS services, and if you plan to as well, you’ll need this file, which <code>docker-compose</code> reads during development. Note, deployed Lambdas have environment variables set through the AWS console, or the AWS CLI.</p> <pre><code>AWS_ACCESS_KEY_ID=*id* AWS_SECRET_ACCESS_KEY=*access_key* AWS_DEFAULT_REGION=us-east-1 </code></pre> <p>If you’re using Git or another version control system, make sure not to commit this file!</p> <h3 id="running-docker-compose-services-for-testing-and-packaging">Running docker-compose services for testing and packaging</h3> <p>The docker-compose.yml file provdes several services. To access them, first build the Docker image:</p> <pre><code>$ docker-compose build </code></pre> <p>Then run one of the services:</p> <pre><code>$ docker-compose run *servicename* </code></pre> <ul> <li><strong>base</strong>: Starts the container and provides a bash shell for working with it interactively. This also mounts the current directory on the host.</li> <li><strong>test</strong>: Use the Nose library to run any tests in the test directory.</li> <li><strong>package</strong>: Runs the packaging script, which will collect needed libraries in the lambda directory, alongside <code>lambda_handler.py</code>, and creates a zip file of that directory.</li> <li><strong>testpackage</strong>: Runs your tests using the base Geolambda image, in order to test with just the files deployed.</li> </ul> <h3 id="deploy-it-and-run-it">Deploy it and run it</h3> <p>The Geolambda images contain a script to <a href="https://github.com/developmentseed/geolambda/blob/master/bin/lambda-package-base.sh">collect and zip everything you need to deploy to AWS</a>. This produces a zip file, which you can can either upload using the AWS console or with the AWS CLI. The below assumes you have already set up a Lambda function through the AWS console called “geolambda-stats” that uses the Python 2.7 runtime.</p> <pre><code>$ aws lambda update-function-code --function-name geolambda-stats --zip-file fileb://lambda-deploy.zip </code></pre> <p>Voila! You’ve deployed your Geolambda. Now you can run it from the command line, passing it any raster image file (supported by GDAL) stored on s3. It will return the statistics of the image.</p> <pre><code>$ aws lambda invoke --function-name geolambda-stats --invocation-type RequestResponse --payload '{"filename": "s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC80010022016230LGN00_B3.TIF"}' stats $ more stats [ 0, 22715, 7036.3087310300425, 6873.612118202581 ] </code></pre> <h3 id="beyond-simple-geolambdas">Beyond simple Geolambdas</h3> <p>If you have an existing Python project, you can easily incorporate Geolambda to run it on AWS. Drop the seed files in the top level of a Python project, so that <code>Dockerfile</code> and <code>docker-compose.yml</code> are alongside <code>setup.py</code>. Then modify your Dockerfile as such:</p> <pre><code>FROM developmentseed/geolambda:full # install app COPY . /build WORKDIR /build RUN \ pip install -r /build/requirements.txt; \ pip install . -v; \ rm -rf /build/*; WORKDIR /home/geolambda </code></pre> <p>This installs your Python package along with any required Python dependencies. Geolambda will automatically include these during packaging. To install other system dependencies beyond what is made available in Geolambda (ie, compiled C code), use <code>lambda-package.sh</code> in the geolambda-seed directory.</p> <h4 id="extending-geolambda">Extending Geolambda</h4> <p>AWS Lambda imposes a 50MB size limit to zip files you upload. Uncompressed (unzipped) files cannot exceed 250MB. This puts a severe constraint on the amount of additional code you can include, since standard geospatial libraries already create a 46MB deployment package. To squeeze out more space, you can extend Geolambda and adjust the underlying geospatial libraries.</p> <p>Many applications will more control over the configuration of the underlying libraries. Production applications should not use the Development Seed Geolambda repository directly from Docker Hub, as it will change over time. To ensure a consistent base image, we recommended forking the <a href="https://github.com/developmentseed/geolambda">GitHub repository</a> and creating your own geolambda images in Docker Hub. This would allow custom images, such as one that includes a specific set of GDAL drivers.</p> <h4 id="processing-at-scale">Processing at scale</h4> <p>We’ve been running Geolambda in production for a while now. Knowing our processing environment will just work has cut down our time-to-deployment, and reduced the surface area for bugs.</p> <p>Combined with cheap cloud storage and on-demand processing, we think the future is bright for impactful, open data applications running Geolambda. Read more about <a href="https://developmentseed.org/blog/2017/04/18/modis-on-aws/">how we’re using AWS to publish application-ready satellite images to the web</a>. We’d love to hear from you, and we’re also <a href="https://developmentseed.org/careers/jobs/">hiring</a>, so give us a shout at one of the links below.</p> 2017-08-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/08/17/geolambdas We're at FOSS4G this week http://www.developmentseed.org/blog/2017/08/14/foss4gwelcome <p class="dropcap">The Development Seed crew is in full force at <a href="http://2017.foss4g.org/">FOSS4G</a> this week. FOSS4G brings together leading experts in open source mapping and satellite image technology. We are proud to take part and further our commitment to building impactful, accessible open source tools and open data products.</p> <p>We’ll be presenting on Wednesday and Thursdsay.</p> <ul> <li><a href="https://developmentseed.org/team/drew-bollinger/">Drew</a> will be talking about the current state of <a href="https://developmentseed.org/projects/skynet/">Skynet</a>, a suite of open source tools for extracting useful data from satellite and drone imagery. <a href="http://foss4g.guide/#9M4SVbSJOpX62oLvzZ">End-to-End Geo Machine Learning</a>, 5pm Wednesday, August 16.</li> <li><a href="https://developmentseed.org/team/matt-hanson/">Matt</a> will discuss how to set up geospatial-aware lambda functions on AWS using Docker. <a href="http://foss4g.guide/#9M4SVbWv7hDeH6VcAz">Geospatial Lambda for scaleable, serverless geo-processing</a>, 2pm Thursday, August 17.</li> <li>Ask <a href="https://developmentseed.org/team/marc-farra/">Marc</a> about updates to <a href="http://www.missingmaps.org/">Missing Maps</a> and real-time OpenStreetMap analytics.</li> </ul> <p>Follow us at FOSS4G on <a href="https://twitter.com/developmentseed">Twitter</a>. If you’re attending, ping <a href="https://twitter.com/drewbo19?lang=en">@drewbo19</a>, <a href="https://twitter.com/GeoSkeptic?lang=en">@GeoSkeptic</a>, and <a href="https://twitter.com/kamicut?lang=en">@kamicut</a>, to talk machine learning, cloud computing, and disaster response. Or just grab a drink 🍺🍺</p> 2017-08-14T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/08/14/foss4gwelcome How to replace your API with vector tiles http://www.developmentseed.org/blog/2017/08/09/mapbox-query-data <p class="dropcap">We’ve written before about how pairing <a href="https://www.mapbox.com/vector-tiles/">vector tiles</a> with Mapbox GL can produce <a href="https://developmentseed.org/blog/2016/02/24/ad-next/">beautiful</a>, <a href="https://developmentseed.org/blog/2016/02/18/india-lights/">complex</a> maps. In recent projects, we’ve started using vector tiles to power the contextual charts and tables that we’d previously built a <em>separate</em> API to support, drastically lowering our deployment and maintenance costs, while minimizing the failure points in our infrastructure.</p> <p>For developers who maintain a separate API on pages that load vector tiles, re-using data from those tiles in elements <em>besides the map</em> can simplify deployment, lower costs, and prevent duplicate work. In this post I’ll walk through how to do that.</p> <h2 id="our-use-case">Our use-case</h2> <p>Development Seed’s task was to build an interactive website to allow exploration of a geographic dataset over time, similar in style and layout to <a href="http://india.nightlights.io/">our India Lights project</a>. We had quarterly measurements over ten years for most of America’s 32,000 <a href="https://www.census.gov/geo/reference/zctas.html">ZIP Code Tabulation Areas (ZCTAs)</a>, and wanted to use a line chart below the map to show temporal change.</p> <p>As you zoomed and panned throughout the map, we wanted to update which ZCTAs we showed in the graph; if you zoomed to the Los Angeles area, then the graph should show lines for 90210 and other 90___ ZCTAs, thus allowing you to spot temporal trends wherever you navigated.</p> <p>India Lights retrieves its chart data using a traditional REST API. Initially we considered a similar strategy, which would have meant building an API that recognized queries like <code>?state=ca</code>, <code>?countyFIPS=06037</code>, or <code>?bounds=[[33.90,-118.51],[34.17,-118.09]]</code>.</p> <p>However, our map’s vector tiles <em>already contained</em> all quarterly measurements for each ZCTA. Instead of building that API from scratch, couldn’t we re-use that data?</p> <p>Turns out, Mapbox GL JS facilitates this by giving us a method to query data on features that are currently visible, returing GeoJSON. With this GeoJSON in hand, it was simple to transform the <code>properties</code> of each feature into the time-series needed for the line chart.</p> <p>This strategy was effective and performant, and saved us both development and maintenance time. And there was now no server-upkeep cost to the client, since our vector tiles fit on free-tier <a href="https://www.mapbox.com/pricing/">Mapbox hosting</a>.</p> <p>Clearly, this solution doesn’t cover all use-cases. Many map-centric visualizations require charting or inspecting values that <em>aren’t</em> in the map’s viewport. Furthermore, having an API can make a platform more accessible and interoperable; for a larger project, even if you could get away with Mapbox GL JS queries at first, it may save time in the long run to design and build a database and API from the beginning.</p> <h2 id="technical-qa">Technical Q&amp;A</h2> <h3 id="how-do-i-get-my-geographic-data-into-vector-tiles">How do I get my geographic data into vector tiles?</h3> <p>If you have ESRI shapefiles, you’ll first have to convert them into GeoJSON format; <a href="https://ben.balter.com/2013/06/26/how-to-convert-shapefiles-to-geojson-for-use-on-github/">this can be done easily</a> using <a href="http://www.gdal.org/ogr2ogr.html"><code>ogr2ogr</code></a>, part of <a href="http://www.gdal.org/">GDAL</a>:</p> <pre><code class="language-bash">ogr2ogr -f GeoJSON -t_srs crs:84 "${YOUR_FILENAME}.geojson" "${YOUR_FILENAME}.shp" </code></pre> <p>With your data in GeoJSON format, <a href="https://github.com/mapbox/tippecanoe">Tippecanoe</a> can create an <code>MBTiles</code> file that’s ready to <a href="https://github.com/mapbox/mapbox-cli-py">upload</a> for Mapbox hosting:</p> <pre><code class="language-bash">tippecanoe -o "${YOUR_FILENAME}.mbtiles" "${YOUR_FILENAME}.geojson" mapbox upload "${YOUR_FILENAME}.mbtiles" </code></pre> <h3 id="what-are-these-mapbox-gl-js-queries-exactly">What <em>are</em> these Mapbox GL JS queries, exactly?</h3> <p>There are two query methods on Mapbox GL JS’s <a href="https://www.mapbox.com/mapbox-gl-js/api/#map"><code>Map</code></a> object. These allow access to the GeoJSON features of visible vector tiles.</p> <ul> <li><a href="https://www.mapbox.com/mapbox-gl-js/api/#map#queryrenderedfeatures"><code>Map.queryRenderedFeatures([geometry], [parameters])</code></a></li> <li><a href="https://www.mapbox.com/mapbox-gl-js/api/#map#querysourcefeatures"><code>Map.querySourceFeatures(sourceID, [parameters])</code></a></li> </ul> <p>A common use-case is calling <code>queryRenderedFeatures</code> with the coordinates of a user’s mouse, to retrieve GeoJSON features for the points, lines, and polygons at that location, as seen in <a href="https://www.mapbox.com/mapbox-gl-js/example/queryrenderedfeatures/">this Mapbox example</a>:</p> <pre><code class="language-javascript">map.on('mousemove', function (e) { var features = map.queryRenderedFeatures(e.point); ... }); </code></pre> <p>If <code>queryRenderedFeatures</code> is given a <code>null</code> geometry argument, it will search the map’s whole viewport. <code>querySourceFeatures</code> <a href="https://www.mapbox.com/mapbox-gl-js/example/query-similar-features/">functions similarly</a>, but will also return features that are hidden due to overlap or other styling restrictions.</p> <h3 id="what-are-the-limits-of-this-approach">What are the limits of this approach?</h3> <p>Including lots of <code>properties</code> on GeoJSON features increases vector tile size, which will slow page-load. Rendering tens of thousands of features can also cause latency in user interactions.</p> <p>There are also some gotchas to the Mapbox GL JS query methods. Depending on Tippecanoe’s settings, features can be duplicated or missed: by default, all tiles have some overlap with redundant features to improve rendering, and tiles at wide zooms will include only a <em>subset</em> of point features in order to improve client performance. Separately, if a line or polygon spans across a tile boundary, then that feature will be returned multiple times by a query.</p> <p>So make sure to stay safe and use responsibly. But if this approach <em>is</em> applicable, then save yourself from duplicate effort the next time you build a visualization alongside vector tiles!</p> 2017-08-09T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/08/09/mapbox-query-data Find us at IGARSS this week http://www.developmentseed.org/blog/2017/07/24/igarss <p class="dropcap">Development Seed is at the IEEE’s <a href="http://www.igarss2017.org/">International Geoscience and Remote Sensing Symposium</a>. IGARSS is one of the top conferences for advanced techniques for getting insight from imagery, from machine learning for feature extraction and change detection to advanced applications for agriculture, forestry, disaster response, and climate resilience.</p> <p>Ian and Alireza will present a paper that was authored with our colleagues at NASA on our work on <a href="https://developmentseed.org/projects/nasa/">Cumulus</a> a cloud framework to process, manage, and distribute massive imagery holdings. The session is on <a href="http://www.igarss2017.org/Papers/PublicSessionIndex3.asp?Sessionid=1048">Intelligence for Big Geospatial Data</a> today at 4:20 and features four other fantastic papers on extracting usable insights from big geospatial data.</p> <p>If you are at IGARSS, ping <a href="https://twitter.com/ianschuler">@ianschuler</a> and <a href="https://twitter.com/scisco7">@scisco7</a> to chat about building pipelines for imagery processing or to discuss using machine learning for detecting roads, power grid mapping, or disaster response.</p> 2017-07-24T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/07/24/igarss Max Grossman joins the team! http://www.developmentseed.org/blog/2017/07/13/welcome-max-grossman <p class="dropcap">We’re happy to welcome <a href="https://developmentseed.org/team/max-grossman/">Max Grossman</a> to our team! Max creates meaningful analysis with maps for humanitarian organizations, and has a knack for making readable disaster response maps. With his depth of knowledge about Open Street Map, Max is working on aggregating and displaying country-level statistics for the <a href="https://developmentseed.org/blog/2016/03/01/osm-stats-launched/">Missing Maps project</a>. He’s also helping us model disaster risk susceptibility.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/max_kayak.jpg" /> </figure> <p>Max values community, particularly the open mapping community. An avid OSM contributor, he has built tools like <a href="https://maxgrossman.github.io/visualizations/whovalidates/">who-validates</a>, a tool that seeks to understand why people map, and how to tailor the experience for them. As a student, he hosted student mapathons and used the Humanitarian Open Street Map leaderboard to encourage newcomers to participate. When he’s not mapping, he’s helping create green space in DC as a <a href="https://gwupstartblog.wordpress.com/2016/02/03/learning-from-experience-why-green-spaces-matter/">peace gardener</a>.</p> <p>Welcome to Development Seed, Max!</p> 2017-07-13T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/07/13/welcome-max-grossman We Remain Committed to Paris and The Planet http://www.developmentseed.org/blog/2017/06/01/climate-change <p class="dropcap">Obviously we are all extremely disappointed in today’s announcement that the US will withdraw from the Paris Climate Accord.</p> <p>Development Seed is driven by a deep belief in our responsibility to protect this Planet. We nearly <a href="http://global-climatescope.org/en/">always</a> <a href="https://developmentseed.org/blog/infoamazonia-maps-deforestation/">say</a> <a href="https://developmentseed.org/blog/2015/05/26/oci-launch/">yes</a> <a href="https://developmentseed.org/projects/landsat-util/">to</a> <a href="https://developmentseed.org/blog/2015/06/03/open-air-quality-data/">any</a> <a href="https://developmentseed.org/projects/grp/">project</a>, <a href="https://developmentseed.org/projects/nasa/">big</a> <a href="https://developmentseed.org/projects/global-adaptation-index/">or</a> <a href="https://developmentseed.org/blog/2011/dec/06/design-decisions-mapping-temperature-precipitation-durban/">small</a>, that we believe can make a meaningful impact on addressing Climate Change. We’ve been able to do this while building a successful business that creates jobs in the United States. The proceeds of our business fund research and development of new open source software for deriving insight from earth data. These free and open tools power the work of climate scientists, government agencies, and industry.</p> <p>Climate Change will be a defining condition of this century. We will continue to do whatever we can to understand our planet, and how human activities are changing our climate. With our partners and clients, we will look for solutions that improve peoples’ lives and our Blue Marble. Our experience shows that this is possible. Our friends at Bloomberg New Energy Finance and Climatescope <a href="http://global-climatescope.org/en/">show</a> that green energy can be good for the planet and good for business. The inspiring team at Carnegie Endowments Energy and Climate Program <a href="http://oci.carnegieendowment.org/">show</a> that <em>we can reduce greenhouse gas without changing oil consumption</em> simply by being smarter about what oil is used and for what purpose.</p> <p>Specifically we will:</p> <ul> <li>continue to support accessible, open earth data such as supporting Earth data products on AWS such as <a href="https://aws.amazon.com/public-datasets/modis/">MODIS</a> and <a href="https://aws.amazon.com/public-datasets/landsat/">Landsat</a> and <a href="https://developmentseed.org/blog/2017/01/30/NASA-on-the-cloud/">working</a> with NASA to move Earth Data streams closer to massive public computing capacity.</li> <li>build and maintain climate.developmentseed.org, an open API of critical climate datasets. (Or if someone else already built this, let us know and we will help.)</li> <li>create a Climate Science Fellowship that offers a competitive salary for a recent machine learning graduate to build open models for understanding climate data.</li> </ul> <p>Our team is made up of people from around this planet, from Pittsburgh to Porto. We know we have a job to do.</p> 2017-06-01T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/06/01/climate-change Eli Litwack joins the team! http://www.developmentseed.org/blog/2017/05/01/welcome-eli-litwack <p class="dropcap">Join us in welcoming <a href="https://developmentseed.org/team/eli-litwack/">Eli Litwack</a> to the team! Eli will bring fresh thinking to our work on machine learning and computer vision. He is experimenting with new approaches to extract knowledge from satellite imagery on projects ranging from water management to designing better energy infrastructure. Eli will work with us this Spring as part of his senior project at The Park School in Baltimore.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/eli-and-friends.jpg" /> </figure> <p>Eli is a natural born builder, and he uses his skills to create positive change in his community. He is the lead engineer on his school’s robotics team. He won the BitCamp hackathon for his assistive reading device <a href="https://devpost.com/software/hawkeyes-eb5v9z">HawkEyes</a>. Eli runs Computers for Kids, a Maryland charity that refurbishes computers and donates them to under-resourced schools, community centers, and families. Previously, Eli interned for the <a href="https://allchildrenreading.org/innovators/institute-for-disabilities-research-and-training-inc/">Institute for Disabilities Research and Training</a> and helped create Moroccan Sign Language Clip and Create, a video dictionary that is currently used in all Moroccan schools for the deaf.</p> <p>Park School encourages student interns to donate their pay to charity. Eli has elected to donate his earnings to the ACLU and Planned Parenthood.</p> <p>Welcome to the team, Eli!</p> 2017-05-01T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/05/01/welcome-eli-litwack End-to-End Satellite Imagery Analysis http://www.developmentseed.org/blog/2017/04/21/end-to-end-ml <p class="dropcap">Today we launched the alpha version of Skynet Scrub, a tool for cleaning the outputs of Skynet, our machine learning and recognition process. This is part of our larger effort to build end-to-end machine learning pipelines that deliver <strong>usable insights</strong> to humanitarian and global development organizations.</p> <p>Machine learning and recognition can quickly extract insights from vast amounts of satellite imagery. Currently, we use <a href="https://developmentseed.org/blog/2016/08/17/sotm-skynet/">Skynet</a> to <a href="https://developmentseed.org/blog/2017/01/30/machine-learning-learnings/">reliably extract roads and buildings</a> from satellite imagery. Skynet gets more capable every day, but the output is still not perfect, and in the less-developed regions where Skynet is most useful, the challenge is often finding good, rich training data.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/skynet-predictions.png" alt="Good predictions, but not there yet" /> </figure> <p><em>Our output still has the occasional spaghetti.</em></p> <p>We built Skynet Scrub so we could start using Skynet data sooner. It’s our take on a familiar tool, the map editor, and it’s focus is on cleaning the squiggly false positives from machine-learning algorithms.</p> <h2 id="automated-feature-extraction-is-in-beta">Automated feature extraction is in beta.</h2> <p>We’re pushing machine learning to pick out roads and buildings because it can reshape mapping. It could provide, for example, a roads planner an accurate map of the existing network, using data sent from a satellite that flew overhead yesterday.</p> <p>The results are getting good, but not accurate enough to use out of the box yet.</p> <p>For urban planners or first responders, any false positive in geographic data is unacceptable. Existing tools for working with geographic data can also be clunky or ill-fitting for the job.</p> <h2 id="made-for-cleaning">Made for cleaning</h2> <p>Skynet Scrub is for cleaning, and cleaning quickly. It has a small feature set by design, backed by strong hotkey support for power users. It supports completeness tagging, and knows to cache your work as you go. It runs in a web browser.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/skynet-scrub-action.gif" alt="Good predictions, but not there yet" /> </figure> <p>It’s powered by Skynet Scrub Server, a dead-simple API for geographic data.</p> <p>We’re working on multi-session support now, and filling out the roadmap. Skynet Scrub is still in it’s early stages, but we’re deploying it with some regional partners for field testing.</p> <p>We’re building in the open on Github, and welcome trackers and contributors at <a href="https://github.com/developmentseed/skynet-scrub">Skynet Scrub</a> and <a href="https://github.com/developmentseed/skynet-scrub-server">Skynet Scrub Server</a>.</p> <h2 id="part-of-a-whole">Part of a whole</h2> <p>As you may have noticed, we’ve been putting a lot of energy into extracting features from satellite images using machine learning. Taking an image and identifying roads, buildings, or other relevant features has tremendous potential for governments, nonprofits, and the private sector alike.</p> <p>To realize this potential, we need an open-source, web-based solution that’s end to end. And until the output is good enough to put directly into use, Skynet Scrub is a missing piece of this puzzle.</p> 2017-04-21T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/04/21/end-to-end-ml How We Publish Satellite Data on the Web http://www.developmentseed.org/blog/2017/04/18/modis-on-aws <p class="dropcap">Cloud computing has opened the door for satellite data to make a daily difference in the lives of farmers, emergency responders, policymakers, and many other professionals. Until recently, the size and complexity of satellite-derived data meant that it was only practical for governments and research institutions to invest in. Now, we have the power to process massive amounts of imagery as it is collected, and deliver insights to smartphones in real time.</p> <p>To accomodate these new users, imagery providers need to adapt their services. We recently worked with Amazon Web Services (AWS) and Astro Digital to design and build MODIS* on AWS, building with web and app users in mind. This article shares what we learned, and provides guidelines for other data providers looking to improve their digital distribution.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/modisscene.jpg" /> </figure> <p><em>A composite MODIS image of the northeastern United States. Cape Code can be seen to the east, New York City and Long Island to the south. It is created from 16 days of imagery centered around July 9th, 2016. This data is <a href="https://modis-pds.s3.amazonaws.com/MCD43A4.006/12/04/2016191/index.html">available on AWS</a>.</em></p> <p><a href="https://aws.amazon.com/earth/">Earth on Amazon Web Services</a> has been tremendously successful in making satellite imagery accessible. Landsat on AWS recorded over 1 billion requests in the first year. The majority of applications and machine learning pipelines that use Landsat now pull their data from AWS rather than USGS. While the data is the same, AWS, along with Frank Warmerdam and the Planet team, created an extremely thoughtful data structure based on feedback from end users. AWS has continued this throughout the Earth on AWS program. These lessons come from the combined experiences of putting dozens of data sets on the cloud.</p> <h2 id="build-for-applications">Build for Applications</h2> <p>AWS development enables access by building regular folder structures that applications can follow, query piecemeal, or crawl at scale. This enables an entire range of use, from real-time apps that directly access the data to indexing scripts that build vast libraries.</p> <h3 id="organize-in-buckets">Organize in Buckets</h3> <p>“Buckets” are units of storage on S3, a storage service of AWS. When needing standardized cloud storage they are a good place to start because they are extensible and scale well. Other AWS services within the same co-located AWS region (e.g., us-east-1) can access buckets quickly at reduced or no cost.</p> <p>The concept of buckets exist at other cloud providers and often carry similar benefits, just under different names.</p> <h3 id="use-predictable-names">Use Predictable Names</h3> <p>MODIS folders use the path PRODUCT/XX/YY/DATE, where XX and YY are tile designations, and DATE is the four digit year and three digit day of the year. So 2017001 would be the first day of this year.</p> <p>In theory, this means you can access MODIS on AWS without searching for it, provided you have the spatial location and capture time. Unfortunately, MODIS data files all contain a scene ID, which includes a “processed-at” timestamp. So while the folders were crawl-able, we had to rely on other methods for the data itself.</p> <h3 id="include-an-index">Include an Index</h3> <p>An index.html page provides an excellent landing page for both humans and machines and should show what files exist for a particular scene.</p> <p>MODIS filenames are irregular, so including an index.html at the root directory gives applications a way to access full URL’s for each file.</p> <h3 id="publish-thumbnails">Publish Thumbnails</h3> <p>MODIS scenes are 2400x2400 pixel resolution. They come with a 10% size thumbnail, so you can preview the image before spending time and resources getting the full-res version. On AWS, you’ll find this thumbnail alongside the full version.</p> <p>When working interactively with large files, as most imagery is, thumbnails save time and cost.</p> <h3 id="publish-metadata">Publish Metadata</h3> <p>MODIS directories on AWS includes full scene metadata in an XML file. Generating an additional JSON metadata file containing this information is relatively straightforward, and makes accessing the data easier. JSON is native to the web, and both faster and safer for applications to parse.</p> <h3 id="provide-smaller-files-when-possible">Provide Smaller Files (When Possible)</h3> <p>The larger the file, the more resources you need to work with it, the harder it becomes. MODIS on AWS publishes one GeoTIFF file for each band. These files are split off from the original data file. This lets you download just the ones you need.</p> <p>MODIS often comes with ancillary data such as quality bands, and these get their own GeoTIFF as well.</p> <p>For very large scenes, they could be re-tiled to break them into into smaller chunks. However, creating too many files can be counterproductive though, as there is overhead involved with accessing each datafile.</p> <h3 id="provide-overviews">Provide Overviews</h3> <p>Overviews are downsampled versions of the data that can speed up access at lower zoom levels (“zoomed out”). Not everyone needs these, so they’re better kept in separate files.</p> <p>MODIS on AWS provides compressed overviews for each band (but not ancillary bands) at 1/2, 1/4, 1/16, and 1/32 sizes. Including four overview levels provides lots of flexibility for remote access.</p> <h3 id="file-format-compression-and-tiling">File Format, Compression, and Tiling</h3> <p>GeoTIFF is an accessible format, supports decent compression, and can be internally tiled to improve remote access. While the most common, it’s not the only choice.</p> <p>Compression greatly reduces file sizes at the cost of additional overhead when reading the file. Depending on the file, this additional overhead can greatly outweigh gains in download time. Tiling can also improve read performance, since you read data in pieces.</p> <p>MODIS uses deflate compression to compress GeoTIFF files (and overview files), and uses a 512x512 internal tile size. As a general rule if only a piece of a tile is needed and is less than 20% of the image it may be better to read the data remotely, otherwise the file should be downloaded before accessing.</p> <h2 id="result-of-community-outreach">Result of Community Outreach</h2> <p>These practices come from the community. AWS organized calls with user groups to figure out which MODIS products were most useful to store, how much of an archive to build, and the best way to store them. They ended up with these:</p> <ul> <li><strong><a href="https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mcd43a4_v006">MCD43A4</a></strong>: The highest quality scientific data from MODIS sensors, available daily.</li> <li><strong><a href="https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mod09ga_v006">MOD09GA</a> and <a href="https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/myd09ga_v006">MYD09GA</a></strong>: Daily readings from bands 1-7.</li> <li><strong><a href="https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mod09gq_v006">MOD09GQ</a> and <a href="https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/myd09gq_v006">MYD09GQ</a></strong>: Bands 1 (red) and 2 (near-infrared).</li> </ul> <p>The archive is currently still backfilling. MCD43A4 will go back 5 years, while the daily products will work on a 30 day rolling archive. It currently takes 4-5 days from capture for NASA to process the data and make it available, at which point the data will be made available on S3.</p> <p>*Landsat and <a href="https://modis.gsfc.nasa.gov/">MODIS</a> are different types of photographic instruments. MODIS flies aboard two spacecraft: Terra and Aqua. Together these cover the earth every 1-2 days.</p> 2017-04-18T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/04/18/modis-on-aws Welcome Seth Vincent! http://www.developmentseed.org/blog/2017/04/13/welcome-seth-vincent <p class="dropcap">Welcome <a href="https://developmentseed.org/team/seth-vincent/">Seth Vincent</a>, the latest member of the Development Seed family! We are delighted to have Seth on the team. We have long admired Seth’s contributions to open source and civic tech communities. Seth is a co-founder and organizer of <a href="http://openseattle.org/">Open Seattle</a>. He helped build community around local civic issues by addressing them through technology.</p> <figure class="align-center"> <img src="/assets/graphics/content/stuffed-cover.jpg" /> </figure> <p>At heart Seth is a teacher and story teller. A journalism major, Seth builds interfaces that tell great stories with data. He has a passion for well-written documentation which has earned him much love in open source software communities. Seth created the NodeSchool workshop <a href="https://github.com/workshopper/javascripting">javascripting</a> and volunteered as a mentor at Seattle NodeSchool events.</p> <p>When Seth isn’t riding his bike to parks and writing JavaScript at picnic tables in Seattle, he’s either experimenting with making arcade machines or eating, cooking, or thinking about pizza. He really loves pizza.</p> 2017-04-13T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/04/13/welcome-seth-vincent Closing the Energy Data Gap http://www.developmentseed.org/blog/2017/04/05/energy-data-launch <p class="dropcap">Today, the World Bank launched <a href="https://energydata.info">energydata.info</a>, an initiative that addresses the global data gap in the energy sector. Energydata.info gathers over 180 highly curated datasets with a special focus on data from developing economies. The datasets range from grid infrastructure and energy resource potential, to household data that provides insight into energy demand.</p> <p>More than 2 billion people around the world have no, or only intermittent, access to electricity. Small scale and off-grid initiatives are transforming the energy sector and have great potential to solve the access gap, especially in developing economies. These small-scale initiatives require high quality, hyper-local data. Such data makes small-scale energy more viable by allowing energy companies to plan operations and raise funding, investors to accurately consider investment options, and policy makers understand where their interventions can generate most impact.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/energydata.png" /> </figure> <p>The World Bank’s energy group recognizes that this challenge can’t be solved by a single data drop by a single organization. They have put together an impressive and growing consortium of data partners. The team behind energydata.info is actively curating data, making sure that the datasets are relevant and represent the best in class. By publishing the data under the right open licenses, they ensure that the data can be adopted without hesitation by both the public and private sector.</p> <p>The technology behind energydata.info bolsters their active approach to data. Energydata.info is not a portal. We don’t expect users to merely find and download a static dataset, but to plug into a stream of data and analysis tools. Data can be accessed programmatically in web friendly formats like Vector Tiles. This allows us and other partners to <a href="https://energydata.info/apps">develop applications directly on top</a> of the platform. Energydata.info is built using entirely open source tools like CKAN. We look forward to seeing how the data evolves and how others imagine new ways to use the data.</p> 2017-04-05T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/04/05/energy-data-launch Open States, Open Community http://www.developmentseed.org/blog/2017/03/20/Open-States-Open_Community <p class="dropcap">This past Sunday, <a href="https://openstates.org/">Open States</a>, a volunteer open-gov project, hosted a full-day code sprint at Development Seed. With 15 attendees spending their entire day coding and sharing ideas, many significant PRs and a ton of additional progress was made towards their <a href="https://blog.openstates.org/post/whats-next-2017/">major goals for 2017</a>.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/open-states-group.jpg" /> <figcaption>Results: 100 developer-hours of code, and smiles all around!</figcaption> </figure> <p>Open States programmatically scrapes and structures data from all 50 state legislatures in America, plus DC’s and Puerto Rico’s. Legislator bios and contact information, bill text, votes, and committee memberships are compiled and released as free and open-source data through their website and API, which receive millions of hits each week.</p> <p>They are one of many great groups contributing to and strengthening the efforts of the open-source community, through civic initiatives and datasets, as a way to make governments more engaged and accountable.</p> <p>To get involved, join the <a href="https://openstates-slack.herokuapp.com/">Open States Slack</a>, follow the <a href="http://docs.openstates.org/en/latest/contributing/getting-started.html">quick-start guide</a>, and submit fixes for some of the <a href="https://github.com/openstates/openstates/issues">existing issues</a>.</p> <p>Development Seed will also host two other community contributors this week:</p> <ul> <li>Maptime <a href="https://www.meetup.com/Maptime-DC/events/238139343/">Meetup</a>: tutorial on the basics of data visualization in D3.js</li> <li>USGS <a href="https://www.eventbrite.com/e/post-geohackjam-mappy-hour-tickets-32854893872">Lightning Talks</a>: members of USGS will present the results of their two-day GeoHackJam</li> </ul> <p><em>Disclosure: I am one of the core developers on Open States.</em></p> 2017-03-20T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/03/20/Open-States-Open_Community SatSummit Highlights http://www.developmentseed.org/blog/2017/02/21/SatSummit-Highlights <p class="dropcap">At SatSummit, 231 global development and satellite industry experts joined us to map how we can do smarter development work with satellite data. The major satellite data providers (including NASA, ESA, DigitalGlobe and Planet) joined big data gurus (from AWS, IBM, and Orbital Insights) and dozens of global development organizations (including USAID, World Bank, Humanitarian OpenStreetMap Team, and World Resources Institute) for a truly inspiring day of discussion.</p> <h2 id="product-launches">Product Launches</h2> <p>SatSummit has several noteworthy launches:</p> <ul> <li>DigitalGlobe launched a new <a href="https://www.digitalglobe.com/opendata">Open Data Program</a> focused on data for disaster response. Planet also debuted their <a href="https://www.planet.com/disasterdata/">Disaster Data</a> open data platform.</li> <li>Astro Digital, Amazon Web Services (AWS), and Development Seed launched <a href="https://www.planet.com/disasterdata/">MODIS on AWS</a> providing daily snapshots of the earth.</li> <li>The Humanitarian OpenStreetMap Team (HOT) launched the official (non-beta) version of <a href="https://openaerialmap.org/">OpenAerialMap</a>.</li> <li><a href="https://www.satellogic.com/">Satellogic</a> committed to provide free global 30-meter resolution hyperspectal imagery for scientific and humanitarian purposes.</li> <li>SatSummit was a great coming out party for Open Imagery Network (OIN), a new effort to support the use of open imagery by the development community. OIN emerged from research around last SatSummit, and is supported by the Gates and Omidyar Foundations.</li> </ul> <p>The SatSummit website has <a href="https://satsummit.io/insights/2017/02/03/launches/">a full list of launches and commitments</a> from the Summit.</p> <h2 id="deep-thoughts">Deep Thoughts</h2> <p>I left SatSummit with many new ideas and some new concepts. The stickiest concepts permeated the discussion throughout the day and many conversations since.</p> <h3 id="geodiversity">Geodiversity</h3> <p><a href="https://twitter.com/ericg">Eric Gundersen</a> kicked off the day reminding us that this diversity is our strength. An extremely diverse set of people are behind our satellite launches, data analysis, and bringing this knowledge to global development. Satellite and remote sensing is a human, global endeavor and does not discriminate. We need to respect and defend our diversity and be vocal in the face of challenges to our diversity.</p> <p>#Geodiversity took on a life through the Summit from celebrating our diverse teams, to diverse imaging approaches, to using different sensors work in combination to give us deeper insights and flexible operation.</p> <h3 id="fat-data">Fat data</h3> <p>Satellite data isn’t just big, it’s fat. <a href="https://twitter.com/bronwynagrios">Bronwyn Agrios</a> argued that imagery files are bulky and big in ways that create unique challenges for people who archive that data, run analysis across that data, or distribute imagery on the web.</p> <h3 id="data-access">Data Access</h3> <p>Your pixels don’t matter if they aren’t in front of people who need them, when they need them, and how they need them. The Distribution Panel flipped the question of distribution to one of Access and recognized it as a human design problem as much as a technical challenge.</p> <h3 id="apis-over-portals">APIs over Portals</h3> <p>Powering the data revolution means distributing data to be accessed by computers. We need to move away from portals toward APIs and data in predictable file structures. As Bronwyn Agrios emphasized, if you love your data then you’ve got to “put an API on it”. This gives you the flexibility to build the UI that your users will love.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/satsummit-browyn.jpg" /> <figcaption>Bronwyn Agrios telling folks to "put an API on it."</figcaption> </figure> <p>What did I miss? I wasn’t able to get to every session. If you have other reflections to share, please send them to info@satsummit.io and we will post them on <a href="https://satsummit.io/insights/">SatSummit Insights</a>.</p> 2017-02-21T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/02/21/SatSummit-Highlights DC's Bold Family Leave Policy http://www.developmentseed.org/blog/2017/02/17/paid-family-leave <figure class="align-center"> <img alt="DJ Thomas at the controls" src="/assets/graphics/content/dj-thomas-card.jpg" /> </figure> <p class="dropcap">We are delighted that <a href="http://dcist.com/2017/02/paid_family_leave_passes.php">a bold family leave policy has moved forward in DC</a>. The policy makes it easier for Development Seed to do right by our team and all the great folks we work with in DC, from our suppliers and vendors, to the people who treat us to heavy pours at happy hour.</p> <p>As a small business, sharing our responsibility to our employees with other DC businesses lets us offer more aggressive leave policies to our team. A flat tax allows us to do that. There is still work to be done. Congress has 30 days to challenge the law, and there are implementation details to work out. In the meantime, we are proud of DC for taking this big step.</p> 2017-02-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/02/17/paid-family-leave Fed Mapping Happy Hour http://www.developmentseed.org/blog/2017/02/05/open-mapping-happy-hour <p class="dropcap">This upcoming Monday we’ll be hosting a Fed Mapping Happy Hour, our way of saying thank you to the hard working geographers and data folk in our Federal Government.</p> <p>Come anytime between 6:00 to 8:00pm for the free drinks and snacks, stay a while for a quick round of lightning talks from our team and our open source friends in town.</p> <p><a href="https://openmappinghappyhour17.splashthat.com/">RSVP here</a> so we have enough brews on hand.</p> <p><a href="https://openmappinghappyhour17.splashthat.com/"><img src="/assets/graphics/content/prose/OpenMappingHappyHour.jpg" alt="" /></a></p> 2017-02-05T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/02/05/open-mapping-happy-hour Open Source Machine Learning http://www.developmentseed.org/blog/2017/01/30/machine-learning-learnings <p class="dropcap">Tomorrow, I will be at <a href="https://satsummit.io/">SatSummit</a> talking about Development Seed’s machine learning work and what we’ve learned over 18 months of building open source tools for imagery analysis. We are building <a href="https://github.com/developmentseed/skynet-train">open source machine learning tools</a> and open algorithms that can be shared and improved by academics and implementers without expensive software or license fees.</p> <p>Machine learning has tremendous potential to improve the work of development organizations. We use machine learning to create smart algorithms to analyze satellite imagery to determine land use or to identify features like roads and buildings. Automating imagery analysis allows us to more quickly map unmapped areas or to monitor vast crop or forest areas and alert us to anomalies.</p> <p>But what does it require to take advantage of machine learning in your work? Here are a few things we’ve learned about using machine learning to identify features like roads and buildings.</p> <h3 id="for-road-detection-use-one-meter-resolution-imagery-or-better-multispectral-imagery-can-improve-results">For road detection use one meter resolution imagery or better. Multispectral imagery can improve results.</h3> <p>We’ve done land use classification and vegetation analysis on all manner of data, from 30 meter to less than one-meter resolution. For many of these applications, higher resolution imagery doesn’t significantly improve results, and can even add noise in some cases. For road and building detection, we’ve had greatest success with imagery that is one-meter resolution or better.</p> <p>Bringing in infrared bands can better distinguish plants from built up areas. It’s helpful for identifying roads in certain contexts, particularly in suburban neighborhoods with lots of trees.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-nir-true-annotate.png" alt="NIR and True Color Imagery, Machine Learning Output" /> <figcaption>left column: input images (infrared, true color), &copy; <a href="https://www.digitalglobe.com/" target="_blank">DigitalGlobe</a>; middle: <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; right: our model prediction</figcaption> </figure> <h3 id="you-need-at-least-100-square-kilometers-of-high-quality-training-data-for-feature-extraction">You need at least 100 square kilometers of high quality training data for feature extraction</h3> <p>Machine learning requires a lot of good training data to produce accurate results. Groups often ask, “How much training data is enough?” We ran some experiments to determine that.</p> <p>For the road detection work that <a href="https://developmentseed.org/blog/2016/08/17/sotm-skynet/">we demonstrated at State of the Map</a> we used 559 square kilometers of training data. We experimented with using gradually less data (372km<sup>2</sup>, 186km<sup>2</sup>, 93km<sup>2</sup>, 47km<sup>2</sup>) to determine at what point the accuracy deteriorated. For less dense mapping across a consistent area, sometimes ~100km<sup>2</sup> of high quality training data produced accurate results on roads. (That’s about the size of Sacramento. Or a little less than Washington DC’s NW and NE quadrants combined)</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-buckets-1.png" alt="Seattle Streets, Machine Learning Output" /> <figcaption>left-to-right: input image, &copy; <a href="https://www.mapbox.com/" target="_blank">Mapbox Satellite</a>; <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; columns 3-6: our model predictions with varying amounts of input data</figcaption> </figure> <p>For denser areas, or when trying to create a model which can make predictions over a larger variety of areas, more input data helps.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-buckets-2.png" alt="Seattle Streets, Machine Learning Output" /> <figcaption>left-to-right: input image, &copy; <a href="https://www.mapbox.com/" target="_blank">Mapbox Satellite</a>; <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; columns 3-6: our model predictions with varying amounts of input data</figcaption> </figure> <p>For buildings, we started with the same 559km<sup>2</sup> baseline for training data and saw promising results but it was tough to get clearly defined edges. We tried to limit our inputs to places where we felt very confident about the training data. Because this limitation reduced the input data to under 100km<sup>2</sup>, we didn’t see good improvement. Jump in <a href="https://github.com/developmentseed/skynet-train">the repo</a> if you have some ideas for how to improve!</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-buildings-1.png" alt="Dar es Salaam Buildings, Machine Learning Output" /> <figcaption>left-to-right: input image, &copy; <a href="https://www.mapbox.com/" target="_blank">Mapbox Satellite</a>; <a href="https://www.openstreetmap.org/" target="_blank">OpenStreetMap</a> data; our updated model with less input data; our first attempt </figcaption> </figure> <h3 id="training-a-model-can-take-a-few-days-but-once-its-trained-you-can-apply-it-in-real-time">Training a model can take a few days. But once its trained you can apply it in real time.</h3> <p>We’ve had most success running about 100 thousand iterations of our model. With one AWS <code>g2.2xlarge</code> instance that takes around 4 days. This is mostly something that can run in the background, but occasionally you’ll need to kick the server.</p> <p>Once the machine learning process has produced a algorithm, you can apply it to additional imagery in real time. We have set up apps that conduct analysis of imagery in real time as you browse.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ml-tile-server.gif" alt="Manilla Roads, Machine Learning Live Prediction" /> <figcaption>Live prediction of roads in Manilla, Philippines</figcaption> </figure> <p>While the results aren’t always perfect, they are pretty good. They can be used to give an immediate idea of where to spot features or to run a quick calculation on approximately how many buildings or kilometers of road fall into an administrative region or disaster area.</p> <h3 id="open-works-well-for-development-work">Open works well for development work</h3> <p>We’ve benefited greatly from using open tools for our work and by sharing our imagery processing pipeline with others. We love collaborating with people with interesting problems to solve or ingenious solutions we haven’t considered. You can check out the <a href="https://github.com/developmentseed/skynet-train">code on Github</a> or find me at SatSummit to chat.</p> 2017-01-30T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/30/machine-learning-learnings All of NASA's Earth Science Data on the Cloud http://www.developmentseed.org/blog/2017/01/30/NASA-on-the-cloud <p class="dropcap">No one processes and provides more earth observation data than NASA. NASA provides access to more than 17.5 petabytes of earth observation data across more than 11,000 unique data products from its satellite and aerial missions. <a href="https://earthdata.nasa.gov/nasa-data-policy">In FY 2016, NASA delivered more than 1.51 billion earth science data products to more than 3 million data users around the world</a>. Managing this data is a huge task. We are working with NASA’s Earth Science Data System (ESDS) Program to build Cumulus, a cloud-based prototype to ingest, process, catalog, archive, and distribute NASA’s Earth Data streams.</p> <p>Cumulus is one of a suite of tools that NASA is developing to better leverage cloud computing for data storage, processing, and access. We are working closely with NASA’s ESDS team as well as several of the Distributed Active Archive Centers (<a href="https://earthdata.nasa.gov/about/daacs">DAACs</a>) that ensure Earth Data missions are available and accessible to users. We are working with these teams to prototype and refine a system that can be used to process, archive, and publish data as varied as lightning strike datasets and MODIS scenes.</p> <p>Cumulus will be open source. We are working with NASA to transfer ownership of Cumulus to NASA and to provide the code and documentation through their platforms. We will keep you posted when the code is available. We hope that Cumulus will be useful to other organizations building imagery publishing pipelines.</p> 2017-01-30T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/30/NASA-on-the-cloud A imagery pipeline for disaster response http://www.developmentseed.org/blog/2017/01/27/imagery-pipeline <p class="dropcap">We are working with the <a href="http://hotosm.org/">Humanitarian OpenStreetMap Team</a> (HOT) to get the best satellite and drone imagery immediately into the hands of disaster responders. Partnering with HOT, we are building an imagery coordination pipeline. In the event of a disaster, this workflow will track the most pressing needs on the ground, and connect with satellite companies to task their satellites and drone operators to prioritize flights where they are most useful. We also built upload tools to OpenAerialMap so that drone operators can quickly distribute their images.</p> <h2 id="coordinating-imagery-needs">Coordinating Imagery Needs</h2> <p>We’re working together with HOT to better plan and coordinate imagery needs in times of response. Through HOT’s Imagery Coordination Working Group, we’re testing ideas and workflows to make it easier for disaster responders to flip through available pre- and post-disaster imagery and, when nothing meets their need, coordinate imagery acquisition with the wider community and imagery providers. The tool will be open in order to make it easier for local drone operators and GIS units to participate in defining needs and sharing data. Below is a sneak peak of comparing imagery within the tool.</p> <p>HOT will share more at <a href="https://satsummit.io">SatSummit</a> on the ongoing conversations to improve coordination among imagery providers.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/big--imagery-requests-search.gif" /> <figcaption>Humanitarian responders, satellite companies, and drone operators can search for imagery to meet their needs</figcaption> </figure> <h2 id="drone-data">Drone Data</h2> <p>Local drone operators can share their imagery through a <a href="https://upload.openaerialmap.org/">simple user interface</a>. OpenAerialMap’s upload tool is integrated directly with Dropbox and Google Drive. Often the drone operators nearest to a disaster area aren’t experts in processing imagery for sharing on the web. The upload tool makes it quick and easy to share some of the first surveys of a disaster.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/oam-uploader-form.png" /> </figure> <p>This is a very important part of the OAM ecosystem as it enables more people and organizations to submit imagery to the ever-growing catalog of OAM. It allows any surveyor to quickly share fresh imagery with the world under the <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY 4.0</a> making it available to whomever needs to use it. It’s now easier than ever to share new images and help with disaster preparedness.</p> 2017-01-27T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/27/imagery-pipeline SatSummit: Solving Global Issues Through Data http://www.developmentseed.org/blog/2017/01/26/satsummit <p class="dropcap">Next week, Development Seed and Mapbox will host <a href="https://satsummit.io/">SatSummit</a> in Washington, D.C. Satellite industry and global development experts from around the globe will come nerd-out on solving social and environmental challenges through practical applications of satellite data. The big focus this year is turning data into action, with deep dives into machine learning, maintaining climate data streams, famine and malaria mapping, and imagery for humanitarian response.</p> <p>Look for us throughout the day:</p> <ul> <li> <p><a href="https://developmentseed.org/team/ian-schuler/">Ian</a> will moderate a discussion with “The Geographers” from The World Bank and USAID. They’ll tackle tough questions relating to how organizations incorporate remote sesning data into daily decision-making.</p> </li> <li> <p><a href="https://developmentseed.org/team/drew-bollinger/">Drew</a> will share our progress with developing a suite of machine learning tools built entirely in “open”. He’ll show what <a href="https://github.com/developmentseed/skynet-data">Skynet</a> is capable of and our pipeline to use OpenStreetMap as a deep source of training data for deep learning.</p> </li> <li> <p>Behind the camera <a href="https://developmentseed.org/team/dylan-moriarty/">Dylan</a> will capture big moments, while <a href="https://developmentseed.org/team/anna-scalamogna/">Anna</a> and <a href="https://developmentseed.org/team/derek-lieu/">Derek</a> live tweet the event sharing key insights that are discussed. Our friendly face at registration <a href="https://developmentseed.org/team/miles-watkins/">Miles</a> will greet people as they arrive, while <a href="https://developmentseed.org/team/olaf-veerman/">Olaf</a> and <a href="https://developmentseed.org/team/alireza-j/">Alireza</a> roam around to chat up new and old friends.</p> </li> <li> <p>I’ll be there as well making sure things run smoothly.</p> </li> </ul> <p>Our good friends will be there from groups like DigitalGlobe, The World Bank, AWS, Planet, and USAID Global Development Lab. We are also excited to meet new folks and hear new approaches to tackling hard problems. SatSummit is a real opportunity to form unexpected partnerships and think outside the box, and we’re looking forward to what comes out of it.</p> <p>The event is unfortunately booked up. If you haven’t signed up and want to attend, add your name to the waitling list <a href="here">here</a>. We’re in the process of confirming attendees, and tickets may become available. For those unable to join in person, follow along <a href="https://twitter.com/sat_summit">@sat_summit</a> on Twitter.</p> 2017-01-26T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/26/satsummit Development Seed, Annual Report http://www.developmentseed.org/blog/2017/01/05/annual-report <p class="dropcap">Development Seed builds products that draw insight from complex data, helping people make better decisions. We work with non-profits and civic for-profits addressing the worlds biggest challenges. We use open technology, modern cloud infrastructure and thoughtful design practices to quickly build and scale our meaningful data driven tools.</p> <p>In 2016, we built bigger and better products; we reached our biggest audience; and we worked with the largest sets of open data we’ve ever touched. We pioneered new approaches to machine learning and monitoring earth from space. We traveled the world working with governments to track agriculture, electricity and road projects.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/AR-egypt1.jpg" alt="Ali in Egypt" width="1400" height="365" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/AR-India-derek.jpg" alt="Derek in front of Taj Mahal" width="1400" height="491" /> </figure> <p>Development Seed operates as a non-profit. Our work with transparency and development partners supports our open source work, e.g. <a href="https://github.com/sat-utils">sat-utils</a>, <a href="http://prose.io/">prose</a>, <a href="http://devseed.com/dirty-reprojectors-app/">dirty reprojections</a>, and <a href="https://github.com/developmentseed/skynet-train">skynet</a>.</p> <h2 id="blowing-the-roof-off">Blowing the roof off</h2> <figure class="align-center"> <a href="https://www.washingtonpost.com/2016-election-results/us-presidential-race/?utm_term=.f0e61c3a004e"><img src="/assets/graphics/content/prose/AR-wapo.jpg" alt="Washington Post election webpage" width="675" height="540" /></a> </figure> <p>Our <a href="https://developmentseed.org/blog/2016/11/08/post-eday/">project with The Washington Post, building live election maps</a> was our most visible project in 2016. Tens of millions of people have interacted with our maps, along with more than 100 other syndicates in The Washington Post network. On election night, The Washington Post broke records for the number of people on the website. The maps not only withstood the load, but were consistently a minute or more faster than competitor sites. Our maps allowed users to track up-to-the-moment election results, through an easy-to-use interface that worked on all devices, and also gave The Washington Post team the flexibility to highlight the most important coverage as results came in.</p> <figure class="align-center"> <a href="https://www.nasa.gov/"><img src="/assets/graphics/content/prose/AR-NASA_logo.svg" alt="NASA logo" width="294" height="236" /></a> </figure> <p>We are working with NASA (Yes, <em>that</em> NASA :swoon:) to build Cumulus, a cloud-based tool for processing science data from satellites, planes, and ground sensors. NASA currently maintains over 6,100 data streams from sensors in space and on the ground. Cumulus is still in the prototype phase, and we are working closely with NASA and USGS data operators to streamline and centralize processing of data streams from near realtime hurricane data to climate data.</p> <figure class="align-center"> <a href="https://openaerialmap.org/"><img src="/assets/graphics/content/prose/AR-oam.jpg" alt="Open Aerial Map" width="675" height="540" /></a> </figure> <p>We’re helping the development community use satellite data to track and understand our Planet. Our <a href="https://github.com/developmentseed/skynet-train">Skynet</a> tools bring modern machine learning methods to automatically extract features from satellite imagery–such as roads or damaged buildings. We built a powerful earth monitoring platform for Astro Digital, that automatically processes new satellite imagery from NASA and the European Space Agency (ESA) and delivers it straight to your app or inbox. We also expanded <a href="https://openaerialmap.org/">OpenAerialMap</a>, enabling drone operators to more easily contribute data, helping them quickly collect drone imagery after a natural disaster.</p> <figure class="align-center"> <a href="http://oci.carnegieendowment.org/"><img src="/assets/graphics/content/prose/AR-oci.jpg" alt="Oil Climate Index" width="675" height="540" /></a> </figure> <p>We built practical tools to address climate change. We partnered with OpenAQ’s effort to <a href="https://openaq.org/">end air inequality</a>, by collecting and organizing the world’s air quality data. The OpenAQ network, collects critical data to inform research and policy decisions. As of this morning, 34.5 million measurements, from 4,706 locations, in 43 countries have been collected. We also worked with the Carnegie Endowment to build the <a href="http://oci.carnegieendowment.org/">Oil Climate Index</a>, a tool to monitor the climate impact of different oils around the world. OCI allows investors, policymakers, industry, and the public, to see how we can reduce greenhouse gas emissions, <em>without changing consumption levels</em>, by making better decisions about the types of oils we use.</p> <p>And much more. Read about some of <a href="https://developmentseed.org/projects/">our other projects</a>.</p> <h2 id="growing-the-family">Growing the Family</h2> <p>I’m so incredibly proud to be part of this team and happy to announce that this truly impressive group grew by <a href="https://developmentseed.org/blog/2016/01/21/welcome-matt/">three</a> <a href="https://developmentseed.org/blog/2016/02/18/welcome-ali/">team</a> <a href="https://developmentseed.org/blog/2016/11/10/welcome-laura/">members</a> this year.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/AR-biketour.jpg" alt="DS on our brewery bike tour" width="738" height="416" /> <figcaption>Devseed brewery bike tour</figcaption> </figure> <p>I’m also incredibly grateful to the wonderful partners, collaborators, and rabble rousers that we work with. I can’t wait to show off what we are cooking up for 2017.</p> <p>See you at <a href="https://satsummit.io/">SatSummit</a>!</p> 2017-01-05T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2017/01/05/annual-report Vote for OpenAQ http://www.developmentseed.org/blog/2016/12/20/vote-for-openaq <p class="dropcap">OpenAQ, a project we helped build to end air inequality, needs your help to continue growing its data. The project is currently one of the six finalists for the <a href="https://www.openscienceprize.org/">Open Science Prize</a>. Winning this prize would allow OpenAQ to expand the geographic coverage of official data sources and architect the integration of low-cost sensor and satellite data. The latter would be an important step toward bridging the data gap that affects places with poor air quality most.</p> <h2 id="air-inequality">Air inequality</h2> <p>Poor air quality is responsible for 1 out of 8 deaths in the world, more than malaria and HIV/AIDS combined. Access to air quality data is fundamental for scientists and policy makers to gain a deeper understanding of the issue, but is often hard to come by.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/openaq-compare.jpg" /> <figcaption>OpenAQ makes it easy to compare air quality around the globe</figcaption> </figure> <p><a href="https://openaq.org">OpenAQ</a> addresses this issue by providing open access to historical and real time air quality data around the globe. The platform currently provides access to over 32 million air quality measurements, and this number continues to grow at the rate of 150 000 a day. Beyond building an open data platform, the OpenAQ team has done a tremendous job building a global community of scientists, journalists and software developers. A community capable of harnessing the data to influence policy and empower the general public.</p> <p>Show you support for OpenAQ <a href="http://event.capconcorp.com/wp/osp/vote-now/">by voting</a> by January 6th.</p> 2016-12-20T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/12/20/vote-for-openaq Dirty Reprojectors http://www.developmentseed.org/blog/2016/12/15/dirty-reprojectors <script src="https://api.tiles.mapbox.com/mapbox-gl-js/v0.28.0/mapbox-gl.js"></script> <link href="https://api.tiles.mapbox.com/mapbox-gl-js/v0.28.0/mapbox-gl.css" rel="stylesheet" /> <style> #map-virtue { height: 600px; margin-bottom: 2rem; } .mapboxgl-ctrl-top-right .mapboxgl-ctrl { margin: 10px 2rem 0 0; } </style> <div class="map-container bleed-full"> <div id="map-virtue"></div> </div> <p><em>The Robinson projection in Mapbox GL</em></p> <p class="dropcap">We’re open sourcing a way to pre-process geometries for use in Mapbox GL, to produce maps that render something besides the standard Web Mercator projection.</p> <p>Almost all web mapping libraries render maps using Web Mercator, making an assumption that you generally can’t change out-of-the-box. This has <a href="https://gist.github.com/tmcw/a3828827c8490f7e13a9">advantages</a>, but it posed a real challenge for us when we set out to build the <a href="https://developmentseed.org/blog/2016/11/08/post-eday/">Washington Post’s live election results map</a>, where using the <a href="http://bl.ocks.org/mbostock/5545680">Albers USA</a> projection was an important requirement. To meet that requirement, we built a pipeline to pre-process geometries.</p> <p>We started with our standard WGS 84 (longitude/latitude) coordinate geojson files and projected those files into our desired projection. Next, we scaled the result to fit within the Web Mercator coordinate system.</p> <figure class="align-center"><img src="/assets/graphics/content/prose/projection-part-1.png" alt="projection" /></figure> <p>We took the scaled files and reverse-projected them back into WGS 84, before converting them to mbtiles and uploading the whole thing to Mapbox Studio.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/projection-part-2.png" alt="projection" /> </figure> <p>We used Mike Bostock’s <a href="https://github.com/d3/d3-geo">d3-geo</a> and <a href="https://github.com/d3/d3-geo-projection">d3-geo-projection</a> libraries to do the heavy lifting on the rather tricky projection math.</p> <p>To show some of the projections you can achieve with this method, we built the <a href="https://www.developmentseed.org/dirty-reprojectors-app/">Dirty Reprojectors App</a>. The site includes the long list of projections d3 comes bundled with, and will also accept and project your own geojson files.</p> <figure class="align-center"> <div class="map-container bleed-full"> <a href="https://devseed.com/dirty-reprojectors-app/" target="_blank"><img src="/assets/graphics/content/prose/projection-examples.gif" alt="projection" /></a> </div> </figure> <p>For greater control over projection variables like parallels and center points, use the <a href="https://github.com/developmentseed/dirty-reprojectors">Dirty Reprojectors CLI</a> instead. There you can tweak the projection code as needed.</p> <h3 id="caveats">Caveats</h3> <p>After reprojecting geometries, the actual longitude/latitude coordinates will not be accurate. This complicates tasks like reverse geocoding.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/projection-part-3.png" alt="Actual coordinates are off" /> </figure> <p>Bounds are also tricky. Depending on the projection, a bounding box (rectangle) in longitutde/latitude coordinates may no longer look like one once it’s reprojected. Functions like map.fitBounds() where the map fills the space of its bounds can take more work to get right. Our work-around was to calculate and save the post-projection bounding box for each US state, instead of using the “real” bounding box coordinates.</p> <p>We’re always looking for novel ways to solve problems. If this helps you solve a problem, we’d love to hear about it. You can always reach out to us on <a href="https://twitter.com/developmentseed">Twitter</a> or <a href="https://github.com/developmentseed">Github</a>.</p> <script> mapboxgl.accessToken = 'pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q'; var map = new mapboxgl.Map({ container: 'map-virtue', style: 'mapbox://styles/devseed/civlnnvyf000y2imxc4xqimcx', center: [0, -10], zoom: 1, minZoom: 1 }); map.addControl(new mapboxgl.NavigationControl()); map.scrollZoom.disable(); var hoverSource = 'composite:hover'; // duplicate the source as a hover source function onStyleLoad (e) { var style = map.getStyle() if (style.sources && style.sources.composite) { map.addSource(hoverSource, Object.assign({}, style.sources.composite)); map.off('style.load', onStyleLoad); } } // once the newly duped hover source is loaded, // reset the map style but alter the hover layer to point to new source // then assign the mousemove listeners function onSourceLoad (e) { if (e.source.id === hoverSource) { var style = map.getStyle() var hover = style.layers.find(function (d) { return d.id === 'Country-Fill-Hover'; }); hover.source = 'composite:hover'; map.setStyle(style); map.on('mousemove', function(e) { var features = map.queryRenderedFeatures(e.point, { layers: ['Country-Fill'] }); if (features.length) { map.setFilter('Country-Fill-Hover', ['==', 'admin', features[0].properties.admin]); } else { map.setFilter('Country-Fill-Hover', ['==', 'admin', '']); } }); map.on('mouseout', function() { map.setFilter('Country-Fill-Hover', ['==', 'admin', '']); }); map.off('source.load', onSourceLoad); } } map.on('style.load', onStyleLoad); map.on('source.load', onSourceLoad); </script> 2016-12-15T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/12/15/dirty-reprojectors Mappers Gather at Development Seed http://www.developmentseed.org/blog/2016/11/18/ds-mapathon <p class="dropcap">The <a href="http://www.redcross.org/">American Red Cross</a>, <a href="http://mapgive.state.gov/">MapGive</a>, and <a href="https://www.usaid.gov/">USAID/OFDA</a> joined our team on Monday night to kickoff GeoWeek in DC. We started the night off with presenters showcasing their work for the OpenStreetMap community. These institutions apply open source and open data sharing for humanitarian response and economic development.</p> <p>Rachel Levine from the American Red Cross began by explaining her role in the community, and used examples of projects people are contributing to around the world. She then spoke about the missing maps project in Malawi.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mapathon.jpg" alt="mapathon" /> </figure> <p>We then heard from Keifer Gonzalez of USAID/OFDA who discussed <a href="https://volcanomapping.github.io/">decreasing volcanic risks in Indonesia</a>, and shared explosive photos of volcanoes. Andrew McKenna from the State Department’s MapGive program talked about U.S. President’s Emergency Plan for AIDS Relief. PEPFAR requests help to inventory buildings and roads for HIV/AIDS prevention and treatment programs in Tanzania.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mapathon-2.jpg" alt="mapathon" /> </figure> <p>We had mappers with all levels of experience join us: active mappers chose the project that resonated with them most and began mapping after presentations concluded; Tom Gertin (MapGive) walked beginners through the ins and outs of contributing to the OSM Community.</p> <p>If you’d like to contribute to mapping these projects, check out the <a href="http://tasks.hotosm.org/">HOT tasking manager</a> for tasks <a href="http://tasks.hotosm.org/project/750">#750</a>, <a href="http://tasks.hotosm.org/project/2296">#2296</a>, <a href="http://tasks.hotosm.org/project/2301">#2301</a>. Happy mapping!</p> 2016-11-18T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/18/ds-mapathon Mapathon at Development Seed http://www.developmentseed.org/blog/2016/11/11/osm-geoweek <figure class="align-center"> <img src="/assets/graphics/content/prose/osmgeoweek.jpg" alt="osm-geoweek" /> </figure> <p class="dropcap">Monday evening (11/14) we are hosting a Mapathon as part of <a href="http://osmgeoweek.org/">OSM GeoWeek</a> at our office. Join Development Seed, USAID, the American Red Cross and MapGive to improve the map for humanitarian efforts around the world. We’ll work on tasks that support these organizations and their focus to decrease volcanic risks in Indonesia, eliminate Measles and Rubella in Malawi, and gain better insight into energy access in Sub-Saharan Africa.</p> <p>You don’t need to be an expert mapper to participate. Just bring your laptop and we’ll help get you set up and equipped to map. <a href="https://www.eventbrite.com/e/osm-geoweek-mapathon-hosted-by-development-seed-tickets-29088088255">Reserve your spot now!</a></p> 2016-11-11T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/11/osm-geoweek Bienvenida Laura! http://www.developmentseed.org/blog/2016/11/10/welcome-laura <p class="dropcap">We are thrilled to welcome Laura Gillen to the Development Seed team! Laura will run Mission Control, helping us to improve our operations, grow Development Seed’s community presence, and run big events like <a href="https://satsummit.io">SatSummit</a>.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/laura-welcome.jpg" alt="Laura on a boat at sea" /> <figcaption></figcaption> </figure> <p>Laura has a fantastic mix of civic and business experiences. She has worked at Women for Women International and worked on the communications team for Senator Al Franken’s 2014 re-elect campaign. Most recently, Laura was the first employee of Laurel Strategies and helped to grow the strategic communications firm and global business advisory. While at Laurel, she worked with clients that span multiple sectors, including software, e-commerce, and telecomm, helping them with the development and execution of strategy, maximizing positioning and media relationships, and refining communications.</p> <p>When she’s not at the office, you can find the native Minnesotan camping at a DC coffee shop, serving aces on the tennis court, jogging to carefully curated playlists, or boarding a plane (or boat) for her next adventure.</p> <p>Welcome Laura! We can’t tell you how excited we are that you are here.</p> 2016-11-10T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/10/welcome-laura It’s Election Day in the USA: Where will you watch the results? http://www.developmentseed.org/blog/2016/11/08/post-eday <p class="dropcap">Wherever you end up watching the election returns tonight, check out The Washington Post live election maps. We worked closely with The Post to build lightning fast maps that provide up-to-the-second data presented in a manner that is easy to understand.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/wapo-preview.gif" alt="election-maps" /> </figure> <p>We iterated and improved our approach over 24 primary nights and we have introduced new features for the today’s election. Here is some of what you can expect tonight:</p> <h3 id="clearinformative-coverage">Clear/informative coverage</h3> <p>The Post’s live election maps take data from 532 races and organize it into an interface that is understandable at a glance but also allows users to explore the context and detail of this election. We include a live cartogram map, to overcome <a href="https://www.washingtonpost.com/graphics/politics/2016-election/how-election-maps-lie/?tid=sm_tw">the disadvantages of traditional election maps</a> and visually represent the electoral weight of each state. Also, after the states are called a wind map will show how the vote has shifted since the last election in each county for every race.</p> <h3 id="updates-and-callouts-to-catch-important-moments">Updates and callouts to catch important moments</h3> <p>The interface is designed to make sure that you don’t miss important updates, even if you look away from your screen. We pull in new data across roughly 3,000 counties, every 4 seconds, to show the freshest data possible. We’ve redesigned our <a href="https://developmentseed.org/blog/2016/03/15/tracking-live-events/">“freshies”</a> – callouts on the map that notify users of tightening races or new vote leaders.</p> <p>Live maps are deployed throughout The Washington Post website. The Post’s reporters and analysts will add additional news and analysis in their features and the <a href="https://www.washingtonpost.com/politics/2016/live-updates/general-election/real-time-updates-on-the-2016-election-voting-and-race-results/">Washington Post Live Blog</a>, which you can read without taking your eyes off the results.</p> <h3 id="coverage-no-matter-where-you-are">Coverage no matter where you are</h3> <p>Many people will track the election returns on the road or on multiple screens. The Post’s maps are just as usable and fast on mobile. No matter where you end up spending this election night, be sure to take a look at <a href="https://www.washingtonpost.com/">washingtonpost.com</a> to keep up-to-date with tonight’s gripping election results.</p> 2016-11-08T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/11/08/post-eday Thanks Maptime! http://www.developmentseed.org/blog/2016/10/06/Maptime <p class="dropcap"><a href="http://maptime.io/">Maptime</a> is a great meetup for people interested in learning modern mapping techniques &amp; applications. We recently hosted a training <a href="http://www.slideshare.net/DylanMoriarty/maps-by-hands">on hand-drawn maps</a> along with Maptime DC. We were delighted to have 40 people come by.</p> <p>When you let people loose to create their own unique map through the good ol’ technique of “pen to paper,” (and in some instances, crayon) you can see some good creativity. By the end of the night we had a variety of maps that reflected the diverse group of attendees.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_3.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_4.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_2.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_5.jpg" alt="maptime photo" /> </figure> <figure class="align-center"> <img src="/assets/graphics/content/prose/maptime_1.jpg" alt="maptime photo" /> <figcaption>Photos by Brian Davidson &amp; Dylan Moriarty.</figcaption> </figure> <p>Hopefully we’ll see you at the next <a href="http://www.meetup.com/Maptime-DC/">Maptime</a>. In the meantime, let us know if there are other geo or tech events you’d like to see or to help us host!</p> 2016-10-06T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/10/06/Maptime OpenAerialMap Improved http://www.developmentseed.org/blog/2016/09/22/oam-improvements <p class="dropcap">It’s been more than a year since we first launched <a href="https://developmentseed.org/blog/2015/05/27/introducing-openaerialmap/">OpenAerialMap</a>, and ever since, we’ve continued to invest in the site and make improvements. We truly believe that this service helps users to share, find and use aerial imagery in as simple and efficient of a way as possible. Below I’ve outlined a few of the ways we’ve improved the site to make for a seamless experience.</p> <p>Some of the latest changes include:</p> <ul> <li>A brand new <a href="http://openaerialmap.org/">homepage</a> that introduces users to the project.</li> <li>Improvements to the <a href="https://upload.openaerialmap.org/">upload form</a> by adding integration with Dropbox, and Google Drive (not yet deployed, but expect to see this soon). This aims to lower the contribution barrier.</li> <li>Added a feedback form to the <a href="https://beta.openaerialmap.org/">browser</a> (allows users to easily report problems with the imagery).</li> <li>Added options for imagery preview, including TMS.</li> <li>Under the hood improvements.</li> </ul> <p>I’ll be at <a href="http://2016.stateofthemap.org/">State of the Map</a> in Brussels this week to talk about the OAM project, specifically these improvements and the changes to come. If you’re there be sure to come listen – you can find me in <a href="http://2016.stateofthemap.org/2016/open-imagery-matters-remote-mapping-with-openaerialmap/">Auditorium C, Friday at 12:30pm</a>.</p> 2016-09-22T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/09/22/oam-improvements Preview The Washington Post's election maps http://www.developmentseed.org/blog/2016/09/08/wapo-preview <p class="dropcap">This week we finally got the chance to show some of the tools that The Washington Post will use to cover election results in November, live on The Post’s website.</p> <p>On Tuesday, The Washington Post and SurveyMonkey released a comprehensive presidential poll of all 50 states. Alongside this ambitious polling effort, The Post also rolled out preview pages detailing the <a href="https://www.washingtonpost.com/2016-election-results/us-presidential-race/">Presidential</a>, <a href="https://www.washingtonpost.com/pb/2016-election-results/us-senate/">Senate</a>, <a href="https://www.washingtonpost.com/pb/2016-election-results/house-of-representatives/">House</a>, and <a href="https://www.washingtonpost.com/pb/2016-election-results/governors-state-race/">Gubernatorial</a> races across all 50 states and Washington DC. These pages show the current state of the race and offer predictions and historic results.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/wapo-preview.gif" alt="election-maps" /> </figure> <p>The maps on these pages lean on much of the technology and workflows that we are developing for Election Day. To render maps, we make heavy use of <a href="https://github.com/mapbox/mapbox-gl-js">Mapbox GL JS</a>, an open-source mapping library that uses your graphics processor to draw fast, highly detailed maps—borrowing from technologies that render detailed video game universes.</p> <p>This framework provides cutting-edge hardware acceleration to draw beautiful maps that effortlessly accommodate highly interactive input from Washington Post readers. However, making WebGL maps work for live reporting an election on a major national news site presented a number of interesting challenges:</p> <ul> <li><strong>Albers US projection</strong> To make the coverage work with US-centric elections, we employed a method called* “Lying to Leaflet,” inserting the more appropriate <a href="http://bl.ocks.org/mbostock/2869946">Albers USA projection</a>, in place of the default <a href="https://en.wikipedia.org/wiki/Web_Mercator">Web Mercator</a>. You can use this method to show other projections, for example <a href="https://www.mapbox.com/blog/mapping-arctic-ice-polar-projection/">the polar projection to map artic ice</a>.</li> <li><strong>Dynamic sources</strong> Although not required for preview pages, the real-time election data on Election Day will require us to join data to vector tiles “on the fly” before feeding these to Mapbox GL to render. We are doing this for preview pages today in order to test and optimize our data pipeline in advance of November.</li> <li><strong>Reactive updates</strong> We employ a <a href="https://gist.github.com/staltz/868e7e9bc2a7b8c1f754">reactive programming design pattern</a> to build the code around Mapbox GL. This allows us to keep program state in one central location and only respond to necessary changes, including using a “style diff” to update the maps. <a href="https://www.mapbox.com/blog/mapbox-gl-js-reactive/">A recent Mapbox blog post further describes this approach.</a></li> <li><strong>Shared rendering resources</strong> The Post’s expansive coverage requires us to load several maps concurrently on the same webpage, in main views and in small multiples. By using a shared pool of web workers, we minimized the overhead for faster loading.</li> <li><strong>Independent components with a central driver</strong> At the same time, we often need maps to behave independently. Components must be maximally flexible, embeddable on pages we don’t control, and usable in a number of configurations. To do this, we created a central, application-wide, reducer-based store. This allows the Post to create pages with an arbitrary number of components without too much orchestration.</li> <li><strong>Image-based fallbacks</strong> Though <a href="http://caniuse.com/#feat=webgl">WebGL</a> support is already high and climbing, a small percentage of users on older browsers don’t have access. To serve these users, we are building a workflow that generates static images directly from our Mapbox GL pipeline. This ensures that all users can see the most recent data no matter their browser.</li> </ul> <p>As this election season progresses, The Post will provide contextual, historical, and demographic data while updating its predictions. In parallel, we will refine and test our approach to make sure it provides powerful data on what will certainly be an interesting election.</p> <figure class="align-center"> <a href="https://www.washingtonpost.com/2016-election-results/us-presidential-race/"><img src="/assets/graphics/content/prose/wapo-small-mult.jpg" alt="demographic-maps" /></a> <figcaption>Small multiples in a box.</figcaption> </figure> <p><em>*h/t <a href="https://twitter.com/mojodna">Seth Fitzsimmons</a> of Stamen Design.</em></p> 2016-09-08T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/09/08/wapo-preview Machine Learning at State of the Map US http://www.developmentseed.org/blog/2016/08/17/sotm-skynet <p class="dropcap">A couple weeks ago, I talked at <a href="http://stateofthemap.us/">State of the Map US</a> about our “skynet” experiments, using deep learning with satellite imagery and OpenStreetMap data to develop algorithms for automated mapping. Thanks to everyone who came to my talk–I’m really grateful for the positive feedback and excited by the great ideas that you shared!</p> <p>The <a href="https://www.youtube.com/watch?v=BftllmqXSpA&amp;list=PLqjPa29lMiE3eR-gK80irr3xdUiRbIMeg&amp;index=2">video</a> and <a href="http://anand.codes/talks/sotmus-2016/">slides</a> are available, but here’s the “tl;dr”:</p> <div class="callout"> <p>Can deep learning get us to highly accurate and complete automated road detection?</p> <p><strong>Almost certainly.</strong></p> <p>Do you need to have hundreds of thousands of dollars and a dedicated research lab full of experts to do it?</p> <p><strong>Definitely not!</strong></p> </div> <figure class="align-center"> <a href="https://www.youtube.com/watch?v=BftllmqXSpA&amp;list=PLqjPa29lMiE3eR-gK80irr3xdUiRbIMeg&amp;index=2"><img src="/assets/graphics/content/prose/sotm-skynet.jpg" alt="skynet" style="max-height: 300px" /></a> <figcaption>Here is the somewhat longer version.</figcaption> </figure> <p>Conversations at State of the Map and afterwards made it clear that there is a lot of interest and activity right now around using computer vision and machine learning for mapping:</p> <ul> <li>The <a href="http://deeposm.org/">DeepOSM</a> team is using a lean, fast neural net to estimate the likelihood of “road registration errors” in OSM.</li> <li>Facebook gave us some good insight into their <a href="http://forum.openstreetmap.org/viewtopic.php?id=55220">internal efforts at automated mapping</a>.</li> <li><a href="https://twitter.com/calimapnerd">Dale Kunce</a> suggested that Dar es Salaam has been sufficiently mapped by the Missing Maps effort to provide good training data. As soon as I got home I started working with this and have already been getting promising results in East Africa: <img src="/assets/graphics/content/prose/skynet-dar.png" alt="Skynet results in Dar es Salaam" /> <em>(left: input image, © <a href="https://www.mapbox.com/">Mapbox Satellite</a>; middle: <a href="https://www.openstreetmap.org/">OpenStreetMap</a> data; right: our model)</em>.</li> <li>We had some great conversations with the World Bank on how to apply this to their own efforts.</li> <li>There were thoughtful (and mildly controversial) reflections in blog posts from <a href="http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.html">Mike Migurski</a> and <a href="https://tomlee.wtf/2016/07/31/introductory-openstreetmap-politics/">Tom Lee</a> about the broader implications of “robot mapping.”</li> </ul> <p>There’s much more to do here, but it finally feels like deep learning is starting to become accessible to us as a community interested in real, practically useful results. Within the next few weeks, a few people working on machine learning for road mapping are going to jump on a hangout. If you are interested to join, please <a href="https://twitter.com/anandthakker">ping me on twitter</a>. I’m excited to see what we can build.</p> <p>Meanwhile, we’ve been working to clean up the code and docs from our experiments, in the hopes that others in the community can replicate our results and take them farther. Check out the readmes for <a href="https://github.com/developmentseed/skynet-data">skynet-data</a> and <a href="https://github.com/developmentseed/skynet-train/tree/master/segnet">skynet-train</a>. If we did it right, you should be able to start training your own neural net to extract features from satellite imagery with little more than an AWS account, and an hour of setup, and 🐳<code>docker run ...</code>. If not, drop us a line, or open an issue or PR!</p> <p><em><a href="https://www.flickr.com/photos/openstreetmapus/27930399144/in/album-72157671548583845/">cover</a> and <a href="https://www.flickr.com/photos/openstreetmapus/28441015702/in/album-72157671548583845/">card</a> photos by <a href="https://twitter.com/tatsvc">@tatsvc</a></em></p> 2016-08-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/08/17/sotm-skynet Get the best open satellite imagery from one interface http://www.developmentseed.org/blog/2016/08/04/ad-sentinel2 <p class="dropcap">Sentinel-2 data is now available in the <a href="https://fetch.astrodigital.com/">Astro Digital Imagery Browser</a>. Now you can quickly search for, publish, and download the best open satellite imagery from NASA and ESA in one interface. The interface is powered by the <a href="https://api.astrodigital.com/">Astro Digital API</a> which recognizes that users just want the best data and don’t always care what satellite it comes from.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/ad-sentinel.png" alt="Sentinel in Astro Digital Imagery Browser" /> </figure> <p>Like our previous work with Landsat, this is built on top of powerful, <a href="https://github.com/sat-utils/">open source processing tools</a> we’re creating for working with Sentinel-2 imagery.</p> <p>Truly open truly works. This was only possible because AWS made <a href="http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/">Sentinel-2 data programmatically accessible</a>.</p> 2016-08-04T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/08/04/ad-sentinel2 Maintain a common user experience across websites with Design Systems http://www.developmentseed.org/blog/2016/07/14/reusable-design-components <p class="dropcap">We use Design Systems to help us more efficiently organize and reuse design assets and code across sites that share elements.</p> <p>We often create and tie together multiple websites, building what feels like a single site to its end user. Doing this makes our projects easier to maintain, allows us to develop in an extremely agile manner, and gives us the freedom to use the technology that best suits each task.</p> <p>For instance, we built a number separate sites for <a href="http://openaerialmap.org">OpenAerialMap</a>, a tool that provides humanitarians with satellite and drone imagery. We built a separate site for the <a href="http://beta.openaerialmap.org">imagery browser</a>, the <a href="https://upload.openaerialmap.org">data uploader form</a>, and the <a href="http://docs.openaerialmap.org/">documentation</a>, just to name a few. Building a site for each of these facets allowed us not only to select the tech stack that best fit the needs of the project, but also prevented us from building a monolith app whose maintenance would be complicated. However, maintaining a consistent design language and interactions across all of them became a real challenge.</p> <p>We found ourselves copying and pasting code between sites, creating a codebase that was hard to maintain. To cope with this we built Design System, a centralized module to contain all shared elements across sites.</p> <figure class="align-center desktop"> <div class="desktop-group-trio"> <div class="media-frame"> <img src="{{site.baseurl}}/assets/graphics/content/prose/reusable-design-components--uploader.png" alt="OAM Uploader Form" width="1920" height="960" /> </div> <div class="media-frame"> <img src="{{site.baseurl}}/assets/graphics/content/prose/reusable-design-components--docs.png" alt="OAM Docs" width="1920" height="960" /> </div> <div class="media-frame"> <img src="{{site.baseurl}}/assets/graphics/content/prose/reusable-design-components--browser.png" alt="OAM Browser" width="1920" height="960" /> </div> </div> <figcaption>These are all separate websites, but we manage the design and user interactions in one place.</figcaption> </figure> <h2 id="design-system">Design System</h2> <p>Building on the approach we took for <a href="http://collecticons.io/">Collecticons</a>, and drawing further inspiration from <a href="http://foundation.zurb.com/sites/docs/">Foundation</a>, <a href="http://v4-alpha.getbootstrap.com/getting-started/introduction/">Bootstrap</a>, and <a href="https://medium.com/salesforce-ux/designing-products-that-scale-c8f3001f709b">this great article by Salesforce</a>, we used methods for distributing code through a package manager, e.g. node package manager (NPM) or Ruby Gems. This allowed us to take all of our shared elements across sites, and turn them into an installable node module that can be included in each website that makes up a project. The three main elements that make up the Design System are:</p> <ul> <li><code>scripts</code> that can be included like any node module;</li> <li><code>styles</code> that can be loaded as a whole or as individual components and;</li> <li><code>images</code> which are common throughout the ecosystem and include logos and other branding elements.</li> </ul> <p>Instead of copying and pasting code across websites, using <a href="https://www.npmjs.com/">node packages</a> let us store all shared elements in one place and make those elements easily accessible and readily available to each and every site. For example, if we wanted to update the logo, we’d only have to change it in one place for it to be automatically updated on all other sites the next time they build.</p> <p>Finally, having one repository for all shared elements, allows codebases to be more easily maintained and standardized, helping developers get started on new products, which is especially important for open source projects.</p> <p>Although we’re still testing out this approach, using OAM, we think having a centralized design system, like the one we’ve shown here, is a great solution for complex and/or open source projects. Want to help? Give the <a href="https://github.com/hotosm/oam-design-system">Design System</a> a try and send any suggestions or recommendations our way. We’re always looking to improve our workflow. A quick look at the docs is all that’s needed to start building. For more tech-savvy users, an example of its usage can be found in the <a href="https://github.com/hotosm/oam-docs/blob/master/gulpfile.js">OAM Docs gulpfile</a>.</p> 2016-07-14T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/07/14/reusable-design-components Development Seed Summer BBQ! http://www.developmentseed.org/blog/2016/07/07/bbq-gathering <figure class="align-center"> <a href="https://devseedbbq.splashthat.com"><img src="/assets/graphics/content/prose/devseedbbq.png" alt="RSVP" style="max-height: 300px" /></a> </figure> <p class="dropcap">Summer is here, so it must be time for a Development Seed summer BBQ. The whole team is in town and we are going to celebrate with our good friends, half smokes, and the finest blue ribbon beverages. Come by our new(ish) <a href="https://a.tiles.mapbox.com/v4/devseed.l73g0j47/attribution,zoompan.html?access_token=pk.eyJ1IjoiZGV2c2VlZCIsImEiOiJnUi1mbkVvIn0.018aLhX0Mb0tdtaT2QNe2Q#17/38.90635/-77.02437">home in Blagden Alley</a> on Wednesday, July 20.</p> <p>We’ll kick things off around 5:30. Bring friends and loved ones. Furry friends are always welcome!</p> <p>We’ll keep the food and beverages flowing. Help us out by <a href="https://devseedbbq.splashthat.com">letting us know if you plan to join the fun!</a></p> 2016-07-07T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/07/07/bbq-gathering Join Development Seed as an engineer http://www.developmentseed.org/blog/2016/06/21/engineer <p class="dropcap">We are looking for an engineer who wants to help organizations better understand and distribute their data. You’ll join a dynamic group of engineers and designers in DC and Lisbon tackling big global challenges with open data and open technology. You’ll collaborate with everyone on the team to produce products and systems with real impact.</p> <p>We’ll want you to design and develop APIs, tackling the problems that come with serving data at scale. You’ll create data pipelines that transform messy, real-world data into structured information. This could mean <a href="projects/astro-digital/">satellite imagery</a>, <a href="/projects/openaq/">air quality data</a>, or even <a href="/projects/wapo/">real-time election reporting</a>. For any project, you’ll get a chance to experiment with new technologies to best meet our client’s needs.</p> <p>Let’s talk about you. You are a great communicator and collaborator. You learn quickly and love trying new things. You know when to use a quick fix and when to invest more time refactoring. And you are looking to use technology to make a real difference with other likeminded people.</p> <p>Check out the <a href="/about/engineer/">job post</a> for more details.</p> 2016-06-21T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/06/21/engineer Run Mission Control at Development Seed http://www.developmentseed.org/blog/2016/06/17/mission-control <p class="dropcap">We are looking for someone to run <a href="/about/mission-control/">Mission Control</a> Development Seed. We are a dynamic group of engineers and designers in DC and Lisbon tackling big global challenges with open data and open technology. Mission Control will run all aspects of our business operations. You will work directly with me to help Development Seed grow, scale and maximize our impact.</p> <p>In Mission Control you will lead out all of our operational streams. You will manage our DC office, run events, streamline our business processes, and help the team to run faster. Every day will be different. On one day you might coordinate team travel; review a million dollar contract; buy a flock of drones; and help us to launch a new product.</p> <p>A little bit about you. You run fast and learn quickly. You don’t need to be a technologist, but you must embrace the open technology community and a humane startup culture. You are a great communicator. You instinctively distill commanders intent. You thrive in a flat organization. You take responsibility for your own time and get things done through orchestration rather than authority. You constantly look for ways to make things better. You are good person and you want to make a real difference with other fun, decent humans.</p> <p>If this sounds like you, take a gander at the <a href="/about/mission-control/">Mission Control job post</a>.</p> <figure class="align-center mono-spacing"> <img src="/assets/graphics/content/prose/datamapsgraphic.svg" width="718" height="425" alt="Data map graphic" /> </figure> 2016-06-17T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/06/17/mission-control Beautiful badges for OSM heroes http://www.developmentseed.org/blog/2016/06/09/mm-badges <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-mapathon.png" alt="Image" style="max-height: 300px" /> </figure> <p class="dropcap">Good illustrations carry a lot of weight. They communicate complex concepts and capture complex emotions. When you are use an illustration as a reward, it is all the more important to be thoughtful about the feelings that illustration inspires.</p> <p>We recently launched <a href="https://developmentseed.org/blog/2016/03/01/osm-stats-launched/">real-time analytics for Missing Maps</a>, a project that grows the community of humanitarian mappers around <a href="https://www.openstreetmap.org/">OpenStreetMap (OSM)</a>. As a part of the projects, we created a badge reward system that highlights the achievements of <a href="http://www.missingmaps.org/">Missing Maps</a> contributors. We created a style for these badges that would recognize the contributions of mappers, while also acknowledging that badges aren’t the primary reason that they participate. We also developed a style that is guided by collaborative design principles and is ready for others to contribute.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badge-heros.png" alt="Image" /> </figure> <h3 id="rewarding-heroes">Rewarding heroes</h3> <p>Our reward system was designed to recognize the contributions of humanitarian mappers, while being clear that we know they aren’t doing it for the badge. We needed to focus the badges on the real-world impact and contributions these mappers were having. We designed it to be fun and encouraging, while also recognizing that mapping would sometimes be done in the aftermath of disasters with devastating consequences.</p> <h4 id="friendly-unique-and-engaging">Friendly, unique, and engaging</h4> <p>The Missing Maps badge system provides a showcase for the work a person has made in contributing to OSM. A badge can provide a small milestone for what they can map next and can be an encouragement to map more. This can help cut down how repetitive, seemingly endless, and often thankless mapping out the world in OSM can be. Badges can help break up a long night of disaster mapping, or can let people know that it is time to take a break.</p> <p>We felt the badges should have a welcoming quality because MissingMaps relies heavily on volunteer effort. MissingMaps encourages contributions by making Mapathons fun, communal events (often with pizza). The importance of the work is the initial hook, but it’s the welcoming community that keeps people coming back, and the badges should reflect that.</p> <p>To make the badges feel like rewards we felt they needed to look interesting, unique, and beautifully designed. Something substantial that can feel more like a reward than a colored circle with text on top might. To this end, the badges were drawn in a modern style but with a fluid, non-gridded style to emphasize that each badge is a unique and thoughtful thing, created lovingly by hand.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badge-working.png" alt="Image" /> </figure> <h3 id="principles-for-missing-maps-badges">Principles for Missing Maps badges</h3> <p>Designing for future contributors is like designing for an internal team where you don’t know who teammates are going to be. We created guidelines, a template for getting started, and examples that serve as blueprints. We created a style that fits with the Missing Maps brand while capturing the goals of any reward system - make it friendly, make it feel substantial, and tell a story to keep the user engaged. And to keep it accessible for future contributors, we made it straightforward and open.</p> <h4 id="consistency--simplicity">Consistency &amp; Simplicity</h4> <p>We used a consistent and simple style for the Missing Maps user page badges to ease the burden on a contributor to make graphics that fit within the system. The elements that make up this style can be anything from line widths, to patterns, or other visual cues. For Missing Maps badges we use a color and the visual cues to create the sense of unity.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badgestyle.png" alt="Image" /> </figure> <p>Most of the badges are tiered, providing a natural flow to tell a story between badges of the same category. For example, our users are rewarded the “mapathoner badge” for attending mapathons. There are three levels for this badge, each rewarded for attending 5, 20, and 50 mapathons respectively. The badges for these levels each features a common character that starts out in a slight jog, then a sweating sprint, and finally reaches a finish line. This provokes the user to get to the next level in order to find out what might happen next, and also makes the badges feel like part of a set.</p> <figure class="align-center"> <img src="/assets/graphics/content/prose/mm-badge-story.png" alt="Image" /> </figure> <h3 id="open-source">Open Source</h3> <p>By making the badges entirely in digital tools and in the svg format we ensured most artists who would want to contribute to the project start out on a level playing field. There aren’t any fancy filters, proprietary fonts, or program-exclusive styling on these graphics.</p> <p>We’ve documented rules and guidelines for creating future badges and provided a template file for getting started. Everything is marked out on Github and available for anyone to contribute. Watch the repository for upcoming opportunities to work on new badges.</p> 2016-06-09T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/06/09/mm-badges OSM-in-a-box http://www.developmentseed.org/blog/2016/05/16/macrocosm-easy <p class="dropcap">We’ve been thinking about how to make <a href="https://github.com/developmentseed/macrocosm">Macrocosm</a>, our fork of the OpenStreetMap API, easier to use. Our vision is that one day you’ll be able to type <code>npm install openstreetmap</code> and be off to the races.</p> <p>We’re not there yet, but to that end, we’ve started to bundle <a href="http://ideditor.com/">iD</a> as a submodule in the Macrocosm repo. Now you can <a href="https://github.com/developmentseed/macrocosm#installation">set up your database, API, and editor all from the same repo</a>, using simple commands.</p> <p>We think there’s a lot of use for this deployable OSM. Testing, creating backups, and developing OSM applications all come to mind. Heck, you could even set up an awesome collaborative editing environment for you and your friends. You can kick the tires on <a href="https://github.com/developmentseed/macrocosm#installation">Macrocosm</a> or <a href="https://github.com/developmentseed/macrocosm/issues">contribute</a>, to get us closer to a full OSM-in-a-box.</p> 2016-05-16T00:00:00+00:00 Development Seed http://www.developmentseed.org/blog/2016/05/16/macrocosm-easy