2020 ESIP Winter Meeting

The Council of Data Facilities (CDF) is committed to working with relevant agencies, professional associations, initiatives, and other complementary efforts to enable transformational science, innovative education, and informed public policy through increased coordination, collaboration, and innovation in the acquisition, curation, preservation, and dissemination of geoscience data, tools, models, and services. Existing and emerging geoscience data facilities – through the Council – are committed to serving as an effective foundation for EarthCube. The General Assembly meeting is open to the official representatives from all member data facilities, additional member organization personnel as desired by the members, as well as observers. How to

Agenda:
400-415 Welcome/introductions/sign-in - Danie415-430 High level Summary of OKN workshop - TBA
430-435 Updates on shared infrastructure - Kerstin, Danie

• White paper/report Sent to Eva, please send to your program managers

435-445 Update on COPDESS-Kerstin, Shelley
445-515 Update and next steps on P419-Doug, Adam
515-530 Progress on EC supplements for CCHDO and MagIC related to P418/P419 (GeoCODES)-Steve
530-550 Update from tech team EarthCube Office-Kenton McHenry
550-600 Summer topics - Danie

• Suggested Charter changes (to be voted on at july 2020)
• Announce  CDF exec elections in july 2020 - 2 co-chair and 3 at large positions

SpatioTemporal Asset Catalogs is an emerging specification of a common metadata model for geospatial data, and a way to make data catalogs indexable and searchable. We have already seen STAC being adopted for both public data and commercial data. Catalogs exist for several AWS Public Datasets, Landsat Collection 2 data will be published along with STAC metadata, and communities like Pangeo are using STAC to organize data repositories in a scalable way. Commercial companies like Planet and Digital Globe are starting to publish STAC metadata for some of their catalogs. Session talks may cover overviews of the STAC, software projects utilizing STAC, and use cases of STAC in organizations. How to Prepare for this Session: See https://stacspec.org/.

View Recording:https://youtu.be/BdZbJLQSNFE.

Takeaways

• Checkout STAC at https://stacspec.org/ and follow along (or join) the discussion at https://gitter.im/SpatioTemporal-Asset-Catalog/Lobby
• See live STAC cataloges at stac.cloud. Some additional tools worth checking out include the STAC validator https://github.com/sparkgeo/stac-validator and STAC web linter http://staclint.com/.

NASA, NOAA and USGS are in the process of moving data onto the cloud. While they have discussed what types of services are available and future plans of what data can be found, it is not completely clear what datasets users can currently access. This session will go over what datasets are currently up in the cloud and what data to expect in the near future. This way as users are transitioning to the cloud for their compute, they can also know what data are available to them on the cloud as well. There will also be presentations from AWS. Speakers:
Katie Baynes - NASA/EOSDIS
Jon O'Neil - NOAA
Jeff de La Beaujardiere - NCAR
Kristi Kliene - USGS/EROS
Joe Flasher - AWS

Presentations: See attached.

View Recording: https://youtu.be/yssgXB7iaxw

Takeaways

• Petabyte scale data is being moved into the cloud. This is concentrated in AWS, Google Cloud and Microsoft depending on the agency and dataset
• Some concern around partnerships with companies (AWS most discussed) in terms of long term relationships, moving data etc. and how those things might impact access or data use
• Need to make clear the authoritative source of the data, who is stewarding it, and any modifications done when copying to cloud. Users should exercise due diligence in selecting and using data.

We will describe our project to upload a 2.4 PB dataset encapsulated into ~80K fused files from the 5 instruments on the Terra satellite into NASA AWS S3.
We will share the bottlenecks points and lessons learned during this process and expect to share experiences with similar projects in order to understand the best practices and collect guidelines for future projects that are adopting cloud solutions for their data needs.

We'll discuss data volumes, data integrity strategies for migration, S3 bucket organization, metadata curation, transfer rates, transfer pipelines, etc. We will also discuss and share data access patterns, costs, and architectures and how we can construct guidelines for access to these datasets efficiently.

We encourage the discussion among different projects that faced similar processes or are looking to migrate their datasets into the cloud.

https://drive.google.com/file/d/1fts06XDM2dbZxxljBTpplCEMSiTqfp6t/view?usp=sharing

Presentations:
https://doi.org/10.6084/m9.figshare.11553147.v1

View Recording: https://youtu.be/1xVJghJI4Gg

Takeaways

• Project required/used a combination of NSF, NASA and AWS resources. Some interesting discussion around AWS or other cloud services as a stand in or follow on to limited term NSF assets
• Some interesting discussion of tailoring to appropriate end users- wide range of potential users and thus requirements for the dataset. This includes access guidelines, user capabilities etc.
• Project aimed to make a paradigm shift from understanding/observing physical processes to a full climate observing objective

The NSF-funded Pangeo project (http://pangeo.io/) is a community-driven architectural framework for big data geoscience. A typical Pangeo software stack leverages Python open-development libraries including elements such as Jupyter Notebooks for interactive data analysis, Intake catalogs to provide a higher level of abstraction, Dask for scalable, parallelized data access, and Xarray for working with labeled multi-dimensional arrays of data, and can support data formats including NetCDF as well the cloud-optimized Zarr format for chunked, compressed, N-dimensional arrays.

This session includes presentations describing implementations, results, or lessons learned from using these tools, as well as some time for open discussion. We encourage attendance by people interested in knowing more about Pangeo.

Draft schedule:
Dr. Amanda Tan, U. Washington: Pangeo overview and lessons learned
Dr. Rich Signell, USGS: The USGS EarthMap Pangeo: Success Stories and Lessons Learned
Dr. Jeff de La Beaujardière, NCAR: Climate model outputs on AWS using Pangeo framework
Dr. Karl Benedict, UNM: Pangeo as a platform for workshops
Open discussion

How to Prepare for this Session:

Presentations:
https://doi.org/10.6084/m9.figshare.11559174.v1

View Recording: https://youtu.be/VNfpGIIjL3E.

Takeaways

• Pangeo is a community platform for Big Data geoscience; A cohesive ecosystem of open community, open source software, open ecosystem; Three core python packages: jupyter, xarray, Dask
• Deploying Pangeo on cloud face challenges
  • Cloud costs
  • Cloud skills
  • Need of cloud-optimized data
  • Best strategy of pangeo deployment in the changing cloud services platform
• Pangeo can be applied to leverage the jupyter notebook and other resources for different level of data users (NCAR: scientists new to cloud computing platform; University of New Mexico: workshop platform etc)

This session provides an overview of the results from the recent OGC Research & Development initiative Testbed-15. The 9-months 5M USD initiative addressed six different topics, Earth Observation Process and Application Discovery, Machine Learning, Federated Cloud Analytics, Open Portrayal Framework, Delta Updates, and Data Centric Security. This session focuses on the results produced by the first three.

Earth Observation Process and Application Discovery developed draft specifications and models for discovery of cloud-provided process and applications. This was achieved by extending existing standards with process and application specific extensions. Now, data processing software can be made available as a service, discovered using catalog interfaces, and executed on demand by customers. This allows to locate the process execution physically close to the data and reduces data transport overheads.

The Machine Learning research developed models in the areas of earth observation data processing, image classification, feature extraction and segmentation, vector attribution, discovery and cataloguing, forest inventory management & optimization, and semantic web-link building and triple generation. Both model discovery and access took place through standardized interfaces.

The Federated Cloud Analytics research analysed how to handle data and processing capacities that are provided by individual cloud environments transparently to the user. The research included how federated membership, resource, and access policy management can be provided within a security environment, while also providing portability and interoperability to all stakeholders. Additionally, the initiative conducted a study of the application of Distributed Ledger Technologies (DLTs), and more specifically Blockchains, for managing provenance information in Federated Cloud.

The other three topics will be briefly introduced in addition. The Open Portrayal Framework provides a fully interoperable portrayal and styling suite of standards. Here, the initiative developed new OGC APIs for styles, maps, images, and tiles. Delta updates explored incremental updates and thus reduced communication payloads between clients and servers, whereas the Data Centric Security thread examined the use of encrypted container formats on standard metadata bindings. How to Prepare for this Session: Al results will be made available as public Engineering Reports that provide full details. These become stepwise available at http://docs.opengeospatial.org/per/

Presentations:
https://doi.org/10.6084/m9.figshare.11551563.v1

View Recording: https://youtu.be/ojMrcIE-SgE

Takeaways

• OGC innovation program: Test fitness for purpose of geospatial community initiatives. TESTBED-15 concluded last November results available soon from document repository. End to end cloud pipeline for data processing and analytics. Call for TESTBED-16 due Feb 9th 2020! 1.6M in funding available. Three major threads: earth observation clouds, data integration and analytics, and modeling and packaging. 
• Way to synergize between needs of user communities competing and collaborating projects, contributing to a more interoperable world. Provides applications, process and catalogues for data processing. 
• Testbeds center around an exploitation/processing platform (for data with relevant applications) like an application market with cloud services. Having some trouble finding application developers. Finding web services with relevant data can be problematic.
  
  
  

Thanks to the rapid developments of hardware and computer science, we have seen a lot of exciting breakthroughs in self driving, voice recognition, street view recognition, cancer detection, check deposit, etc. Sooner or later the fire of AI will burn in Earth science field. Scientists need high-level automation to discover in-time accurate geospatial information from big amount of Earth observations, but few of the existing algorithms can ideally solve the sophisticated problems within automation. However, nowadays the transition from manual to automatic is actually undergoing gradually, a bit by a bit. Many early-bird researchers have started to transplant the AI theory and algorithms from computer science to GIScience, and a number of promising results have been achieved. In this session, we will invite speakers to talk about their experiences of using AI in geospatial information (GI) discovery. We will discuss all aspects of "AI for GI" such as the algorithms, technical frameworks, used tools & libraries, and model evaluation in various individual use case scenarios. How to Prepare for this Session: https://esip.figshare.com/articles/Geoweaver_for_Better_Deep_Learning_A_Review_of_Cyberinfrastructure/9037091
https://esip.figshare.com/articles/Some_Basics_of_Deep_Learning_in_Agriculture/7631615

Presentations:
https://doi.org/10.6084/m9.figshare.11626299.v1

View Recording: https://youtu.be/W0q8WiMw9Hs

Takeaways

• There is a significant uptake of machine learning/artificial intelligence for earth science applications in the recent decade;
• The challenge of machine learning applications for earth science domain includes:
  • the quality and availability of training data sets;
  • Requires a team with diverse skill background to implement the application
  • Need better understanding of the underlying mechanism of ML/AI models
• There are many promising applications/ developments on streamlining the process and application of machine learning applications for different sectors of the society (weather monitoring, emergency responses, social good)
• 
  
• 
  

Citizen science is scientific data collection and research performed primarily or in part by non-professional and amateur scientists. Citizen science data has been used in a variety of the physical sciences, including physics, ecology, biology, and water quality. As volunteer-contributed datasets continue to grow, they represent a unique opportunity to collect and analyze earth-science data on spatial and temporal scales impossible to achieve by individual researchers. This session will explore the ways open citizen science data sets can be used in earth science research and some of the associated challenges and opportunities for the ESIP community to use and partner with citizen science organizations.

Speakers:

• Ruth Duerr, Citizen Science Examples and Issues https://tinyurl.com/ydszbeqv
  View Slides:https://doi.org/10.6084/m9.figshare.11549256.v1
• Rebecca Forkner, Improving `big unstructured’ citizen data to monitor diversity
• Lourdes Vera
• Brian Hostetler
• Alexis Garretson, Citizen Science Data Management Challenges and Opportunities
  View Slides: https://doi.org/10.6084/m9.figshare.11559081.v1

View Recording: https://youtu.be/jTNgWZI6Cik

Takeaways


How to Prepare for this Session: https://www.nationalgeographic.org/encyclopedia/citizen-science/
http://www.earthsciweek.org/citizen-science

This workshop session will follow up on the OGC Coverage Analytics sprint, focusing specifically on advanced services for spatio-temporal datacubes. In the Earth sciences datacubes are accepted as an enabling paradigm for offering massive spatio-temporal Earth data analysis-ready, more generally: easing access, extraction, analysis, and fusion. Also, datacubes homogenizes APIs across dimensions, allowing unified wrangling of 1-D sensor data, 2-D imagery, 3-D x/y/t image timeseries and x/y/z geophysics voxel data, and 4-D x/y/z/t climate and weather data.
Based on the OGC datacube reference implementation we introduce datacube concepts, state of standardization, and real-life 2D, 3D, and 4D examples utilizing services from three continents. Ample time will be available for discussion, and Internet-connected participants will be able to replay and modify many of the examples shown. Further, key datacube activities worldwide, within and beyond Earth sciences, will be related to.
Session outcomes could take a number of forms: ideas and issues for OGC, ISO, or ESIP to consider; example use cases; challenges not yet addressed sufficiently, and entirely novel use cases; work and collaboration plans for future ESIP work. Outcomes of the session will be reported at the next OGC TC meeting's Big Data and Coverage sessions. How to Prepare for this Session: Introductory and advanced material is available from http://myogc.org/go/coveragesDWG

Presentations
https://doi.org/10.6084/m9.figshare.11562552.v1

View Recording: https://youtu.be/82WG7soc5bk

Takeaways

• Abstract coverage construct defines the base which can be filled up with a coverage implementation schema. Important as previously implementation wasn’t interoperable with different servers and clients. 
• Have embedded the coordinate system retrieved from sensors reporting in real time into their xml schema to be able to integrate the sensor data into the broader system. Can deliver the data in addition to GML but JSON, and RDF which could be used to link into semantic web tech. 
• Principle is send HTTP url-encoded query to server and get some results that are extracted from datacube, e.g., sources from many hyperspectral images.