Go Back   This Blue Marble, a Global Current Events Discussion Forum > Main Floor > Science Center

Science Center Our science center is equipped to facilitate a variety of general scientific discussions, including physics, environment science, astronomy and technology research.

Reply
 
Thread Tools Search this Thread Display Modes
Old 01-19-2017, 10:39 AM   #1
Potemkin
Omne ignotum pro magnifico
 
Potemkin's Avatar
 
Join Date: Aug 2008
Posts: 27,932
Blog Entries: 3
Thanks: 110
Thanked 5,660 Times in 2,809 Posts
Default Rogue Scientists Race to Save Climate Data from Trump

No one wants to work on getting the "deep web" saved and indexed and one campaign and someone figures it out.

Hopefully someone will continue the work and get all that non-Google indexed data in the Government out.

https://www.wired.com/2017/01/rogue-...te-data-trump/

Rogue Scientists Race to Save Climate Data from Trump

At 10 AM the Saturday before inauguration day, on the sixth floor of the Van Pelt Library at the University of Pennsylvania, roughly 60 hackers, scientists, archivists, and librarians were hunched over laptops, drawing flow charts on whiteboards, and shouting opinions on computer scripts across the room. They had hundreds of government web pages and data sets to get through before the end of the day—all strategically chosen from the pages of the Environmental Protection Agency and the National Oceanic and Atmospheric Administration—any of which, they felt, might be deleted, altered, or removed from the public domain by the incoming Trump administration.
Their undertaking, at the time, was purely speculative, based on travails of Canadian government scientists under the Stephen Harper administration, which muzzled them from speaking about climate change. Researchers watched as Harper officials threw thousands of books of aquatic data into dumpsters as federal environmental research libraries closed.

But three days later, speculation became reality as news broke that the incoming Trump administration’s EPA transition team does indeed intend to remove some climate data from the agency’s website. That will include references to President Barack Obama’s June 2013 Climate Action Plan and the strategies for 2014 and 2015 to cut methane, according to an unnamed source who spoke with Inside EPA. “It’s entirely unsurprising,” said Bethany Wiggin, director of the environmental humanities program at Penn and one of the organizers of the data-rescuing event.

Back at the library, dozens of cups coffee sat precariously close to electronics, and coders were passing around 32-gigabyte zip drives from the university bookshop like precious artifacts.

The group was split in two. One half was setting web crawlers upon NOAA web pages that could be easily copied and sent to the Internet Archive. The other was working their way through the harder-to-crack data sets—the ones that fuel pages like the EPA’s incredibly detailed interactive map of greenhouse gas emissions, zoomable down to each high-emitting factory and power plant. “In that case, you have to find a back door,” said Michelle Murphy, a technoscience scholar at the University of Toronto.

Murphy had traveled to Philly from Toronto, where another data-rescuing hackathon had taken place a month prior. Murphy brought with her a list of all the data sets that were too tough for the Toronto volunteers to crack before their event ended. “Part of the work is finding where the data set is downloadable—and then sometimes that data set is hooked up to many other data sets,” she said, making a tree-like motion with her hands.

At Penn, a group of coders that called themselves “baggers” set upon these tougher sets immediately, writing scripts to scrape the data and collect them in data bundles to be uploaded to DataRefuge.org, an Amazon Web Services-hosted site which will serve as an alternate repository for government climate and environmental research during the Trump administration. (A digital “bag” is like a safe, which would alert the user if anything within it is changed.)

“We’re yanking the data out of a page,” said Laurie Allen, the assistant director for digital scholarship in the Penn libraries and the technical lead on the data rescuing event. Some of the most important federal data sets can’t be extracted with web crawlers: Either they’re too big, or too complicated, or they’re hosted in aging software and their URLs no longer work, redirecting to error pages. “So we have to write custom code for that,” Allen says, which is where the improvised data-harvesting scripts that the “baggers” write will come in.

More at link
__________________
“The price of freedom is the willingness to do sudden battle anywhere, any time and with utter recklessness.”
― Robert A. Heinlein, The Puppet Masters
Potemkin is offline   Reply With Quote
Reply

Tags
climate, data, race, rogue, save, scientists, trump

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -4. The time now is 11:50 PM.


Powered by vBulletin®
Copyright © Jelsoft Enterprises Ltd.