Scraping NOAA and CDIAC

Given the various “rumors” about a large defunding of various “Climate Programs”, that might well include some of the data archives at NOAA or CDIAC, I thought I’d ‘remind’ folks how to scrape a site for data.

This is particularly important for CDIAC (Carbon Dioxide Information Analysis Center) as they are having a “Going Out Of Business” sale on data at the moment… Per their website (linked above):

NOTICE: CDIAC as currently configured and hosted by ORNL will cease operations on September 30, 2017. Data will continue to be available through this portal until that time. Data transition plans are being developed with DOE to ensure preservation and availability beyond 2017.

Well, IMHO it doesn’t matter if you are a True Global Warming Believer, or a Died In The Wool Skeptic:

Data ought to be preserved. Period. FULL stop.

So be you skeptic or AGW Activist, IF you want to assure the ‘transition plan’ doesn’t drop things, well, you can always get your own copy.

I covered that to some extent in earlier postings here:

https://chiefio.wordpress.com/2015/10/06/cdiac-scrape-finished/

https://chiefio.wordpress.com/2015/10/01/noaa-data-scrape-completes-at-last/

https://chiefio.wordpress.com/2015/08/28/cloning-ushcn-daily-data/

But right now I think a ‘quick review’ might be in order.

As you can see, I have a 2015 copy from each of those. I’ve since updated most of them. But, should any of you want your own copy, here’s how to do it. I would advise using the “polite” options that rate limit, especially if you share a line with the rest of the family, OR if you are on a very very fast line. If a dozen folks on very fast links run scrapes wide-open, the admins will end up quashing it and nobody wins. I only ran mine wide open as 1) It was only me. 2) I was only updating a substantially complete copy. 3) I’m on an Orange Pi $16 computer on a slow link so can’t speed slam anybody if I tried…

This script is an example of how you can collect the contents of a web site. (several examples, really, as each scrape has a couple of commented out examples too). Just “uncomment” by removing the leading “#” any particular site you would like to ‘scrape’ with the options you choose (mostly it is nice to use the -w to wait a few seconds between file pulls or the –limit-rate to make sure you don’t saturate your internet connection and have the spouse or boss breathing down your neck…)

You will want to change the “cd …/Archives” line to have your destination directory instead of the /Archives one. This can be passed in at execution time as parameter 1, so you could say “syncnoaa /My/Directory/On/BIG_disk” and have it send stuff there.

I’ve uncommented the more “rude” example of “just make a mirror pronto” versions of the command. As explained above I was a “man in a hurry” on a slow link with a $16 computer and $59 disk drive so wasn’t likely to slam anybody with rapid data demands… Tune your wget line as appropriate to be a good net citizen AND preserve copies of the data. (Yes, all up, including the $25 hub and $7 power supply, the needed hardware and software to build an archive from scratch is about $107 plus shipping. Call it $125 max. I got free shipping on most of mine.)

chiefio@orangepione:~$ cat bin/syncnoaa 
# Fetch a mirrored copy of the NOAA, GHCN Daily temperature data,
# or cdiac web sites.
#
# wget is the command that does the fetching.  
# It can be fed an http: address or an ftp: address.
#
# The -w or --wait command specifies a number of seconds to pause 
# between file fetches.  This helps to prevent over pestering a 
# server by nailing a connection constantly; while the 
# --limit-rate={size} limits via an average of pausing between 
# transfers.  Over time this is about the rate of bandwidth used, 
# but on a gaggle of small files can take a while to stabilize, 
# thus the use of both.
#
# Since CDIAC uses a "parent" link that points "up one" you need 
# to not follow those or you will end up duplicating the whole 
# structure ( I know... don't ask...) thus the -np or 
# --no-parent option.
#
# The -m or --mirror option sets a bunch of other flags (in effect)
# so as to recursively copy the entire subdirectory of the target 
# given in the address.  Fine, unless they use 'parent' a lot...
#
# Then you list the http://name.site.domain/directory or 
# ftp://ftp.site.domain/directory to clone
#
# Long form looks like:
#
# wget --wait 10 --limit-rate=100k --no-parent --mirror http://cdiac.ornl.gov/ftp/ushcn_daily
#
# but I think the --commands look silly and are for people who can't
# keep a table of 4000 things that -c does in 3074 Unix / Linux 
# commands, all of them different, in their head at all times ;-) 
# so I use the short forms.  Eventually not typing all those wasted
# letters will give me decades more time to spend on useful things,
# like comparing the merits of salami vs. prosciutto... 

#Flags:  -w n {wait n seconds}  --limit-rate=fook {limit rate to foo in k}
#        -np  {do not follow parent links upward}  
#	-nc  --no-clobber:  Complex, but leave old copies. Default for -r or -p
#	can not be specified with -N as -N says keep the newer time stamp.
#	-N  Use time stamping
#        -r  recursively follow the directory structure to 5 deep.
#	-l n  Go n deep on recursion instead of the default 5.
#	--no-remove-listing  keep the listing of what is on the server
#  	-m
#       --mirror
#           Turn on options suitable for mirroring.  This option turns on
#           recursion and time-stamping, sets infinite recursion depth and keeps
#           FTP directory listings.  It is currently equivalent to -r -N -l inf
#           --no-remove-listing.


cd ${1-/Archives}

# cdiac.ornl.gov USHCN Daily

echo
echo Doing cdiac.ornl.gov USHCN Daily
echo

wget -m -np http://cdiac.ornl.gov/ftp/ushcn_daily

#wget -m -np -w 10 http://cdiac.ornl.gov/ftp/ushcn_daily

#wget -w 10 --limit-rate=100k -np -m http://cdiac.ornl.gov/ftp/ushcn_daily

#wget -r -N -l inf --no-remove-listing -w 10 --limit-rate=100k -np http://cdiac.ornl.gov/ftp/ushcn_daily

echo
echo Doing World Weather Records
echo

wget -np -m ftp://ftp.ncdc.noaa.gov/pub/data/wwr/

#wget -np -m -w 20 ftp://ftp.ncdc.noaa.gov/pub/data/wwr/

#wget --limit-rate=100k -np -m ftp://ftp.ncdc.noaa.gov/pub/data/wwr/

#wget --limit-rate=100k -nc -np -r -l inf ftp://ftp.ncdc.noaa.gov/pub/data/wwr/

echo
echo Doing World War II Data
echo

wget -np -m ftp://ftp.ncdc.noaa.gov/pub/data/ww-ii-data/

#wget -np -m -w 20  ftp://ftp.ncdc.noaa.gov/pub/data/ww-ii-data/

#wget --limit-rate=100k -np -m ftp://ftp.ncdc.noaa.gov/pub/data/ww-ii-data/

#wget --limit-rate=100k -nc -np -r -l inf ftp://ftp.ncdc.noaa.gov/pub/data/ww-ii-data/

echo
echo Doing NOAA set
echo

wget -np -m  ftp://ftp.ncdc.noaa.gov/pub/data/noaa/

#wget -np -m  -w 10 ftp://ftp.ncdc.noaa.gov/pub/data/noaa/

#wget --limit-rate=100k -np -m ftp://ftp.ncdc.noaa.gov/pub/data/noaa/

#wget -nc -np -r -l inf ftp://ftp.ncdc.noaa.gov/pub/data/noaa/

echo
echo Doing Global Data Bank set
echo

wget  -np -m ftp://ftp.ncdc.noaa.gov/pub/data/globaldatabank/

#wget -w 10 --limit-rate=100k -np -m ftp://ftp.ncdc.noaa.gov/pub/data/globaldatabank/

#wget  -np -m -w 10 ftp://ftp.ncdc.noaa.gov/pub/data/globaldatabank/

echo
echo Doing GHCN
echo

wget  -np -m ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/

#wget -w 10 --limit-rate=100k -np -m ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/

#wget  -np -m -w 10 ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/

echo
echo  ALL DONE!!! 
echo

Note that this takes a LOT of disk and a LOT of time. On the order of a week on a moderately fast home network and about 3/4 of a TB of disk would be a good place to start. (A 1 TB disk better as you might want to unpack some things and look them over).

You can use this basic syntax on just about any ftp: or http: website, with a little tuning sometimes.

Subscribe to feed

Advertisement