Downloading all the back data


#42

I still have PC’s about and can write lines of instructions at the prompt like MSDOS or terminal. My first useful discovery in my shopping stage is learning about the versions of raspberry Pi and required accessories which are very well explained here: https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up
:slight_smile:


#43

O.k. Perfect. Do you have questions ?


#44

Buy a RPi 3B+, 2.5 Amp power supply and fast SDCard. That’s all you need to start.


#45

Lesson 2 during purchase…Seems to me it must be a MICRO SD card, as opposed to a ‘normal’ SD card. :slight_smile:


#46

Yes. Micro SD card it must be.


#47

@iladyman . Coincidentally, we’ve been working on a fresh IFTTT applet to easily log your 1min data automatically to a Google Sheet. The reason you got an error last night (24 October) was that we were literally in the midst of publishing the new applet — the one you reference above is the old stale version.

Take another look today as we just published the new version. The new version now has a complete set of data parameters as well as user-configured units of measure as requested. It’s really easy to setup…now that it actually works without errors. :wink: No extra hardware or software required.

PS - anyone using the original IFTTT applet to log data to Google Sheets should update to the new version for additional functionality.


#48

There are many many ways to download and the data collected by WeatherFlow. IFTTT is but one way. The best is a direct download and store into a full SQL database so you have access to the millions of records at one time. GoogleSheets is fine for small sets but it’s definitely NOT the solution.


#49

@iladyman - agree - there are ‘many’ ways to do this and many database variants and graphing/display engines you can pick from. I use grafana to graph influxdb timeseries data that I grab from MQTT via telegraf on a pi, but that is more of a learning exercise for me than anything. It can be done very simply.

If you want a gentle database introduction then I’d suggest fiddling around with sqlite3 if you haven’t played much with databases. Generating a quick schema and importing data is very trivial once you’ve done it once or twice to get your feet wet.

hint - buy a pi, and run weewx in simulator mode to generate simulated weather data, then poke around the data with sqlite3 commands. It’s far easier than a full-fledged highly available database like mysql/mariadb and the like. There’s also a very nice weewx driver for WeatherFlow that works great. If you want to clean your fake data, it’s as simple as deleting one file. Much simpler as a gentle introduction…

Run - do not walk - and get yourself some Linux training immediately. While being DOS fluent is great, they’re not the same. I’d suggest the free edx.org classes offered by the Linux Foundation as a gentle self-paced instruction (link) - just ignore the offfers and links asking you to pay for a certificate. The actual class is totally free.

For a known-good shopping list of gear that I bought a month or so ago:

The reason I suggest the CanaKit is that they’re a known-good supplier, and this kit really is a nice assembly of the minimum to get you going, including the required ‘good’ power supply. Don’t skimp on a couple dollars for your power supply, it’s the one thing that cause most SD card corruption historically.


split this topic #50

5 posts were merged into an existing topic: WFArchive - Data Archive Storage they were more related to wfarchive then to this topic.


ArchiveSW - Display & Data Archive Storage
#51

Hi @WFstaff and those wondering about implementing this,
I was able to implement the IFTTT applet. For those who havnt tried it and wonder about it I am posting what I learned in the process. I have learned that I didnt need to set anything up in google sheets first. But I did already have a google login using my gmail email address and password. When there are 2000 rows (one every minute) in the sheet (even if I copy the headers into the first row leaving space for 1999 rows of records) the first workbook containing one sheet is left as it is and a new workbook containing one sheet named “Smart Weather Station Data (1)” begins storing rows.

The data has the units of measure within the cell to make it easy to read. Which means they are text and to use the data as values the contents of the cells require parsing which is easy (for me) with extra formulas if required.


For my purpose of analysing historical data if required for a particular point in time when I can not access the data using the weatherflow App these google sheets can provide my history. :smile:
To use the sheets for displaying history will require some work but I can easily imagine how I could do it if I wanted to spend the time but I will experiment with other storage and display options first.
Several have commented about the problem of the 2000 row limit from IFTTT. I discovered a suggestion how to get around it by using a script which copies the rows from the active IFTTT sheet onto an archive sheet and deletes the rows from the IFTTT sheet to maintain it less than 2000 rows.
Summary review of the IFTTT applet is that it is free, easy, no other hardware is required, no computer needs to be left running, and provides our one minute history stored for review if required. It is not so easy to find and display a graph of that history but it is possible if you understand spreadsheets. And if you are keen it would be possible to have scripts running to analyse and respond to the data coming in.
Thankyou @WFstaff
cheers Ian :smile: