ArchiveSW - Display & Data Archive Storage

[GaryFunk]

Which Raspberry Pi are you using?

[phil]

Raspberry Pi 3B+ It’s the latest incarnation and requires a distribution dated March 2018 or later. The GUI installation does an automatic apt update && apt upgrade at first boot.

$ cat /etc/os-release
PRETTY_NAME=“Raspbian GNU/Linux 9 (stretch)”
NAME=“Raspbian GNU/Linux”
VERSION_ID=“9”
VERSION=“9 (stretch)”
ID=raspbian
ID_LIKE=debian
HOME_URL=“http://www.raspbian.org/”
SUPPORT_URL=“http://www.raspbian.org/RaspbianForums”
BUG_REPORT_URL=“http://www.raspbian.org/RaspbianBugs”

and

$ uname -a
Linux raspberrypi 4.14.70-v7+ #1144 SMP Tue Sep 18 17:34:46 BST 2018 armv7l GNU/Linux

[GaryFunk]

I don’t have a clue. It’s failing on the nodejs Installation.

I’ll run through it on the latest files and see what I can find.

[phil]

Many thanks Gary - it saves me writing something in Python!

[bpaauwe]

@GaryFunk

At least I now understand why you are having issues. I will add logging to capture as many issues as possible.

Good thing I have this bad data to test with. I’ve worked through a few more cases where things are out of range so I’m getting closer.

I’d say the biggest issue is that the current error messages don’t really provide any info to track down the actual problem. Better logging will make a big difference. I’ve been adding logging of each SQL command just so that when it hits one of these, I know what it was trying to do.

So maybe having a verbose or debug flag that can enable really verbose logging.

Also, is it possible to output a better error message? For example, when I run the SQL on the table with the out of range values I get:

ERROR 1264 (22003): Out of range value for column ‘report_interval’ at row 35941

That gives me enough info to track down the bad value, look at it and change it.

[GaryFunk]

@bpaauwe

ERROR 1264 (22003): Out of range value for column ‘report_interval’ at row 35941

That comes from the SQL server and is as good as it gets. It means that the value of row 35941 is larger than 117. What you can do is simply set all rows to 1, since that is the only viable value.

Face it Bob, you and I really screw up data like no others.

[bpaauwe]

I’m now getting a failure with this SQL:
var SQL = “SELECT column_name FROM information_schema.columns WHERE table_schema=’” + db + "’ AND table_name = wftest AND ordinal_position = " + newPosition;

I think it’s because table_name = wftest needs to have wftest in quotes.

With that fixed, alternodes then crashes because the results of that query are empty. I’m not sure what this is trying to do so I’m a bit stuck debugging any farther.

[GaryFunk]

Each time you run it starts over so the first entry in Z_diff is what it is working on. I wish I had your table schema.

[GaryFunk]

I have a suggestion.

Wipe the SDCard and start fresh with a new OS install, then manually install Nodejs before you do anything else.

[bpaauwe]

MariaDB [weatherflow]> SELECT column_name FROM information_schema.columns WHERE table_schema=‘weatherflow’ and table_name=‘wftest’;
Empty set (0.00 sec)

MariaDB [wftest]> select * from ZZ_diff;
±-------------±-----------±------------±-----------------±---------------±----------±------------±--------+
| table_schema | table_name | column_name | ordinal_position | column_default | data_type | column_type | changed |
±-------------±-----------±------------±-----------------±---------------±----------±------------±--------+
| wftest | Xpiio | marked | 6 | ‘N’ | char | char(1) | |
±-------------±-----------±------------±-----------------±---------------±----------±------------±--------+
MariaDB [weatherflow]> show tables;
±----------------------+
| Tables_in_weatherflow |
±----------------------+
| AirBackfill |
| AirObservation |
| DailyAir |
| DailySensor |
| DailySky |
| DeviceEvents |
| DeviceStatus |
| DySensor |
| FWUpdate |
| HubEvents |
| HubStatus |
| PrecipEvent |
| RapidWind |
| SkyBackfill |
| SkyObservation |
| StrikeEvent |
| Xpiio |
| Xrain |
| Z_debug |
| Zdupe |
| Zudp |
±----------------------+

There’s no table called wftest in either the weatherflow database or the wftest database and my knowledge of SQL is pretty basic, so I’m confused by the ‘table_name = wftest’ in the select statement.

[GaryFunk]

Each time you run altertables it creates the database, creates the tables, then drops the database on exit.

[GaryFunk]

@bpaauwe

table_name = wftest AND

OHHHHHHH. Crap. That’s WRONG. Hang on.

[bpaauwe]

should be table_name = ‘" + table + "’ maybe?

[GaryFunk]

@GaryFunk

table_name = wftest

“SELECT column_name FROM information_schema.columns WHERE table_schema = wftest AND table_name = '” + table + "’ AND ordinal_position = " + newPosition;

I meant to hard code the schema variable. It’s part of being able to rename columns that I am working on.

[bpaauwe]

So table_schema is hard coded as wftest?

Ok, I think it got to the end but does throw a ER_PARSE_ERROR after dropping database wftest.

[GaryFunk]

@bpaauwe

So table_schema is hard coded as wftest?

Yes, on line 267 it should be.

[phil]

I’ll give that a try later today.

[phil]

Gary, Managed to load everything following your suggestions but I ended up doing a manual installation in the end.

Server starts up but when I try to enter the station details I but when I press ‘OK’, I get this:-

TypeError: Assignment to constant variable.
at /home/pi/wfarchive/server.js:216:9
at Layer.handle [as handle_request] (/home/pi/wfarchive/node_modules/express/lib/router/layer.js:95:5)
at next (/home/pi/wfarchive/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/home/pi/wfarchive/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/home/pi/wfarchive/node_modules/express/lib/router/layer.js:95:5)
at /home/pi/wfarchive/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/home/pi/wfarchive/node_modules/express/lib/router/index.js:335:12)
at next (/home/pi/wfarchive/node_modules/express/lib/router/index.js:275:10)
at /home/pi/wfarchive/node_modules/body-parser/lib/read.js:130:5
at invokeCallback (/home/pi/wfarchive/node_modules/raw-body/index.js:224:16)

I am just doing a cut and paste to transfer the station details from the info page

Phil

[phil]

The plot thickens!
In spite of the error above it seems to have worked. I started up the archive process and data appears in the database - so it seems to be running - at present. Shall leave it for an overnight catchup

Phil

[phil]

Ok Final couple of issues for tonight!

1: getting errors in log files that look as if some table field names are wrong.
e.g. in Archive.log
2018-10-07 17:59:36 Starting Archive v1.6.12.034
2018-10-07 17:59:37 Broadcasting UDP on: :50225
2018-10-07 17:59:37 MariaDB connected
2018-10-07 17:59:37 appRestart : :
2018-10-07 17:59:37 Listening to Hub on: 0.0.0.0:50222
2018-10-07 17:59:37 Listening to Panel on: 0.0.0.0:50226
2018-10-07 18:00:00 Archive is running
2018-10-07 18:00:01 Error(a2): ER_BAD_FIELD_ERROR: Unknown column ‘presH’ in ‘field list’
2018-10-07 18:00:23 Error(s2): ER_BAD_FIELD_ERROR: Unknown column ‘illuH’ in ‘field list’
2018-10-07 18:01:01 Error(a2): ER_BAD_FIELD_ERROR: Unknown column ‘presL’ in ‘field list’
2018-10-07 18:01:23 Error(s2): ER_BAD_FIELD_ERROR: Unknown column ‘illuL’ in ‘field list’
2018-10-07 18:02:23 Error(s2): ER_BAD_FIELD_ERROR: Unknown column ‘illuL’ in ‘field list’
2018-10-07 18:03:01 Error(a2): ER_BAD_FIELD_ERROR: Unknown column ‘presL’ in ‘field list’
2018-10-07 18:03:23 Error(s2): ER_BAD_FIELD_ERROR: Unknown column ‘illuL’ in ‘field list’
2018-10-07 18:04:01 Error(a2): ER_BAD_FIELD_ERROR: Unknown column ‘tempL’ in ‘field list’

etc

and
2/ Starting Panel process
get connection error in browser

192.168.1.163 refused to connect.
I am using the correct port to attempt connection also no Panel log created

Phil