I am experimenting with the Rest API to extract all observations between a specified time_start and time_end. Unfortunately I’m struggling to understand the output. For example, if I submit the following request:
I get output from the latest observation with (in this case) a UNIX timestamp equal to 1524033360 or 04/18/2018 @ 6:36am (UTC). Now, if I subtract 10800 seconds from that timestamp in order to use time_start=1524022560 (or 04/18/2018 @ 3:36am (UTC)) and time_end=1524033360 to extract all observations in the three hours preceding the latest observation using this command:
Then the first observation returned has a UNIX timestamp of 1524023100 which corresponds to 04/18/2018 @ 3:45am (UTC). The question then is what has happened to the observations between 3:36am (UTC) and 3:45am (UTC), and why are they not being returned by the API request?
If for the sake of argument I change time_start to 1524020000 (04/18/2018 @ 2:53am (UTC)):
Then I can find the observation at 3:36am (UTC), so it is not an issue that the observation doesn’t exist. However, again the first observation returned by the API request has a UNIX timestamp of 1524020580 or 04/18/2018 @ 3:03am (UTC), a whole 10 minutes after the specified time_start.
Any help on understand where these missing observations are would be much appreciated!
Many thanks.
First, 152402000 is actually 2018-04-18T02:53:20+00:00 so if you add 40 seconds to the second time example you will see the both examples are exactly 540 seconds off.
They both skip the fitst nine minutes. That’s very strange and I never paid any attention to that. David (@dsj) or Corrine (@corrineb) will have to inquire about this.
Hi @corrineb. I appreciate everyone at WF is probably incredibly busy, and this is likely not the issue at the top of your to-do list, but would you be able to provide an update on when this bug will be fixed?
Another thing I have noticed is that if you use time_start, time_end or day_offset in your query, all the timestamps in the resultant JSON message are rounded to the minute. Others may have a different opinion, but for me I am not sure that this should be the expected behavior. The timestamp should still respect exactly when the observation was taken.
We took a look and were able to identify an issue with the API and the time_start parameter. We are incorrectly handing a start time of 3:36 AM UTC and starting to pull observations at 3:45 AM UTC. You can test by setting time_start to 4/18/18 3:30 AM UTC and you will see that you will correctly get observations back starting at 3:30 AM UTC.
Thanks for getting back @corrineb and updating me on the issue. I will try and work around the bug in the meantime, but hopefully you’ll be able to get it fixed soon
I am having a similar issue… when I query with time_start of 1528730520 (Monday, June 11, 2018 10:22:00 AM GMT-05:00 DST) and time_end of 1528730880 (Monday, June 11, 2018 10:28:00 AM GMT-05:00 DST) , I get NO results. However, if I query with a ‘day_offset’ of 0 (today), I get data…
If I use a time_start of 1528700040 (Monday, June 11, 2018 1:54:00 AM GMT-05:00 DST) and that end_time… I get data, but starting at 1528700400 (Monday, June 11, 2018 2:00:00 AM GMT-05:00 DST).
There is a big bug in the API when it comes to specifying start and end times. It seems like the day offsets of 0 (today) and -1 (yesterday) work, but nothing else.
@corrineb mentioned earlier in the thread that the API was incorrectly starting to pull observations at 3:45 am UTC. I’m not sure if you can use this to create a workaround. I have just given up trying to use this functionality in the API for the time being, and am waiting for the WF team to issue an update to fix the bug. It means, for example, that I can’t calculate a numeric value for the 3 hour pressure trend, or the total rainfall for the year.
Unfortunately, I don’t imagine that fixing this is particularly high on their to-do list!
Does ‘todays’ data end with the most recent observation, or does it end earlier? It would help to check how quickly the 60 second data is made available to API when using data ranges.
I guess that makes sense. A usual API call gets you THIS minute, whilst using day_offset=0 gets you all data from the beginning of the day to LAST minute.
