Anyone used the new BOM XML data before?

Discussion in 'Programming & Software Development' started by Azrael, Feb 3, 2019.

  1. Azrael

    Azrael Member

    Joined:
    Jun 27, 2001
    Messages:
    8,621
    Location:
    Melbourne
    Well after the seeming debacle that the BOM RESTful interface was, and now that the Weather Underground API has closed, I need to find a new source of data for my magic mirror module.

    I tested using the Dark Sky data, but its so severely out of wack that its not workable. Was still returning values of 37C when it was 27C on the BOM data and 28C on the WUnderground.

    Ages ago I used to use the FTP service that they offered to scrape the XML data, but this was through a python script, and it seems that the data format has changed. What makes it worse is that the GRIDS linking (https://developer.bom.gov.au/catalogue/gridded-forecast/) now requires an API key which there is no way to register for, as the developer portal has now been closed too.

    What is worse is that the majority of documentation for the BOM data is now linked to the defunct developer portal, and so is lost in limbo.

    I notice that a bunch of apps (such as Pocket Weather AU) are evidently still getting data from the BOM, and likely even GRIDS data given that the current weather changes per suburb rather than just per weather station.

    Anyone played with the new BOM system enough to know how it works these days?
     
  2. RnR

    RnR Member

    Joined:
    Oct 9, 2002
    Messages:
    12,318
    Location:
    Brisbane
    I once wrote a node.js script that would process and store BOM data from their ID*.tgz files found at ftp://ftp.bom.gov.au/anon/gen/fwo/. There is a json(and xml) file for every station inside the .tgz's containing 72hours worth of data points. However you are then dl'ing a fair bit of data for each 10min update that I think BOM generates their outputs.

    Without subbing to their service ie http://reg.bom.gov.au/other/charges.shtml thats about the best you can do I think.
     
    Last edited: Feb 4, 2019
  3. OP
    OP
    Azrael

    Azrael Member

    Joined:
    Jun 27, 2001
    Messages:
    8,621
    Location:
    Melbourne
    Yeah, thats what i was planning on doing, but as you say it is a fairly serious amount of data overhead.
     
  4. neRok

    neRok Member

    Joined:
    Aug 19, 2006
    Messages:
    2,838
    Location:
    Perth NOR
    I'm not 100% sure on this, but you should be able to effectively stream the file, and then only obtain the portion you need. So for example, if the new data is at the start of the file, you can just download and process in small chunks until you get as much as you need (python instructions). If the data is at the end, I imagine the server would need to have the ability to send data from a specific point, and I imagine you would have to include that start-offset in a header or something.
     
  5. RnR

    RnR Member

    Joined:
    Oct 9, 2002
    Messages:
    12,318
    Location:
    Brisbane
    The data in the .tgz is not organised as nicely as that. Its individual .json/.xml files for each station that gets updated.
     
  6. OP
    OP
    Azrael

    Azrael Member

    Joined:
    Jun 27, 2001
    Messages:
    8,621
    Location:
    Melbourne
  7. RnR

    RnR Member

    Joined:
    Oct 9, 2002
    Messages:
    12,318
    Location:
    Brisbane
    The 95936 is the wmo code - international I believe - so unlikely to change.
     
  8. OP
    OP
    Azrael

    Azrael Member

    Joined:
    Jun 27, 2001
    Messages:
    8,621
    Location:
    Melbourne
    Cool. So its only every half hour, but that means i can pull the data from a native JSON source with only needing a station code.
     
  9. RnR

    RnR Member

    Joined:
    Oct 9, 2002
    Messages:
    12,318
    Location:
    Brisbane
    The V60901 part is the state based product code from memory - just had a quick look at my old source code but nothing there to confirm that. QLD product code is D60910. And I think some states may have multiple product codes covering different stations. But the main point is that these codes are unlikely to change I think, so the json url should be reliable for the long term.
     
  10. OP
    OP
    Azrael

    Azrael Member

    Joined:
    Jun 27, 2001
    Messages:
    8,621
    Location:
    Melbourne
    Yeah, i mucked around with it a bit last night and got it pulling some data nicely. Slightly annoying that ill have to pull one JSON for the forecast, and another for the observations. But nothing insurmountable.
     
  11. Renza

    Renza Member

    Joined:
    Dec 1, 2004
    Messages:
    4,661
    Location:
    Melbourne

Share This Page