Free Screen Scraper App for Bureau of Meteorology (BOM)

Discussion in 'Programming & Software Development' started by FearTec, Oct 15, 2010.

  1. FearTec

    FearTec Member

    Joined:
    Jul 22, 2003
    Messages:
    2,389
    Location:
    NSW
    Hello

    I am working on a free screen scraper app for Bureau of Meteorology (BOM) that others may be interested in. I am looking for alpha/beta testers.

    Application Images (so far):
    Main screen that shows a summary of your locations weather.
    [​IMG]

    Option screen to load known locations from BOM website. You can save your default location.
    [​IMG]

    Includes interactive Graphs.
    [​IMG]

    Quick Access tabs to load the latest graphics from BOM (e.g Rain).
    [​IMG]

    Also quick links to lesser known but powerful temp forecasting page at BOM.
    [​IMG]


    Also has quick access tabs to the Radar, Satellite, Interactive forecasts, 24hr rainfall, River conditions, Rainfall totals, rain outlook, wind, temp outlook, uv forecast, frost forecast, evaporation, solar exposure, vegitation, sea surface temps.

    Also displays your locations temp in the system tray.

    Future Features:
    • See Bugs/Issues list below.
    • Save past images (so you can rewind and playback loops locally).
    • Analyse radar image (pixel read) and alert of storms, heavy rain etc.
    • Speech API.

    Download the latest Alpha version here: http://www.fearby.com/utils/myweather/MyWeather-Alpha-LatestBuild.rar (168kb, updated 2:50 AM 17/10/2010).
    Built with VB.NET 2008 - Framework 3.5

    Next Version:
    • Auto refresh
    • More bugfixes (see below)

    Current Bugs/Issues:
    • It's a bit cold in melbourne at the moment. Currently showing -9999c (half fixed)
    • The radar image is showing gunnedah, the 24hr rainfall is showing NSW, as is river conditions.
    • seems like my radar view is stuck on Namoi - any chance you could make it choose the nearest radar? (for me, that would be Terry Hills)
    • Not working for me at work. I believe it to be the lack of an option to input a proxy address (http)
    • Rain to 9am should probably be a Bargraph
    • Need to disable appropriate graphs when BOM feed reports -9999 (e.g melbourne)

    Feature Request:
    • Radar other than Northern NSW.
    • Doppler wind.
    • Predict storm cell path

    Change Log:
    • Release v0.1.3 - 1:20 AM 21/10/2010

      [*]Forecast locations (states) now sortable by alphabetical order (click column header).
      [*]Links in copyright screen now work
      [*]If observation data is not available, Feels Like, Wind Speed, Wind Gusts do not display -9999.*
      [*]Added ZedGraph Chart library License details to copyright tab and zip file.
      [*]Ability to right click copy to clipboard/navigate on many listviews.
      [*]fixed Vegetation typo
    • Release v0.1.2 - 2:50 AM 17/10/2010

      [*]Find Forecast Location: State Locations other than NSW have an " in the URL. State Locations other than NSW have an " in the URL. - BOM Format states A HREF's differently. Tested All States.
      [*]Fixed Satellite typo.
      [*]Now compatible with smaller resolutions. (e.g 1280x1024 and smaller).
      [*]Added build number to title bar
      [*]Smaller UI.
    • Initial Release v0.1.1.0 - 11:31 PM 15/10/2010

    Notes
    Some smaller BOM observation sites do not have history or all observation values, this may cause history graphs to not display in full.

    Beta Testers:
    • TheWedgie
     
    Last edited: Oct 21, 2010
  2. TheChemist

    TheChemist R.I.P

    Joined:
    Apr 5, 2007
    Messages:
    1,035
    Your entire "Find forecast location" thing is broken. You obviously have not tested it for any state other than NSW.

    Also, your forms are all fixed width/height and cut off elements if not using the same res you are.

    Additionally, you spell "Satellite" wrong everywhere.

    And why do you use embedded browser windows to do simple things like display a picture? Why use browser controls at all, for that matter? Why not parse the data and display it in your own way?
     
    Last edited: Oct 16, 2010
  3. mfearby

    mfearby Member

    Joined:
    Jan 18, 2004
    Messages:
    125
    Location:
    Tamworth
    You seem to be unfamiliar with software version numbering. A version number of "0.1.1" means that the things you are pointing out are to be expected in an early testing release.
     
    Last edited: Oct 16, 2010
  4. OP
    OP
    FearTec

    FearTec Member

    Joined:
    Jul 22, 2003
    Messages:
    2,389
    Location:
    NSW
    Update:

    Reported Bugs (Thanks).
    - Find Forecast Location broken for states other than NSW.
    - Needed 1920x1200 resolution.

    Will fix/add in next version:
    - Find Forecast Location: State Locations other than NSW have an " in the URL. State Locations other than NSW have an " in the URL. - BOM Format states A HREF's differently. Tested All States.
    - Fixed Satellite typo.
    - Now compatible with smaller resolutions. (e.g 1280x1024 and smaller).
    - Added build number to title bar
    - Smaller UI.

    Notes:
    Will replace the web browser controls and load graphics from local cache when local cache stuf is added.

    Alpha version 0.1.2.0 due soon.
     
  5. TheChemist

    TheChemist R.I.P

    Joined:
    Apr 5, 2007
    Messages:
    1,035
    Few suggestions:

    With your URL parsing, you should maybe look into using Regex parsing instead of how you're doing it now (not sure how that is, but probably less than ideal if you have quotes stuck in there).

    With the resolution thing, this could easily be solved for almost any resolution by correctly using docking with panels.

    With tabs that only display a single image, *don't* stick them in a browser window. Download the image and put them in a PictureBox!
     
  6. OP
    OP
    FearTec

    FearTec Member

    Joined:
    Jul 22, 2003
    Messages:
    2,389
    Location:
    NSW
  7. TheWedgie

    TheWedgie Insert Custom Title Here

    Joined:
    Jun 16, 2002
    Messages:
    2,869
    I'm in. This is pretty damn close to what I've been trying to write for work.

    Might send you a PM later re some customisations :)

    EDIT:
    - "Vegitation" tab should be "Vegetation" :D
    - If I change my location to Adelaide, it doesn't change the 24hr rainfall / river tabs to SA.
     
    Last edited: Oct 19, 2010
  8. flightcrank

    flightcrank Member

    Joined:
    Aug 6, 2004
    Messages:
    744
    looks like a nice application withh allot of work put into it, and allot more work still to do. very nice job. a quick question tho.

    why use this app ? why not go right to the web site. ?

    what happens when they change they layout of the website - you have to re - write your app all over again.

    also this app only works on windows. what about mac/linux users. symbian/iphone users ? i guess they can probably just go to the website to get the info.
     
  9. OP
    OP
    FearTec

    FearTec Member

    Joined:
    Jul 22, 2003
    Messages:
    2,389
    Location:
    NSW
    Will add those feature soon.
     
  10. OP
    OP
    FearTec

    FearTec Member

    Joined:
    Jul 22, 2003
    Messages:
    2,389
    Location:
    NSW
    I want an app that gives a quick summary of the weather without the digging on the BOM website.

    I plan to add speech to the application, popups of preset or significant changes in the weather, and local analysis of radar images with predictions of the likelihood of nearby rain/storm of hitting a preset x/y coord.

    Archiving of BOM images will allow for quick replay via the mouse scroll wheel.

    I will move major source BOM urls and screen scraping configuration to the web and allow sync of screen scraping knowhow instead of a new application.

    Keep the requests coming.
     
  11. OP
    OP
    FearTec

    FearTec Member

    Joined:
    Jul 22, 2003
    Messages:
    2,389
    Location:
    NSW
    Will look into Mac and iDevices after I get a Mac.

    Will have no problems releasing source code so others can copy if need be.
     
  12. TheWedgie

    TheWedgie Insert Custom Title Here

    Joined:
    Jun 16, 2002
    Messages:
    2,869
    Have you considered something like Trac for tracking bugs and feature requests?

    I'd love to see how you're doing the scraping - I tried a while ago to write a similar app but ran out of ideas.
     
  13. flightcrank

    flightcrank Member

    Joined:
    Aug 6, 2004
    Messages:
    744
    your using winforms which uses the winapi :S
     
  14. Tekin

    Tekin Member

    Joined:
    Nov 16, 2002
    Messages:
    4,039
    Location:
    Elsewhere.
    It's a bit cold in melbourne at the moment.

    Currently showing -9999c

    Fortuantely I think that's counteracted by the negative windchill of -9999km/h.

    Heh, anyways, so I don't think your scrapers are working correctly.

    Also:

    The radar image is showing gunnedah, the 24hr rainfall is showing NSW, as is river conditions.

    Otherwise, I like it, keep it up!
     
  15. TheChemist

    TheChemist R.I.P

    Joined:
    Apr 5, 2007
    Messages:
    1,035
    Could still be ported to Silverlight which would then run in a browser for Linux / OSX while still being a Desktop app on Windows :)
     
  16. lui_gough

    lui_gough Member

    Joined:
    Sep 18, 2007
    Messages:
    132
    Location:
    Chester Hill, NSW, AU
    Nice work! Just a question - seems like my radar view is stuck on Namoi - any chance you could make it choose the nearest radar? (for me, that would be Terry Hills) and make it look like the BOM's 5-frame radar loop?
     
  17. Volupheles

    Volupheles Member

    Joined:
    Oct 24, 2007
    Messages:
    30
    Location:
    Melbourne
    Not working for me at work. I believe it to be the lack of an option to input a proxy address, as we use a http proxy here.
     
  18. Eater

    Eater Member

    Joined:
    May 31, 2006
    Messages:
    6
    Location:
    Darwin
    Rippa app

    What a great application
    Has made my browsing of weather far easier ;)

    Keep up the good work, look forward to selectable locations
     
  19. JayKay777

    JayKay777 Member

    Joined:
    Sep 4, 2008
    Messages:
    236
    Location:
    South Melbourne
    Or use what everyone else has been using for this scenario for the last 10 years, Java :)

    But looks to be a hobbyist that has learnt Visual Studio express C# - which is a good way to get in WinForms. From what I understand, C# -> Silverlight is not a simple port.

    I don't like IE browsers embedded in WinForms. For one it assumes (a new secure versions of) IE is isntalled and is working OK. Hopefully you can get all the info from cached images/data.

    Also, the ZedGraph has an LGPL license - I think you need to attach this in your copyright -

    "1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and distribute a copy of this License along with the Library. "

    Although I would have ago at writing your own graph class - not hard with System.Drawing to a bitmap/graphics.

    Rain to 9am should probably be a Bargraph, with Bars the width of the time period they measure over (9am-6pm and 6pm-9am i think??).

    One last thing - what .NET version are you compiling on? You should disclose this, so old OSs can install it.
     
  20. Flamin Joe

    Flamin Joe Member

    Joined:
    Jun 28, 2001
    Messages:
    4,130
    Location:
    4300
    Great stuff. Just one little thing, when selecting the forecast location, there not in Alphabetical order. As a result, it took me a while to find Brisbane.

    Also just a tip to broaden your user base. Amatuer Astronomy is one hobby which is very heavily dependant on weather conditions. I know a lot of Amatuer Astronomers, like myself, use a variety of online sources (such as BOM etc) to get accurate forecasts to plan viewing/imaging. Something like this having all the information in one would be fantastic, especially as I know quite a few fellow Astronomers use Computers to control their equipment so this software would be a great addition to what they currently use. I'd imagine there would also be quite a few suggestions as to what they'd like to see on it too.

    So with that in mind, I would encourage you to post this over at IceInSpace. It's one of the largest Australian Amatuer Astronomy sites and I'm sure they'll appreciate it. :thumbup:

    http://www.iceinspace.com.au/forum/index.php
     

Share This Page