Wednesday, May 22, 2013 19:20 GMT
There is a lot of work to do on the new recon system. The deeper I get into it, and oddly enough the more I get done, the more I realize I have to do. (Sometimes coming up with new ideas for new products.) The first draft of the new system will have more features than I thought, and therefore will take longer to complete. I now believe the first draft will not be ready until late 2013, with the system being completed in early to mid 2014. I am pretty much rewriting most of it and it makes sense to rewrite it now how I want it to be in the end rather than waiting on doing some of the new features. I am fairly confident that the live Google Maps system will not be ready until 2014. While I already developed some of it, I have yet to determine how it will access data in real time in an efficient manner. I need to think about that a lot, a whole lot, and while there are probably good methods to do it, more and more on how best to do it will be available the later I wait. However, some of what I already developed could be used in the new Google Maps mapping for individual observations. That simply displays the data, with no live component, and that part I already worked out way back in late 2011. Yes, 2011. I just finally rediscovered the work again today that I last worked on on January 6th, 2012. Each individual decoded ob needs its own mapping component in the new system. The old mapping was awful and I am not using that. So I am left with doing the new system without web based mapping for each ob until the next draft of the system, which will have live Google Maps mapping I hope, or to go ahead and add the feature I figured out in late 2011 now which will give me a good idea of how the live Google Maps mapping will work, aside from the live component. Of course that adds a lot more I have to figure out before the first draft is done. I am going to rewrite the URNT15 decoder and work out the new mapping. I also have a new feature idea for the HDOB output and I think I will go ahead and do that now too since when I am rewriting the decoder I might as well write it how it will need to be to work for that idea too, if it works out. So again, lots to do. Another something I want in the first draft of the recon system will be a system to gather data from the NHC recon archive every so often as a backup of checking the same individual product files constantly which leads to missed obs. That might be more difficult than I think. I know the obvious way, get the entire directory listing and see what is new since the last directory listing I downloaded. However, then how do I know what files I already have and which I do not? It would mean having to get all the files since by the file name I do not know what they are. Then, I would need to either just go ahead and process all that data, or determine which data is already in the system and which is not and therefore needs to be added. Hopefully there is some other way. I haven't got into that much yet, but it is again something I want in the first draft of the system. Additionally, I would like NOAA AXBT data in Google Earth in the first draft as well. That requires additional steps on how to associate that with a specific mission since the precise mission information is not in the actual product. Basically, the second draft of the recon system would have live Google Maps mapping and perhaps GlobalHawk track data, both assuming I can work it out. The GlobalHawk track data is a little tricky because it is not easily available and involves downloading large files which would mean downloading infrequently, like every half hour perhaps. That data however could disappear from the web at any time or change formats, so that system is the lowest priority. The dropsonde decoder in the new system will be able to process sonde data in real time from the GlobalHawk UAV that comes across transmitted under UZNT13, so that is the most important data anyway. The actual track data and observations at every moment of the flight is less important way up that high compared to the sonde data.
Finally, I wanted to also say that the new recon system, like the model system, will be made available for free to weather professionals, media organizations, educational institutions and very popular websites. I do not know when the new recon system might be available for download, such as by the time of the first draft or final version.
And now for some temporary sneak peaks of the new recon system that I will remove eventually. Some images have features that are incomplete and parts just don't work yet and have made up information: 1 | 2 | 3 | 4 | 5 | 6 | 7 (Draft of Google Maps recon from 1/6/2012 version I will pick up again. Has made up data.)
Yesterday best track and model data was updated on our site for older storms in the Atlantic, East Pacific and Central Pacific. All data prior to 1990 was recreated along with all storm data, other than invests, for 2012. Select storms from the remaining years were also updated based upon best track data that changed since we last recreated the data. You can read about some of the actualy changes here. We updated all the older information rather than go through the old years and find what specifically needed to be updated.
In addition to working on the new recon system this month, some links to the sites we link to under Analysis & Forecasts will be updated. We did that some on the "Model Data" page, though the work is incomplete there and has yet to be done on other pages.
A lot of work is being done offline on the web based interface of the recon archive. Some major changes are being made to how the data is stored. The new directory structure will require rearranging tens of thousands of files. (Update: I created the system to do that and shockingly it reorganized everything in the Atlantic and East & Central Pacific in under 4 minutes.) This will be done offline for now and when the recon system is ready later this year to be put online, the current system that is online will be completely replaced. Every part of the new system will involve data being located in a different location that in the old system. The entire web based interface of the system will change.
This particular organizational change is being done so that data can be managed more efficiently in the future. Products will be organized into mission folders rather than having mission folders organized into product folders. The change could potentially allow for various other files to be created regarding the mission that could be added into a mission folder where all the other product data would then be located for that mission. I'm not sure what will come of this move, but it will be more beneficial to do it the new way. However, it means a lot of work rewriting a lot of things and writing a system to move all the years of recon data to the new structure, but with the massive rewrite underway, now is the time to do it. When I first wrote the recon system, I organized data by product and then had mission data in each of those folders. Mission "01" would exist in the "URNT15" folder and then also in the UZNT13: folder for example. For navigational purposes based on how the web based interface works, that made some sense at the time. Even then though, the Google Earth maps were always put in the URNT15 folder (or whatever HDOB product type when it comes to older messages) because that is how things started out on our site. Eventually all the other products were in the Google Earth file and it was too late to rewrite the system to organize data differently without major changes. So now is the time to make a change. A Google Earth file can now be put in the main mission folder rather than putting it in an HDOB folder.
I have no idea when the new system will be moved online. I plan to move it online before a lot of new features are added, but with all the work to do before I can even think about new features, it will likely still be in the middle of the hurricane season before it is ready. The system will be very different internally than it used to be so it will need a lot of testing. The Google Earth files will still look the same and there really will not be any new features at first, it is just that the system will be rewritten with the ability to add to it in the future. Recon data in Google Maps will not be available at the start. That will be the main new feature that will come later. (I actually created how the wind barbs would work in Google Maps, but I have yet to decide how all the data would update in real time efficiently in Google Maps. There is a lot of thinking to do about that.) The only new feature at the start would probably be dropsonde data from NASA which was being rejected by the old system. (I'm talking about just dropsonde data from the Global Hawk unmanned aerial vehicle that come in under UZNT13, not track data. Track data is something that may come later in the year or next year.) Some other dropsonde data was also not getting processed that the new system will process. Non-tropical RECCO data will also be available in the new system, probably from the start, though it will take a long time to add all the historic data. I'm not sure when NOAA AXBT data will be in the Google Earth file. That may take some time and may not be available at the start. That system will run differently than the main part of the recon system as it will not be organized by storm. I will attempt to write something that tries to associate that data with a storm if possible, but I have yet to do that. That is where a file in the mission folder would help. I could come up with a system that would attempt to associate AXBT data elsewhere on the site with a mission for a storm. If that worked, I could add that to a file and then the web based interface of the system would know to have that data available for that storm on its storm page.
Upon thinking about it more, there will be some minor things here and there probably at the start. The last location reported by the aircraft will be on the front page in a Google Map. The front page will also be designed better when it comes to showing what recon is current. There will also be dedicated pages for aircraft where you can see the observations from that aircraft only. That is helpful if there are two flights going from the same agency at the same time. You can pick which flight you want to track and focus on those obs rather than keep checking to see which obs are from what mission. A lot of that I have yet to work out, but that is a core part of the system so I expect it to be released when the new system is ready. Additional improvements would likely be made to that in the future. One very important thing I have to do is add information to a file that will have all active recon in it. I have yet to do that. That information will be used on the site in various places. That is something I need to think about more. Current recon information on the top of every page perhaps?
I have done a lot of work on the web based interface in the past few weeks. Again, all offline. It is really taking shape. In fact it kind of is how it will look at this point, but there is still a lot of work to do to make everything work. Especially since now I am changing the directory structure of the data. It looks very much like the model system's layout. I am trying to make the user interface more friendly when it comes to the page where you access data. The page will no longer use actual frames, though it will appear as though there is. I am adding a CSS menu, like the main menu at the top of our site, that will give you other options, like viewing other data for the mission and accessing other main features of the recon system. Putting it in a menu gives it a little more room and makes things less cluttered.
Lots of work has been done and a lot of work yet to do! If anyone has any suggestions, let me know! It could be something simple. Questions like: Wouldn't it be cool of this did this or wouldn't it be nicer if this thing here went over there instead? That is the kind of thing I am doing in the web based interface, trying to imagine how people will use it and try to make everything easier to navigate through. Suggestions don't have to be limited to something simple. One thing I plan on probably doing is having a file created after each vortex message comes through which will have every vortex message in the storm in one file over the life of the storm. Then you can have them all in one file to open. I like to do that manually by opening all the previous missions or saving them in Google Earth as they happen so I can see the previous storm track based on vortex messages. An easy file to open would be much better though. That is actually something that might just be released when I actually do release the system online.We'll see. Again, a lot to do!
Work continues offline on the web based interface of the new recon system.
April 12th, 2013
Work continues on the new recon system. I am currently working on the web based interface of the archive and how that will work in the new system. All the work continues to be done offline. While I hope to have the framework of the new recon system done early in the hurricane season, any new features will likely not be available until late this year.
Have a suggestion for our site's recon system? Some important decisions on how it will work in the future are continuously being made. If you have any suggestions please let us know.
I was originally going to post an update on what part of the new recon system I am working on, but I keep working on so many parts, I'm simply posting an update saying I am working on the recon system. The new recon system will not be released until the middle of the year at the earliest. I know some dropsonde observations cannot currently be decoded at times even though they are valid. The dropsonde decoder I rewrote at the end of last year does work for them, but that will not be online until the rest of the system. The code of the system has changed too much and that decoder needs to be tested more before release. At the moment, I am working on the framework of how the new system will work and simultaneously working on all aspects of the live system. There is a massive amount of work to do over the next many months as I basically rewrite all the code and slowly add new features. By request, I will recreate some sections of historic recon data if needed. However, I would rather recreate all the data when I have more of the new system done. I am at times working on that system as well which recreates all the data on demand. However, while rewriting the system everything is broken in my offline version. I am slowly trying to rewrite everything that would allow me to add and also recreate data. However, that part uses most of the code of the live system so I am working on a lot of things.
Someone asked me about winter missions. As a reminder, our site carries winter missions as well. I have increased the updating interval for the live systems in the Atlantic and the East & Central Pacific. The new recon system should handle these types of missions better when organizing data. That is one of many things I am working on.
I am working on the recon system full time, so even though I don't post any updates, I am working on it all the time.
I have added back the East and Central Pacific recon system here for winter missions. I have resumed work on the recon system after a long break. I am working on restoring older Google Earth data. I am working on updating the mapping system that will be used to process all that old data. Older decoded data will continue to remain unavailable after that point while the new web based interface of the recon system is developed.
January 1, 2013I have been reorganizing my computer and taking a break for the holidays. Older Google Earth recon data will be recreated in the next few months and later in the year a new recon system will be launched.
Older Site Updates...Travel back in time all the way to 2007 to see some of our site updates.
