Wednesday, May 23, 2012 12:22 GMT
These are old site updates. Some links and images are no longer available. Additionally, some information is no longer applicable.
For the latest site updates, click here.
It's that time of year again. The site has gone into sleep mode for the next 6 months. Some systems are operating on a reduced schedule for the off season. The recon system is updating slower and only checking HDOB and dropsonde messages. Some other changes have also been made. The old model system has finally been deactivated. After five years, the simple, yet demanding, system has been retired. The new model system, which was in beta testing, is now our official model system. It has performed very well this season and therefore I decided to retire the old system. There is only one behind the scenes error with our model system. On some occassions it leaves behind some old data when a storm is no longer active. (not in the public section, the server side) I still have not pinpointed that error. If I can't find it, I'll have to write a utility to run sometimes to delete the little bit of old data. Our model system is currently running every half hour and that will continue at least until 90L is finished. I will likely then slow it to updating every hour.
Work continues on other areas of this site. A redesign of certain areas, specifically the recon system, will continue throughout the off season. I'm not sure what final changes will be made yet as the current level of development is still in the early stages, though early testing is going well. The older recon data in the recon archive has still not undergone any review. That will likely occur sometime during the offseason. The changes I will likely make to the recon system will likely require reprocessing all of the data. I want to wait until I have made the changes I plan to make. There is still a lot of work to do and until I get even deeper into the testing, I'm not sure what will come of it all.
For nearly two months I have been adding historical reconnaissance data to the recon archive. The data has been added but the data needs to be reviewed still. The data will always contain errors as this is raw data after all, but I will try to reduce the amount of errors in the recon archive where the highest wind data for a mission is listed. This will take time. I am also going to be redesigning some things in the recon archive and eventually around the site.
Some mission data has not been added. Most historical NOAA missions are not available because the raw data from the missions are not available in a World Meteorological Organization (WMO) format. For example, one file will contain observations every second from the plane, or every 30 seconds perhaps, but our site does not decode these files. It would be a lot of work to do so and it would never work perfectly because there would not be other data, like vortex messages. I could convert dropsonde files to a WMO format but I would rather not get involved with creating files that don't currently exist.
In order to add some data from 1989 to 1991, I had to add another decoder. The recon decoder now decodes URNT50 files. The decoder still doesn't decode the older supplementary vortex format. (so those messages will not appear in the archive) I'm not sure if I will ever make a decoder for that.
I decided to start making some various changes around the site. I'm not sure how far I will take the redesign yet, but you may notice that the look of some things might change.
One change I have made is to make the new model system in beta testing the primary model system. The old system is still available, but is less prominent now. You can still access the old system here.
Historical statistics are now available in the best track and model system in beta testing.
All of the data has been reprocessed and added back.
July 23rd, 2011 - Update 2...All of the data has been reprocessed and the upload will begin overnight tonight and possibly into the overnight hours of Sunday/Monday. I added back 2011's data.
July 23rd, 2011 - Update 1...
Some small changes and small errors are requiring me to recreate all the data in the best track and model archive in beta testing. I expect that this will be complete by the morning of the 25th. Rather than try and fix the errors I decided to simply reprocess all the data.
Work on the origin feature and files with every storm included continue. That might be delayed about two weeks though while I work on some other projects.
The live best track system in beta testing had a lot of issues on the evening of the 17th. A whole lot. At first I couldn't figure it out. After awhile I realized that for some unknown reason the file in the tcweb directory of the ATCF system for Bret was disappearing. That was something our system was not prepared for. 98L kept coming back and when the file was not there for 02L it went away. I think I have now fixed that problem. The system should now bypass checking for data on a storm that is active but is not in the tcweb directory listing. The storm will be frozen in time. While the system can't find the best track file it will not download model data. As for the invest number coming back I have something in place now to stop accessing it when this circumstance occurs. It all worked in testing.
Another issue is that when 98L was upgraded to 02L, the development was still noted as a disturbance. That was what was still listed. From now on however, when something has a name of a depression but has not yet had any development that is not a depression, storm, or hurricane, I will call it a depression. That should take care of that issue.
And now back to what I was creating previously.
I have worked further on creating Google Earth files for all storm tracks. It is still not online yet. I have created versions for everything depression or higher, tropical storm+, cat 1+, cat 2+, cat3+, cat4+ and cat 5. Now I have started working on Google Earth files for origin points and then also a file with statistics on storms in history that can be accessed later for various purposes that can be anything from a table to sort through the most powerful storms or table of the formation date of the first (second, etc) storm of the season and even different charts, such as named storms per year. Once I get the origin points done I may go ahead and put the files that were created online. A single script creates all this data on demand, not in real time. I have yet to make a final decision on when it will run. I might actually have it run when a depression or higher storm ends and have it run just for that basin. I might also make it available to be run by an adminstrator or even the general public every so often. I also forgot to mention that my site's distance calculator has been integrated into the new best track and model system. It allows someone to see how far a storm is away from them.
July 14th, 2011...
I have continued the code cleanup in data.cgi. I have not done anything in the main script which generates the data by checking the ATCF system yet. I instead decided to test some new things. The image below is a screenshot from Google Earth of offline testing where I create a file with every depression or higher in each basin. I am continuing that testing offline and perhaps in a few weeks I will have it online. I plan on adding one other thing too. I want to make a file that shows just the origin points of storms. I will probably have several versions of that, like one for including depressions and one for tropical storms. (could maybe do one for just hurricanes and maybe for each category too) I still need to think about it. I also might do storm track version files that also feature just a certain type of storm. I also think I will create yearly track file as well for each basin.
I was surprised by the low file size of the Google Earth files that you see in the screenshot below. 570KB kmz file for Atlantic, 275KB for East Pacific and 20KB for Central Pacific. That's KMZ files, which are zipped KML files. The KML files were 15MB, 7.5MB and 500KB respectively.
These new features in development will probably work differently than other things on the site which will take some thinking on my part. These files take a lot of resources to create. That means they will not be created constantly. For example, for a current storm, it will probably not appear in the file. I may have a utility that runs every week and does storms that are no longer active. Or I might have it so someone could activate it on demand, but only allow it to run every week. The file with every storm in history might be limited to being created once a month. Still need to think about it all.
I have split data.cgi into a lot of different scripts so it is now much more efficient than it was. (From always loading 350KB+ to now 30KB plus the script it actually needs to load.) There is no visible change to end users other than perhaps noticing the web based archive loads quicker. I still have a lot of code cleanup to do though and I will continue to make it work better. I also really need to cleanup the main script which generates the data by checking the ATCF system. That script is about 300KB. That is rather large because it has to be, there is a lot going on. However, at this point it has a lot of code that is needed to generate archived data. I think I will work on splitting everything I can that does that out of the script so that it is only loaded when someone generates archived data. That would make the script work a lot better.
July 11th, 2011 - Update 4...It's done. Over a quarter of a million files and about 3 gigabytes of data. At this point I will begin code cleanup and splitting the data.cgi script to run better. Then I can look into ideas on what I can do further with the data now that I have all of it available. If you see any problems in the archive let me know. There is a massive amount of data so I am unable to go through it.
July 11th, 2011 - Update 3...Today I let my computer process the rest of the data and I have begun uploading the last of the data. Based on the rate of upload, it should be done late tonight if things continue to go well.
July 11th, 2011 - Update 2...Things have gone very well this morning. I have uploaded everything except 2003 through 2009. Recent years take a very long time to process and upload. Tonight I will start on the rest. I should have the rest done likely by Tuesday night if things continue to go well.
July 11th, 2011 - Update 1...After considering one other feature I decided not to add I have again started processing older data. Everything processed so far, through 1980, went well. The script generated the proper notes for things that were different from the storm table file in the ATCF database. If things go well, and I could always find something that needs to be corrected, all the data could be added by Tuesday evening. Don't count on it, but we'll see. There are hundreds of thousands of files to be created and uploaded.
July 10th, 2011
I have updated all the scripts online and made Calvin's older data unavailable. I had to do that to update the script.
Unfortunately I came up with one more thing I needed in one of the behind the scenes files. That meant everything I did today, trying to upload all data from 1851 - present, was just rendered useless. All the data uploaded was deleted. I'm not going to set a timetable for readding everything. However, based upon what I did today I see that it would take about two days to process and upload all the data. Before that however, I need to add the one thing I forgot. Additionally, before I process the data again I am going to make sure all the data is ready before I upload. There were about a few dozen subtropical storms and a few other storms that need some slight adjustments in the script before I process the data. I don't want to have to interupt the process again so I am going to really make sure I am ready next time.
Offline testing has continued to go very well. The last minor new feature I planned to add has been added. None of the recent changes are online yet. I am going to try to wait until Calvin weakens. I'll probably go ahead and update the system even with it active and just hide the older model data that I recreate for it. I made a lot of changes to the back end of the system and I will have to recreate all the old data again. This might be all the changes I make to the back end of the system. Tonight I am going to start further indepth testing of older data. I have done a lot of testing so far and it seems to have went well. I may start adding older archived data as soon as early next week. (Combined, over 2600 storms in the Atlantic, East Pacific, and Central Pacific, not including invests) There is one thing I also want to do but I don't have to do that now and that is allow invest data to be added from the "store_invest" folder. I haven't done that yet so older invests can't be easily added yet.
I tested offline creating data from 1851 through 1953 this morning and it created successfully. That's easy though, no model data until 1954. Later tonight I'll start testing more and more years with actual model data.
Work continues offline on making sure older data is displayed in the best way possible. It's going well. I keep coming up with slight tweaks for older data though. There is a lot of things to consider because older data has less information available and some of that information I have previously counted on having, like the level of development. I am also working on using the file storm.table as well, in the archive folder of the ATCF system, which has a list of all old storms. I plan on using that to get storm names and the highest level of development when I can't get that from the best track or model file. After that, the one major thing I was thinking about doing is too complex, at least right now, so that leaves a few minor actual additions that I can think of at the moment before the system will be ready to generate older data. Within a few weeks I might have all the archived data available.
July 4th, 2011...This morning I completed the update that should allow data before 1970. I wanted to get the update live to start testing it on the live system as it changed the file structure again of some of the files. The system might be a lot less stable until the bugs get worked out. I tested some data for various old years and things went well. I have removed the old data from the live system I was testing. I wanted to check for any bugs once online. I will continue offline testing. Next I will see what other features I may want to add. After any new features I want to add are added I will then process and add all old data, from 1851 to present. I'm not sure when that might occur. At that point I will probably have a new range of features that I could add given all of the historical data to work with.
July 3rd, 2011...
Work began yesterday evening on handling data prior to 1970. It is going very well. I think I already finished the changes to the main script. Now I have to adapt the web based archive. None of these changes are live yet. The model error feature already worked for all years in which model data was available which was great. A negative unix time is created for older data and it works using that. There is still a a lot of changes though. I am once again changing the file structure and I am changing how time is calculated. (from a unix time stamp back to the original 10 character time code I get from best track and model files.)
I may be ready with the update within a day or two. There are a lot of changes so it will make the system more unstable when I apply the update until the bugs are worked out.
A significant update occurred to the new best track and model system in beta testing today. There are few changes you will notice visibly. The update changed the file structure of things and could bring about some new bugs. I have still not updated the script to allow archived data prior to 1970. That will be an extremely significant update if I do that.
All older data was deleted as it would no longer work with this update. I have however recreated all data for this year. If someone wants older data, let me know and I will still create some of the old data for historic storms. I will eventually add all data from now through 1970. I still have more features I would like to explore before I do all of that though.
Among the updates: Past track in Google Maps. (And options on how to show it.) Extra data in Google Maps and Google Earth for model data, when available. (Not simple wind or sea swath text data though.) Show best track storm position in Google Maps. Change storm icon size in Google Maps. Show actual coordinates for a model point in Google Earth info window. Model data that does not have intensity for the "0" hour should not have intensity available for the other hourly forecast points. (Fixes a problem with a few track models that had completely incorrect intensity values.) Added new model names for 2011. Model names now appear in Google Earth for models that were created in historical mode. Numerous other updates to the file structure behind the scenes. (Eventually, some of the additional text data that appears now in Google Maps and Google Earth may appear on the model text page in the web interface.)
I think I have completed the past track feature I was working on for Google Maps. I might add to it in the future though. I have not yet made the change live though. The change will require that I delete all old data. I will probably do that this evening and I will recreate all of 2011's data and add it. Extra text model data now appears in both Google Maps and Google Earth, when available. However, I still don't add the text wind swath model data in Google Maps or even the main Google Earth file. You still have to go to the separate Google Earth wind swath file, when available, for that. I definitely did not want to include it in the Google Maps version because it added too much to the file size of the XML file that was loaded.
June 30th, 2011...
Our site yesterday switched data sources for gathering the lastest recon because our site was not working well for some reason with the previous data source. Our site was not updating every time it should when the previous data source updated. The system may work better or worse now in terms of how fast it updates. Please use our "Add an Observation" feature if our site is not properly adding data. A bug was fixed two days ago that was preventing that from working properly. The recon system now contacts a NOAA FTP server. We'll see how that goes. I think my site was not always contact the previous NOAA site I was using because it didn't think the page would update that frequently so it saved a cache. The new NOAA server I contact might not always be as easy to contact but seems like it updates slightly faster.
I also made a few small changes to the recon system to accept some West Pacific observations into the decoder when the aircraft is an Astra jet.
Progress on adding some extra text data for models in the best track and model system in beta testing is going well.
I was sick for a few days so I didn't get too much done. I'm back to working on the past best track for Google Maps. I also want to add some extra CARQ data and other extra model data, when available, and I haven't done that part yet. I'll probably hold off on making these updates live. I am making a lot of changes to the system behind the scenes. While most are not features that would be noticeable to end users, the changes would make the older data I have in the system now not work.
June 23rd, 2011...
I plan on making a lot of tweaks to the model and best track system in beta testing. I am not ready to make them live yet however. I will have to clear all the model data in the archive to do it. I am changing some of the file structure on some of the files which will require data to be recreated. Once I decide upon the file structure I might make the changes live even though the small tweaks may not be implemented so that end users can see them yet. The one change of real note will be the addition of the past best track in the Google Maps model plots. I have been meaning to do that for a long time. I also plan on having a little more data available for the center position as well and maybe for the OFCL model as well.
A reminder that our live recon system allows observations to be added into it by submitting the link to the ob in the NHC's web based recon directory. I made an update to it today that has not been tested live yet. You can now add obs from the past 24 hours as long as the storm directory on our site has not been moved and you are not replacing a previous observation. For the East Pacific storm Beatriz our site (Tropical Globe) missed several obs but no one added them. Our site updates only at certain intervals and sometimes misses data. Setting this update interval too frequently puts extra load on our site and NOAA's server which is why our site sometimes misses obs. I don't follow every mission live so someone who does can add the obs for us and get the benefit of having a complete mission map more rapidly.
For the past week and a half I got sidetracked on something else but now I'm returning back to working on the model and best track archive.
Someone asked for 1992, 2004, and 2005 Atlantic data so I added it the archive. I will still recreate all that in the future when I make some further tweaks to the system. (One small new feature you may notice is the timeline feature on each yearly archive page.) The list of things to do keeps growing, so the availability of releasing the older data from now through 1970 may be a month away. (And longer for data earlier than that due to the problem mentioned in the last site update.) Recreating the data now and then again when I finish making the updates is not worth it. That is a massive amount of data to upload. But, if you have some specific data you would like to see, let me know.
I realized that our site recently turned five years old. It doesn't seem like it is that old, but I'm not sure why, because thousands of hours have been spent coding the site since its start. On June 11th, 2006 I bought this domain. At the time the site was one page full of links. (A look back at our one page site on June 13th, 2006. I lost the original logo, I think it had some extra white space, but the one on the page probably appeared a few days later.) About a month later we started offering model plots in Google Maps and Google Earth and later in 2006 had an early version of our recon decoder that only decoded HDOB messages.
You may have noticed that older data in the new best track and model archive, currently in beta testing, has been removed. I added a new line in a file and removed a line in another and as a result of the first change I need to recreate that old data. That is simple to do. However, before I do I plan on making some updates to the system. I have been making a lot of changes to the system in the past week and I probably have a lot more changes to go. The system seems to work good for current storms but I am in the process of getting ready to process all historical model and best track data for release on my site. The best track file format was slightly different a few decades ago so I had to adjust the system for that. That seems to have went well. However, in order to process data from 1850 to 1969 I need to make some significant changes to the system because my script relies a lot on getting the time by counting the number of seconds from 1970. (Unix time stamp) Unfortunately, that means I can't process data earlier than that until I come up with a new way to handle time in my script for data prior to 1970. (Based on the new time line feature I added to the model archive I got some ideas on how I might do that.) I'm also preparing to add a little more model text data for CARQ and OFCL because there is a little bit of extra data like storm speed and direction sometimes that I do not currently grab from the model file. Meanwhile, testing is going well on data from 1970 to now. I keep making some improvements to the system so I decided to hold off a bit longer before releasing all of that historical data publicly in case I add any new features that would require me to reprocess all that data again. (It will probably create a few hundred thousand files for the archive) I might recreate the old data before I do all the possible new features only because some of the new features may take awhile to implement or I might decide against some of them in the end. I do have some interesting ideas that I'll consider over the new few months. -- Update 1: A total of four new lines have been added to one of the files in the system which required another manual update. They new lines will have some extra CARQ and OFCL model data. I have not implemented it yet, I just wanted the system to start creating blank lines so they can be read when the update is implemented. It might be a week or more since I want to make some other updates as well. -- Update 2: Now there are five extra lines I added.
June 2nd, 2011...I think the new best track system is good enough to be upgraded from alpha testing to beta testing. The system has had some bugs that I corrected in the past 24 hours but given how well the system did during the testing in other aspects I have decided to upgrade its development status. There is still one bug where old raw data does not always delete when a storm is no longer active, but that is on the administrative end. I am still making lots of tweaks to the system here and there. I also plan to integrate my site's distance calculator into this system so people can see how far a storm is away from them right inside the new system. A lot of the bugs that I found over the past 24 hours were for generating historical data. Eventually I will likely add all old model data into the system so that there is a complete archive for those who wish to research the model and best track data. I might still need to work out a few kinks in that. The model error feature is now stable enough that I have removed the notice not to use it. It is very important that people understand how that works before ever using it though. I reread the product description page and tried to make some things a little clearer. I know it could still use some work but the model error feature is complex to explain but it is something important that people need to understand before using it which is why the product description is so wordy.
June 1st, 2011...
Our site has now resumed normal operations for the hurricane season... for real this time. I forgot to set the decoders to update for the four main recon products. Everything should now be working properly.
I have yet to check to make sure all the links on the site work. Each season I do that since sites sometimes move. I'll get to that soon.
I have made a product update to how I handle wind direction. Here is the old way I did things:
349°-10°: the N
11°: between N and NNE
12°-32°: the NNE
33°: between the NNE and NE
34°-55°: the NE
56°: between the NE and ENE
57°-77°: the ENE
78°: between the ENE and E
79°-100°: the E
101°: between the E and ESE
102°-122°: the ESE
123°: between the ESE and SE
124°-145°: the SE
146°: between the SE and SSE
147°-167°: the SSE
168°: between the SSE and S
169°-190°: the S
191°: between the S and SSW
192°-212°: the SSW
213°: between the SSW and SW
214°-235°: the SW
236°: between the SW and WSW
237°-257°: the WSW
258°: between the WSW and W
259°-280°: the W
281°: between the W and WNW
282°-302°: the WNW
303°: between the WNW and NW
304°-325°: the NW
326°: between the NW and NNW
327°-347°: the NNW
348°: between the NNW and N
You can find the new way here. The mathematical separations between cardinal directions that end in .75 should be rounded up rather down, which is what I had done previously, when it comes to reporting something that is between the sixteen cardinal directions this site uses. Previously, our site reported NNE/NE when something was 33 degrees. The actual mathematical separation occurs at 33.75 degrees. Our site will now report NNE/NE for the whole number 34 degrees. Because this change is so minor, the recon archive will not be updated for messages that have already been processed. As mentioned previously on this site about how cardinal direction is handled, it is subject to being changed in the future. I still don't know of an official way to handle things which is why it remains subject to change. This change has been made, effective immediately, across all site systems on Tropical Atlantic and Tropical Globe.
Our site has now resumed normal operations for the hurricane season.
The NHC's "tcweb" directory has not been updating for 92L and 90E. Our site has some backup methods in place to attempt to handle this. However, it doesn't work well at the start of the season for the new system in alpha testing. The old system looks at the "aidpublic" folder as its method of determining what is available. That is better in some respects. However, the new system looks at the "tcweb" folder instead as its focus so it can organize everything, even if there is no model data. As a backup, the "btk" folder is checked in some cases. If nothing is active in the "tcweb" folder in 8 hours then the new system will check the "btk" folder if something has been active in the "tcweb" folder or "btk" folder in the past 24 hours. If not, then the system assumes it is a slow period and it skips checking the "btk" folder. This did not work today because it is the offseason and nothing occurred in the last 24 hours. I could check the "btk" folder more often automatically, but I would rather not put even more strain on the NOAA server to do that since it is an error that is unlikely to occur to often when things really matter. Today, I designed something that will now allow me to manually run the system using the "btk" folder very easily and after that it will continue to check the "btk" folder as needed until nothing happens for 24 hours.
At some point the old model system will be turned off once the new system is well tested. Last year I had planned on doing that at the start of this year's season but I am going to see how the new system works as the season progresses. I still consider the new system to be in alpha testing, below beta testing, so it may not be until next season until the old system is turned off. When available, the new system offers more data and more options that make that system much better than the old one.
The Atlantic and East/Central Pacific recon systems remain turned on. If things gets very busy in the Atlantic, the East/Central Pacific recon system may be turned off to save resources.
I don't have any new features planned for this year. I plan on testing the new model system and making sure the model error works as expected. I also need to revisit the help page about model error and improve it. (along with some tweaks here and there that make it more clear that model error data can be misleading at times) I might make little improvements here and there throughout the site, but I don't have anything major planned. I eventually want to do NHC advisory data in Google Earth, and have had that system sitting around for many years now, but I need to do the error cone and that is what always held me up. I have been working on an ecommerce venture at the moment and so NHC advisory data will probably be something that doesn't show up until at least 2012.
I often receive requests to turn on the Pacific recon system. I finally decided to turn on the East and Central Pacific live recon system on a trial basis, at least through some of the winter missions. I also updated the system to match the Atlantic system and West Pacific system. (the West Pacific system will still operate only when I am notified of storms)
I'll have to see how things go with two systems being live at the same time to see if I will continue the live system for the East and Central Pacific.
I made a small update to the model system. It was administrative. It was now something on the public side of things, but it could make it buggy in some way, though I doubt it. It basically just allows me to create historic data a little easier.
One thing I did do was create a lite version of my model script for media organizations and researchers. You can read more about it on the media page. It uses a part of the new alpha system to create just model data for Google Earth. Nothing else. It is quite easy to run the CGI script. It does not however do things automatically. For that reason, I have made it completely public. This means someone cannot use it to put stress on the NOAA server but can for data they want allow them to process it themselves on their own computer. That means if this site is down and they have that script, all they need to do is download the best track and model file from the ATCF system and add it to a folder on their own computer to get the Google Earth model data by simply running the script on their computer. (Anyone can do that on a Windows computer, you just need to download and install Apache and then Perl.)
After five months of working full time on the new model system I have completed the major updates I planned on making this year. For the next 7 or 8 months I will be working on a different project unrelated to this site. Around the start of the 2011 hurricane season I will continue work on the new model system. I will be making any needed updates on this site as needed until then.
The last update I made to the new model system was the new center fix system. I will post more about it once it gets tested some more. It takes center fixes from the "fix" folder in the ATCF system and processes it for display on a Google map or in Google Earth.
Despite not posting any updates in the past month, work has continued on the the new model system and there has also been improvements to the recon system. The recon system now has the option to view mission data within your browser using the Google Earth plugin. While I have also worked on the model system, most all of the work involves something not related to models but to something else in the ATCF system. The new feature is currently being alpha tested offline and will likely not be available for a few more weeks to the public. The new model system itself has been working fairly well, though there are problems at times when an invest is upgraded. Sometimes the invest data sticks around when it should not.
September 7, 2010...
Update: It turns out the entire "tcweb" directory in the ATCF system is not working. It was not just a case of 91L's model data coming out first. This means I need to go redesign things to take care of another error that the NHC has. Only this one I cannot easily bypass. I have no idea how complex the redesign might need to be for the new model system. The old system works at the moment, but parts will eventually fail if the error is not fixed. It's the new system, based around the "tcweb" directory, that has the most issues at the moment. Hopefully the NHC will fix the error soon. I don't even know if I can easily redesign the new system. The old system I am not touching until it starts failing.
Work continues on the new model system. The system is getting somewhat more stable, though problems still seem to exist when a storm is upgraded from an invest to a depression or higher. Sometimes invest data is recreated when it should not be since the area was upgraded. Some updates to the code seem to have caused this new problem. I did make some changes but I am not confident that those changes will fix the problem.
I have been working on a new feature for the model / best track system that I have not finished yet. Maybe in a few weeks I will have the new feature running.
I noticed today that for 91L we have one of those rare instances where model data is released before best track data on a new invest. That is an unusual circumstance. The old model system is displaying the data since its primary focus is on the model directory. The new model system will not until best track data is released. While I could design the new model system to also look at the model directory, I don't think it is worth it to check that directory as well. While it would be nice to have the models for 91L if we did not have the old model system running, the models would not be extremely important or else the NHC would have realized a best track file was not released and fixed it. The most someone should have to wait is 6 hours when it would be time for the NHC to do another best track update and then realize the error. It is not worth using extra resources to check the model directory for this rare error.
Work continues on the new model system. There are various small updates being made here and there. There is not anything big left to add, it is now a lot about improving the interface and fixing errors. I still have a list of things to do, but there are no major additions.
August 13, 2010...Minor improvements continue with the new model system. Some navigation tweaks, a page with a listing of most models, Google Maps model pages and model error tables now have model names popup when you hover your mouse over a model, and seasonal Google Maps with all storms from the season on it are the latest features. I still have some things to add probably and of course code cleanup, but the system is coming along.
August 10, 2010...A lot of changes were made in the past few days to the new model system. The wind and pressure diagrams were updated to go through 7 days. I also redesigned how they work due to an error when there is a lot of data. That is now fixed. The model error feature is currently available now for public testing, though error data on current storms active as of when I made the recent changes will not be available until after the storm ends. Anything that forms from now on should be available, unless I make additional changes to the file structure. I am going through my todo list to see what else I have to add to the system. I need to test the model error feature more. I also need to do some code cleanup and try to make it run a little better. Then, it is all about testing. I can't think of a whole lot left to add to the new model system in terms of the public appearance. There are some administrative things I might need to add.
August 6, 2010... Work on the model error feature is nearing completion and is nearly ready to be tested publicly ...
The model error feature appears to be working as it should. However, the model error feature is very complex. It will need a lot of testing before the data can come anywhere close to being trusted. I am in the process of writing the product description page. That page will eventually explain, in exhaustive detail, how the feature works. It is a very important page because data that is created on our site after a storm was active will not match the data had it been processed in real time on our site. (except for the final average run for the storm, which should match up unless model data changed or additional model data became available later on, which did happen in testing) Point is, it is complex and definitely needs a lot of explanation. It may take a few days to write the page. It is very hard to explain. Data will not be available for Colin or 92L until those storms end due to errors in data processing early on along with another error that occurred. (that is why the entire system is in alpha testing)
After this I work on extending the wind and pressure data to 174 hours as well. Then on to some back end features to the system which should then wrap up most of the work on the system. Then it is on to lots of public alpha testing for the rest of the year. If it does well, then it gets the beta label. The old model system will run at least through this season. The old model system will probably be turned off for the off season, leaving the new system on, and then I might fully integrate the new system into our site so that it is then the only system. At that time the new system would continue in beta testing though it should be stabler at that point.
I really need to get back to working on my ecommerce site, so the NHC advisory data system will probably be something I work on for 2011 or 2012 despite the back end being written for years. I need to do the web interface, which would probably be easy at this point, but then there is always the NHC error cone which has haunted me for years. (I never have been able to get the code 100% correct to display it in Google Earth so that you could adjust the transparency. Solid color fine, but not transparent without it looking weird.)
I have been doing a lot of work on the model error feature in the last month. I am currently designing the web interface for it. After that, I'll continue with more testing. The feature is currently running on the live system for my own testing and it is not available to the public yet.
I updated a lot of the pages that contain links on this site to make sure they were still valid. I need to add a lot more links at some point, but right now my focus is on the new model system. I also updated the Google Earth overlay product.
An example of the model error feature with the heat map feature turned on, which will compare the average error of all the models on the page and use color to indicate which are above and below the average error for each particular model.
From fixing the recon system on this site and the mirror to allow for the slight change to HDOB messages for missing data to coding the new model system to bypass the NOAA FTP server error that does not put out a renumbering file, it has been a long 15 hours of coding and testing and recoding and testing and a lot more where that came from. It would be nice if NOAA always had a renumbering file. (That is why NRL still has 97L as of this writing at 5:30AM if you were wondering). I'm too sleepy to go over everything. But at the moment, eveything works. Except that past data for 98L and Bonnie are not available. That got lost in the massive amount of testing I did. I'll recreate the past data once the storms end. As for the model error system, that needs a lot of work still. I have a new plan. One that requires much more processing, but I'm just going to do that and be done with it. It has been too difficult to find other methods. One issue was the renumbering file not being there. I have fixed, oh I do hope I have, that error, but there is too much chance for the model error system to do something wrong unless I do the more processing method. I was trying to save the error up to a certain point to save processing then add to it for the current run. But the numerous ways that could go very wrong is too much. I just need to open all the error files and add up the error each time. (4 files a day, so if a storm was active 7 days, 28 files whereas if I did the way I wanted, I would have had one file with the latest model errors all added up and then added just the latest error. Would have saved processing, but too hard to do.) Now I'm off to bed. If something goes wrong email me. If something goes really wrong with the recon system, call me.
July 22, 2010... Our new real time NHC model error feature is being tested on the live system ...
It is not yet available to the public, but starting with 98L, the new feature is being tested internally on the live system. It needs a lot of testing and a lot of work still needs to be done. Here is an example of the feature when the model data was created for this year's TD 6E in our offline tests:
Please note that the above feature has yet to be well tested and may not contain an accurate error for that storm. The chart will act like our other diagrams. They will be interactive. This one is just an example to show the progress.
Testing is currently taking place on an offline setup due to the large amount of testing that is taking place. It is much easier to test the next round of features that way and also does not use a large amount of resources on our host's server. Current offline testing involves coding and then testing our new real time NHC model error feature. A lot of testing needs to take place on this feature before it is added to the live system. Even after it is, it may not be publicly available until a lot of real time testing takes place with live storms. A considerable amount of work still needs to be done on it.
The recon system has been updated to handle the latest change to HDOB messages. Missing data will now be coded as "///" rather than "999". This has not been well tested on our end yet, but the tests we did run were successful.
While I have not updated this page in awhile, testing has continued non stop. Countless updates have been made to the system and many more updates still need to be done. For the next week or two there will likely not be any really visible changes to the new model system since I will perform a lot of tests offline on new features that are easier to test on an offline server. The system will remain in alpha testing for months to come before it will become stable enough to be in beta testing. There are a lot of things I still need to do.
June 16, 2010... Testing continues and the web interface is being constructed...
I have not worked on any of the updates I want to make to the back end of the system yet. I am still just testing it and developing the web interface. As I need more data in the web interface, I need to make adjustments to some of the output to the system, such as adding additional files it creates or changing the file structure. I do anticipate more of both. A preview of the web interface can be found here. It will take some time to get the archive built and running along with the Google Maps and Flash diagrams. While Pacific data is currently on that page, it may not stay there. There will be an active model page on the East Pacific and Central Pacific sites, but until I construct the Atlantic one, it probably will not be available on those two sites.
The system got its first rename file test today for 93E/02E. It almost went perfect. I needed to make one adjustment as 93E did not go away as it was supposed to. I know exactly what happened and minutes after I was able to code the fix and it then operated as it should.
Lots of testing has occurred over the past two days. There have been countless errors in the system, but as they popup I have fixed them. The errors have been easy to fix so far. Right now the system is operating for three storms, an invest in the Atlantic and two invests in the East Pacific.
The system needs more testing. The biggest test being what happens when a storm is upgraded from an invest to a depression or higher. That will be the big test. In addition, if a storm ever switches basins, I need to think about what should happen. The system is currently designed to move all the data to the storm it ends up being. I'm not sure I want to do it that way.
I am working on the web interface now before I get back to improvements to the Google Earth part of the system. I need to make sure I have all the data saved that I will need to generate storm pages. When the web interface is a little more complete, I'll release it here.
In order to begin testing of the system, the back end of the new model system has been turned on. It check for new data at :02 and :32 past every hour. New features will be added, but since testing will be done offline for those features and then added in there was no time like the present to go ahead and turn it on. The system was turned on yesterday and has been operating for the North Atlantic, Eastern North Pacific, and Central North Pacific basins. You can access the new system below:
http://www.tropicalatlantic.com/models/googleearth/Models_for_the_North_Atlantic_Basin.kmz
http://www.tropicaleastpacific.com/models/googleearth/Models_for_the_Eastern_North_Pacific_Basin.kmz
http://www.tropicalcentralpacific.com/models/googleearth/Models_for_the_Central_North_Pacific_Basin.kmz
The new system is in early alpha testing. There will be bugs, so the old model system will continue to run and serve as the primary system on our site for now. If you notice any issues or have suggestions, let me know.
I'll be adding a legend for the wind swaths, looking into a better display of the wind swaths, improving the wind and pressure diagrams, working on the web interface, and looking into adding model error rates. I also need to work on seeing what data I will need for the web interface and creating a file that has what I need to create storm pages.
As of now, there is no web interface to this system. However, that will change in the coming days or week. The new system archives all of the data so you can get the previous model run you want to look at. The system is currently archiving data. For now you can manually get the data you want like this:
http://www.tropicalatlantic.com/models/archive/al/2010/92/0612/nhc/atcf/2010061412/2010061412.kmz
Changing it to get what data you want. Since invests repeat, the start date of invests is included in the link. "0612" above is June 12, the start date. For depressions and higher, it would look something like this (not a real example below):
http://www.tropicalatlantic.com/models/archive/al/2010/01/nhc/atcf/2010061412/2010061412.kmz
For invests that develop, all data, in theory assuming the system works correctly, will move all the data to the new folder for the developed storm. And if a storm ever switches basins, the system is designed to handle that as well, but until it occurs I'll be unsure if the system will do it properly.
The back end of the new model system just had its first successful live online test for 92L. I'll be testing some more combinations of tests tonight and adding more features. If the testing goes well, I might start running the system just after models are expected, to save server load.
June 13, 2010... The back end of the new model system has been completed and online testing has begun...
A whole bunch of progress has been made on many aspects of the new model system. Live public alpha testing could start at any time. Check back.
Last night I added the new experimental model plots using Google Maps Version 3 to the current model system. Look for "View our experimental Google Maps page." just above the Google Map when a storm is active. If you notice any problems or have any suggestions, let me know. I will likely improve the layout of the options somehow and will also probably have a way to display the model map full screen when the new model system gets going in the next few months. Right now I have a temporary solution to have a bigger view of it.
Internal online testing began today on the Google Earth part of the ATCF database system that displays the NHC model data, past best storm track, model wind swaths, and wind speed and pressure diagrams (though the diagrams do not display if there is a lot of data). Things are fluid at the moment and I have no idea when I might release it for testing. I am doing testing with it for the current active system. I am making a lot of changes to a lot of different things. I am still deciding for example what the folder structure will look like on my site for the new system. Since I do have at least one other system that will eventually be added, displaying advisory data from the NHC in Google Earth though without the error cone for now, I need to decide where everything will go. That advisory system (again without the error cone which I have spent hundreds of hours over the last four years trying to figure out), along with a UKMET system to display the model line, have been finished for years. I just never built a web interface for it. Once I decide on where stuff will go, I might release the system for alpha testing. There are still things to be added. I do plan on having model error data, but since the new model system is extremely easy to test offline now, I can release updates to the new system so that it remains operating. The way the new system is setup, I don't have to change anything in the main system file in order to test it on an online server and switch to and from am offline server. All the main configurations are in a separate file so that the paths to everything do not have to be in the main script. (Meaning a config file for the offline server and one for the online server and that's it!) I probably will set it to run just a few times after models come out and not all the time since I don't want to stress the server too much.
While I have not built any of the web interface for the new system, it would not be difficult to create one. It took me less than a half hour to add the Google Maps test onto the current model system. And, I have the recon archive system as a guide.
If you have any ideas on the Google Maps or Earth system, let me know. I have already received some feedback that is being incorporated into the new systems. Contact me here! Someone suggested adding a legend for swaths and I am going to do that. Also, I will try to see what solid color swaths looks like. I will do it for best track swaths, current and past, if it looks good, but I'm not sure about model wind swaths yet.
And here is something that might interest some, the new model system will run for the East Pacific and Central Pacific at the same frequency as our Atlantic system. This does not apply to our recon system or the future NHC advisory system, though the advisory system might run for strong storms. And the recon system we do turn on at times for the Pacific, but the complexity of that system will take awhile too convert so that it can easily operate in the Pacific. And, the recon system would increase the load on the server while the new model system will actually reduce the load operating for the additional basins due to the way it is written. Tropical Atlantic has finally sorted out how to display Pacific data when it's name is Tropical ATLANTIC. Buy more sites! Rather than move Tropical Atlantic to Tropical Globe, we instead acquired two additional domains, Tropical East Pacific and Tropical Central Pacific. And the interesting news is, when the new model system goes live for testing, these basins will start having model data on our other sites. The single new model system that operates at Tropical Atlantic distributes the data to the other sites since the sites reside on the same server.
And now I'll get back to coding and testing.
Work is is continuing with the new model system. I have made a lot of progress on it. Here is a prototype for the Google Maps model plots for the new system. It is written for version 3 of Google Maps. Here is the latest test of the Google Earth file. (open in Google Earth) Take a look at the swaths that are now separate as a network link and the new pressure diagram.
June 6, 2010...Work is ongoing with the new model system.
May 29, 2010 - Entry 2...
I have begun special model runs on Tropical Globe for the system in the Pacific. Other than an initial error that had everything labeled with an "L" rather than "E" to represent the East Pacific, it is working well.
The model swaths will be out of the main file in the updated model system. (The best track swaths for the current position and past track will be in.) It just makes the file size too big and could really impact bandwidth. And, very few people would use it. There will likely be a network link in the main file to call the big file of swaths in the archive, if there is any, for that run. It actually makes things easier for the system. The nice thing about this system is that it will be designed to be much less demanding than the current one is on the server. That is because the current system downloads the model data every time it checks for the data and then processes it, even if nothing has changed. I never could get the coding in the current system to work so that it would only check if needed. The newer version of the model system will check to see if new model data or a new track update is available and only proceed with downloading the large model file and track file if needed. And it will be intelligent enough to redo Google Earth files by loading text files for data that has not changed. For example, if only the track of the storm has been updated, the track file will be downloaded and processed. The model data will be already processed from the last time the system downloaded and processed the data. It will be read from a text file and added into a new Google Earth file that is created that will be loaded into the Google Earth network link you refresh. However, it doesn't stop there. (And here is the part I am still working on.) Then I need to print a file for the archive. Here is where things get tricky. (And while I have the logic down on how I will do it, I am currently in the process of implementing this part.) All model data and past storm data will be archive. But, it will be archived by time. That means that I can't simply copy the Google Earth file I just created in the example I just discussed because the track data is 6 hours newer than the early cycle model data (and the late cycle model data that would have been available when the early cycle model data came out). I need to, in this example in the archive, add the new track data to a new Google Earth file that will not contain any model data. This will be the archive file. When new model data comes out, if it does, then I will download and process the new models available that are valid at the run time for that location update. It's confusing, but it will make sense when I complete the system. Think about a year from now if you were to look for a storm in the archive. You would want the track up to that point and the model data for that run time to be in the same file, not from different times. There are some complexities involved that I also need to code for.
The NHC advisory system, if I release that as well for testing this season, will be separate from the model/best track system. That is because I don't want any confusion of things like the NHC advisory track and best track or best track intensity and an intermediate advisory that might upgrade the intensity significantly. If the NHC advisory system is released later this season for testing, that system will be more visible than the model/best track system. I have not worked out how it will all work yet. The biggest things to do is building the system that displays everything. But, it should not be that difficult. The most difficult part will be simply deciding how best to display the data. (Aside from the NHC error cone if I decide to tackle that again.)
Work continues on the new model system. While the system is more complex than I remembered, I am hopeful that something could be active for testing during this season since many parts of the system work and many parts are similar to work I have already done on other systems.
Here is a more robust example of the features of the model system in progress. Because there are so many early cycle models trying to load, the early cycle intensity diagram does not work. And for those looking for a technically correct version of this file for Katrina at the time it happened, it is not completely accurate as it would have been then. The best track data used in this example file was redone at some point in perhaps post season analysis because the best track data differs slightly from what the CARQ position is. The new model system will not readjust for post season analysis as each archive of each model run and storm track to that point is important as that was what was known at the time.
I might not have wind swaths directly loaded into file. They are huge. I might load that separate to save bandwidth. That is likely I think. It is possible I might separate the spaghetti data as well, but I might go ahead and leave it since all of that is not as considerable as what the wind swaths can be.
I am currently working on the three to four year old, yet never released, updated model system. Since I am determined to actually get it finished in the next few months, I thought I would once again release a non updating example of what it looks like. Once again, it doesn't update, just a static example. (Although based on comments and my progress, I might add a different version from time to time.) You can view an example of it here. It is still a work in progress. (I know it says "Add time of last valid data" at the top, but since the data will eventually be called by a Google Earth network link, I'm not sure what the top of the directory tree will look like yet.) If you have any comments, let me know. Take a look at the new features like the intensity diagrams, wind swaths, past track, and the ability to display 5 days worth of model data or all of it. These will be the main new Google Earth features of the new model system. That part of the system is nearly built as you can see. (but needs a lot of real world testing) But I do not have the Google Maps models in the system yet. That system will likely not contain the ability to view 5 days worth of model plots or all. I will simply display all of them. (Though I do have an indea on how I might do it simply. I just don't want to write two separate pages but I might be able to do it with one.) My simply idea worked and it will have the ability to display either the complete plots or 5 days worth of plots. Took a few lines of code. In fact it was so easy, I might try to even make it so that you can choose how many hours of data you want to display in Google Maps. The intensity diagrams will eventually be on a separate page, in addition to sometimes being in the Google Earth file. When there are a lot of models, Google Earth will not display the flash intensity diagram within Google Earth with the way I have it currently setup. The part of the new model system I have yet to work on is the archive. The new system will eventually archive all old model data. It will likely look a lot like our recon archive. In addition to the new model system, I will also work on a new system which displays advisory data, though when that system will be ready I do not know. That system is also nearly complete, at least the Google Earth part. (Although as previously stated through the years, I never have figured out the NHC's error cone so that it looked perfect when partially transparent. I can make it a single color, but making it so it looks right when transparent, which is generally how you would want to look at it, I never have figured it out. It is always just a tad bit off in some circumstances.) I need to build an archive system for the advisory data system as well. I might just release some different versions of the NHC error cone too, ones that work but that just can't be made transparent. I still don't know how these systems will interact. I think the model system and advisory system will be completely separate to reduce confusion. But, I am also working on a way to integrate everything into once place. (including the recon system) An early mock up of that is here and you can also let me know what you think on it as well. (Note how I was working on this updated system way back in 2006.) I would likely also make one Google Earth file that contained the recon system, model system, advisory system, and overlays, but all it would do is combine the network links into one Google Earth file. It would be a little crowded to make that the main product, but nice to show off all our Google Earth products in one place.
You can contact me here with comments.
Lastly, as for the Pacific data on the mock up page, the model system will be designed, though not well tested, so that it could be turned on for the East and Central Pacific basins. (The new system is being designed so that people who run our system on their own server for commericial use can easily run it and customize it.) It doesn't mean that our site will provide Pacific data on a routine basis. Additionally, advisory data and recon data will likely not be part of any new system. Although we currently provide recon data occasionally for the Pacific when people request it for active storms, I have not yet customized the Atlantic recon system so that I could easily have Pacific recon data all combined into one system. (As you can see from the look of that page, it is not well designed over there.) In the future, long in the future, I might move everything over to tropicalglobe.com and then provide Atlantic and Pacific data, but since it would put double the load on my host, I'm not sure I could get away with doing it. But I am a long way from being able to do it.
The model system is now operating at in-season frequency for 90L and will continue to operate at that frequency until the end of hurricane season. Meaning, our site checks every half hour (:00 and :30) for new model data. It also updates at 0:45Z, 6:45Z, 12:45Z, and 18:45Z to try and catch recently posted model data up to 15 minutes sooner than it normally would. (During off-season mode, our site was only checking every hour (:00) for new model data.)
May 22, 2010...
It's almost time to begin another season. So far this year I have been working on another project, an ecommerce site I plan on launching later this year. That for profit venture should permanently fund this non profit site. However, since I have been making some money doing something other than that as well, I now have enough in reserve to work on this site some more again. And while I promise nothing, since I have been dragging my feet around on it for three years, I am finally revisiting the updated model system. I hope to have something working and available for public testing during this season. I am only starting today again on the project and revisiting old code, so it will take some time to get used to what I was doing previously and to see how far I have yet to go on it. It will not include the NHC's error cone. (I still have nightmares trying to get that thing to form correctly in Google Earth, lol.) It will likely not include NHC advisory data to begin with, even though that system was fairly complete. (aside from the NHC error cone) The new system is designed so that new things could be added later.
I just don't like the fact that I don't have past model archives. It would be helpful to see where the models have shifted. (Trust me, I've heard your feedback on that one.) And, I would like to offer 5 day plots in addition to the sometimes two week long plots which can sometimes confuse things. And, I want the intensity chart I have had completed for a few years to finally go live. Since there is still no really good sites online to display the data how the new system does it, it's about time I finally got it done.
So in review, I promise nothing, but hopefully I'll finally get the new model system up and running this year. The new model system will run parallel to the current one when ready. It will not impact the old one. Eventually, the old system will be discontinued and the new system, in a new location, would then take over. (The old system would let you know where the new one is.) That is unlikely to occur before the season ends because there will need to be a lot of testing.
Tis' the off season. Time for off season mode. Model information is now downloaded hourly. (0:00Z, 1:00Z, etc.) Recon data is now limited to four live decoders. Vortex messages and RECCO messages will not be checked and the Air Force backup product for HDOBs and dropsondes will not be checked either. This saves some processing. If something develops I will resume normal site operations.
November 29, 2009...
I have made a lot of changes to the recon system in November. Therefore, things may not work as well as they used to. The most significant visual change is that missions that only release dropsondes (and do not provide high density observations) now show an approximate path of the plane. The path is not perfect of course because the path simply connects the points between each dropsonde. (The interval between sondes can be very large.) There are other changes and additions as well. I added historical recon. Previously 2007 only had HDOB messages. I have now added all the other recon data for that year. I am also going to add other recon data over time for historically significant storms. Katrina and Andrew have been added so far. (Keep in mind that our site does not decode supplementary vortex messages. That product is no longer used. We never built a decoder for it so it does not exist in our archive for any of the historical storms.) One last major visual change is something I add manually. That is a picture of all recon missions that have taken place for a storm. It is more just for interest since the image is too cluttered to make much sense of it. It just gives an idea of the amount of recon that occurred in the storm. Keep in mind however that I create these images manually. They will likely not be created until after a storm has dissipated. It might not even occur until the end of the season. There were also a lot of little updates. Some minor fixes here and there. HDOB messages in the Google Earth file now have the time of the observation in the sidebar rather than just the OB number. This is helpful for some historic recon that had no observation numbers. There was also a lot of tweaks done in order to process historical recon. The decoder did not handle historical recon as well as I thought. It now works much better, but if anything still does not work for you, let me know.
As for the update to the model system that has been years behind schedule, it will be delayed even longer. I have a large project that I need to get started for another site I am starting and it will take a significant amount of time. (And even after many months of a ton of work it will continue to be time consuming for many years to come, but it will fund this site and all my sites for many years to come, so that makes it worth it.) The updated model system might not be ready by next year. As for the hopes of ever having an NHC error cone in Google Earth, that system will not be part of the model system. If I ever take that back up again it will be a long time from now. I gave up trying to figure out that error cone in July. And then in August when I got the bright idea to take it up again, my computer got destroyed by lightning (like just a few days later), so I think something is trying to tell me something about that product. I do want to get the updated model system out though, but I do really need to get started on this other project that will fund my sites for a long time. Maybe I will have time before next season starts to do the updated model system, but I'm not confident in that.
Just when I fix one problem with the error cone, three more appear. I'm starting to think I will never have the NHC's error cone in Google Earth. I get so close but then it all falls apart after further testing. As is, it would work pretty good in a lot of cases. But when a storm makes a sharp turn and/or a storm gets high up in latitude, errors are very apparent. Also, cosmetic issues exist when one error circle lies within another error circle. I really don't want to release a product that has mathematical errors. I'll keep working on trying to figure it out full time, but at some point I'll just have to quit trying. I've been working on and off on the error cone for three years. It's getting old.
June 26, 2009 - Update 2...The system wrongly displayed TD1 again today. When I tried to patch a previous error, I think I removed the code that checked for today's error. It should be fixed now.
June 26, 2009...I realize that the way I am doing the NHC error cone involves a lot of coding. I wanted to include multiple versions of the cone, like a 3 and 5 day. However, the way I am coding it involves so much code. Additionally, I'm not sure how it will look with multiple polygon shapes that are supposed to be seamlessly joined with others. The way I was going to do the cone was by taking many polygon segments and displaying them all so that it looked like a single cone. I think I might go back and try what I wanted to do originally, draw the cone so that it is simply one polygon. While the way I am doing it will look fine far away, if you got extremely close to where some of the polygons meet, you would probably see some areas not covered and I would prefer not to have it like that. With that in mind, I have decided to try to do one more mathematical formula to calculate intersecting lines when a cone turns. If I can't figure that out, I realized I can do what I was going to do in a way that will use less code and not have as much of the problem I mentioned. (Additionally, what I would be able to do is by default have the three day cone shown and have an option to display the next polygon which would be of the 4 and 5 day part of the cone, meaning I would not need to repeat the cone for the 0 to 3 day again.) I'll see how it all works in the next few weeks. I have not yet tested the equations I found to calculate the intersection of two lines. I think I might need to find an easier equation, because the one I found is massively long.
June 11, 2009...I have finally completed the project I was working on for a client's site. Now it is back to working on the Tropical Atlantic site. I have no idea when I will finish the new model / NHC advisory system. I am just now revisiting trying to do the NHC's error cone in Google Earth. I stopped working on it on August 27th, 2008 just when I think I finally figured out how to do it. Now I can see if the idea I had works! Now I just need to find what that idea was. I wrote it down somewhere, lol. Just kidding, I know where it is. (Although it took some time finding where it was on my computer. Windows Vista has a terrible built in search engine.) I keep a text file in every folder I am working on that has everything that needs to be done and my ideas on how to accomplish each particular task. So, I'll start working on the error cone immediately. If that does finally work I will begin to integrate all the scripts together into a bigger management system that will never delete model data. That is going to take a lot of time though. I'm not sure if I'll have it live this year. I will probably be testing it with real storms, but I don't know at what point I will let other people see the alpha product.
April 26, 2009...Our site has once again resumed off season operations. All systems are working at the moment. I am currently on the same host I was. They told me what to do to solve the problem so that it should not occur again. However, it could still break at any time. During the busy part of the season I will consider signing up for a VPS for a few months to make sure that at least one data source will be available to handle heavy traffic load.
April 23, 2009...I have asked a lot of different hosts about whether they can host this site and none will host it on shared hosting. My current host got tired of this site I think, so that is why the site is not currently updating. (It is not able to contact another site to get data.) A virtual private server (VPS) is a lot of money and I don't know much about it, but I have no choice but to get one. In mid May I will have time to make the move. To save money, I will likely shut down the model and recon system during off seasons and only signup for a VPS during the hurricane season. The rest of the year I will move the site to a shared hosting account. I will probably keep a shared hosting account all year that way I only have to move the Tropical Atlantic site each year. I will likely be getting at least $900 this year from managing some things for my neighborhood, so that will definitely help. All that will be going into a bank account I setup for Hollis Innovations. If I didn't have that money coming in, I'm not exactly sure what I would do. (I am getting paid $100 a month and started getting paid the first of the month, starting on April 1, 2009 for the month of March, which is the reason I have been busy.) That will help pay for several years. Hopefully I will continue to be paid past November, but that is when a new board would take over and I'm not sure if they would continue my services. It is possible that as a result of the increased costs I might have to put advertising on the site in the future. I will never charge any money for the data, so you don't need to worry about that. I just need to make sure I know where I am going to get about $30 a month if my neighborhood no longer pays for my services.
April 17, 2009...Tropical Atlantic is "@TropicalATL" on Twitter. If the site goes down, I will use Twitter to provide outage updates and when I can to process data on my own computer and link to it from the Twitter feed. I have also have some other ideas for the feed for next year.
In order to further reduce the load of our site, I have removed our KBIX live decoders until after the off season. I might remove RECCO messages completely during the off season. Our host doesn't seem to like the amount of updates our site does. So, if I have a less demanding site during the offseason, maybe they will put up with the other 6 months of the year.
September 6, 2008...I'm currently side tracked on a project for someone. Not sure when I will get back to the 2009 model system.
August 24, 2008...Work on the new model system for 2009 continues. The examples below, and the alpha preview for Fay, will still be redesigned quite a bit before release next year. I found a problem with combining all the data into one file for Google Earth. I will likely create separate network links for the data that will be available, such as for the NHC models, NHC advisory data, and ATCF track. There is a lot of work left to do. I am revisiting the NHC error cone I tried a year or two ago too. I might finally have a way to do it in an acceptable enough way.
August 14, 2008...
Here is a sneak peak of some examples of the 2009 model system: 92L (August 14, 2008 6Z) | 92L (August 14, 2008 0Z)
What do you think? Let us know of any suggestions you have.
Work on the new model system for 2009 continues.
August 12, 2008...
So what went wrong today with all our systems? Well, all but the model system, which helped make the error even more difficult to diagnose since it remained working.
I don't understand the technical details, but basically our host made an administrative change that must have disallowed a lot of access to sites on the internet. They disabled the change they made and the site is now operating. However, it doesn't mean that it will not happen again. There is unfortunately nothing I can do when it happens.
Meanwhile, some news about the 2009 model system. It is going very well. I might have an example file that I release for feedback in the next few weeks, though the system itself will not operate until likely 2009. It has a very complex folder structure and I will be looking for feedback about how the data is presented.
Anyone have any suggestions for our recon system? I've been getting some feedback about it. Let me know of what you like, what you don't like, and how we can improve it. I've received some good suggestions so far, but unfortunately most of the things appear a little too difficult to do in Google Earth at the moment.
July 24, 2008...The live recon system has been working fairly well, but it has had some hiccups. I have made some tweaks here and there, including some minor updates and improvements to actual product decoders themselves. Our site was able to handle most of the traffic load for Dolly. Early on with Dolly the site went down for about 10 or 15 minutes, but the rest of the time it seemed to be up. (Except for Saturday's errors which was not due to traffic but to a rewrite of part of the system.) For not serving much of any images, our site experienced quite a bit of bandwidth this month. Our site experienced over 11 gigabytes of bandwidth with just 1,065 unique users. (Bandwidth: July 20th - 2.1GB; 21st - 1.8GB; 22nd - 2.1GB; 23rd - 1.7GB; Most Hits: 172,300 this month on our live Air Force HDOB decoder page, totalling 1.65GB of bandwidth, meaning if people had to actually process that data in our decoder, our site would have crashed a long time ago.) This would be the reason why we don't ask media outlets to display our name on air. We'd be swamped many times over if we had more visitors.
July 21, 2008...I have begun work on the new model system. It's going to take awhile to get my feet wet again. I have made improvements to the old system over time that are not in the new system that has been on the shelf for a year, so I am running a comparative analysis to see where updates are needed.
July 21, 2008...
My next project will be a new model system that will offer a few more options than the previous one. Some of it is done. I just need to build a management system that works like the recon archive. In addition, I will provide an intensity output. I built an intensity graph last year and forgot all about it! It was an early experiment when I was building the dropsonde table:
It is based in Flash and gives you control of plotting just those intensity models you want. You can choose to plot all or none too, just like the Google Map for the current model system. I have been waiting for a year or two to do all this. I was going to wait until I had my storm system ready, but I think I like the idea of separate parts for now. As you may have seen from the recon system problems of the last two days, having a lot of things working together means more things can go wrong. Once I get all the separate things working, then I can think about having a system that connects it somewhat.
Some of the features of the model system will include 5 day plots rather than the current "however long they are they are" type plots. This can be confusing to some. Is that long line 5 days out? No, it might be two weeks out. Having the option to plot just the 5 day would be a good idea. Will also have a storm history track with ATCF database data.
It will probably be a few months before I have it all completed. After that, then I might try to tacke the elusive NHC error cone in Google Earth again. Never could get it to work. (Because I didn't know how it should be done.) I might see what I can figure out about it.
Long term...
The really long term goal is a complete storm system with NHC models, advisory data, and recon data. There is likely no way I will be able complete all of this by the end of the year. (I have other projects, non tropical related, that I work on too.) The advisory decoder is actually already built, has been for a couple years, but it lacks an error cone which continues to confound me and a management system like the live recon archive. I also want to have model data available in a system like the recon archive so you can view runs you missed. Currently, the system deletes model data when a new run is available.
Our vortex decoder is now in alpha testing. A more stable version will likely be available within a few weeks after testing of numerous vortex messages. Things are looking rather good with it at the moment. I still need to test more vortex messages, but I like the consistency so far. Have a comment or suggestion for it? Let me know!
Meanwhile, our other decoders are being slightly reworked behind the scenes. You will not notice anything different, unless something stops working briefly. If something stops working for an extended period, let me know. What is currently being worked on is the ability for other site systems to use the decoder to decode things, such as how the live decoder for URNT15 works. All the decoders have had the groundwork laid for that transition and will allow for at least two other ideas for the future. Those will eventually be all decoded vortex messages on one page in real time and the even longer term goal, sorting all the data for display in Google Earth (and possibly, though not likely, maps) in connection with the even longer long term goal of a complete storm system with NHC models, advisory data, and recon data. There is likely (almost definitely) no way I will be able complete all of this by the end of the year. (I have other projects, non tropical related, that I work on too.)
For the past month I have been working on a dropsonde decoder. It is currently in alpha testing to anyone who wishes to test it. The decoder almost certainly has lots of errors, so none of the data should be used for any purpose other than for helping me to determine errors. If you find a dropsonde report that cannot be decoded, please send it to me here. Just paste it in the text box on that page. I will then look it up in the recon archives to get the actual formatting of the message. There are a few things noted in the dropsonde decoder that mention that the decoder can't or does not know for sure how to decode. If you know how that particular part should be handled, please let me know. The decoder is now "more friendly" to older messages, though I still have work to do.
October 12, 2007...In the most recent invest, 97L, old model data appeared. This data was associated with the previous 97L that existed several weeks ago. I have put in an extra error check to make sure that the last late cycle model data should be associated with the current invest.
September 25, 2007...TD13 for a few hours did not appear correctly. A mistake existed in the ATCF database. There was no file for the storm, but there were models, so it threw off my site.
September 12, 2007...So there is an error where an invest and the storm that was upgraded from that invest both display on our site at the same time. It turns out I did fix that error in the more simplified system and it is the NHC's site that currently has the problem. The error would occur no matter what system I had running. They don't have a renumber file that says 91L should have become 08L. That creates a problem. I have added a temporary line to the script to ignore 91L. I can see where that kind of error in the NHC ATCF database would cause the model system I am rewriting to have problems that would require a manual intervention that would be rather complex. That is now another complication I will have to think about being able to handle.
September 10, 2007...I have released a new site layout.
September 6, 2007...I completed a beta version of my new decoded recon archive. This is an excellent addition since it saves most every mission that goes through the live recon decoder. (Non tasked missions are dropped for now.) I have a lot of ideas for the recon decoder, but I'm not sure when I'll get to them. Once again I'm not sure what I will work on next.
September 2, 2007...I have added a few new things. I added a distance calculator and also setup the live recon decoder to include NOAA HDOB messages. It also tells you on the main page if the product has been updated in the past 30 minutes.
August 29, 2007...I have finally got my old computer working after the newer one's power supply unit died. I reinstalled my old computer and now have it up to date. I decided to reinstall it in the hopes it might work better if I started fresh. Unfortunately the hardware is very outdated, so until I get a new computer, I will not be adding any more Google Earth overlays because I can't easily test things. I'm sure I'll still be able to test my new Google Earth model plot system. I haven't decided yet what I will start working on next.
August 14, 2007...Storm and model point icons were briefly unavailable tonight. I changed the path to the images and did not update it correctly. All is fine now. I moved the image locations from where they were on my test model site. In the new model system I am about to return to working on, images will be zipped inside Google Earth files so that you do not need to connect to download the imagery each time. That will be just one of several improvements. The new system will not be available for several months are longer.
August 13, 2007...I'm still working on Google Earth overlays. I think in a day or two I might get back to the model/storm system I began work on last year. I really would like to get parts of it in beta testing running along side the current system. I really hope I can have it working before the hurricane season ends. For the next few days I am running models for Hurricane Flossie. As soon as the threat is over, I will remove that system. I have other things planned in the future, such as a redesign of the site, possibly adding model data permanently for the East Pacific, building a vortex decoder, more Google Earth products, and other various things around the site. Lots of things to do in the coming year.
August 8, 2007...I'm still adding overlays to my Google Earth overlay product, so I have yet to add it to the site. I have now completed a live recon decoder which automatically downloads the latest Air Force HDOB data and decodes it.
August 2, 2007...I have yet to add a link to it on the site, but I have created a Google Earth file containing a large amount of satellite overlays. It can be used in conjunction with our model system. You can download it here. Please read the "About / Help" section in the file for some notes about the product. The product works nicely for displaying model data over certain satellite overlays.
July 14, 2007...I have been working on a lot of different things in the past week. I have been adding things here and there and researching some of the different NHC models so that I can label all the models in the newer model system. I also bought http://www.tropicalglobe.com/. Eventually I will have links to information in other basins. However, I will never provide up to date content for other basins. Too much work. I will be providing model data for East Pacific storms, or at least I plan to, when I release the new model system, but that will likely be for next year. I might possibly move the new model system over to the new site I bought since it will be for the Atlantic and East Pacific, but I don't know yet. You may notice in the future that certain parts of this site will move to the new one. I doubt I will move much over though.
Older updates...
I have returned working on the newer model system. I might not do anything with the old system that is currently active. Now that it seems to be working, I prefer to leave it alone and work on the newer system. However, I might still make a few more upgrades after I test part of the new model system. There are some important and informative updates I would like to make that I am working on right now in the new model system. The entire new system will take many, many months to test, but certain elements, like the model system, I think are worth releasing sooner rather than later. I will probably run a test alongside the current model system for a bit when I am done and then add it into the older system. The past storm track using ATCF database data is something I might add to the earlier release, if I make one. The newer system will provide both the past storm track using ATCF database data and advisory data.
Our old model system has now been mostly restored. The plots in Google Earth and Google Maps have been restored. They appear to all being working and updating properly. I don't think I will add back the CSU imagery. Those images are very large and require a large amount of bandwidth to serve them. Last year they used as much as a gigabyte per month and they were not used by many people. Less than 500 people per month used them and only a small fraction probably spent time looking at them. My model plots provide nearly the same data, though the intensity guidance is best displayed on the CSU site. (I will be displaying SHF5 and SHFR intensity guidance in the new system.) I like the idea of using Google Maps or Google Earth better than images. You can add the model data you want and remove what you do not want. Otherwise it gets all cluttered. People like to post the images in message boards, but I think if people take the time to visit a link they will find that there are better ways to display the data.
The HWRF model is now available through my site's model plots when it is available.
I am working on this site full time throughout the summer and into the fall, so hopefully I will make a lot of progress on the model and storm system I began last year. When it is finally rolled out in beta form, it will run alongside the system that was just restored. There is a lot of complexity to the newer system and it will take a long time before all the bugs are worked out. It will be at least several months, probably longer, before the newer system is ready. I think it might be next season before it is operational. That would give me time to test everything all throughout this season. The entire new system will also contain NHC advisory data, past storm track, UKMET model data, and other various improvements. The newer model system is rather complete but is not yet integrated into anything. Of course the same can be said of the other parts of the system, which seem to work well so far by themselves. I need to write the script that combines all the data and manages it properly. (such as being able to view any previous model run of a storm) The part that needs to be tested well next will be the part that renames files from invest to a numbered storm. I am still deciding on some of those features.
A beta release of the new HDOB decoder for 2007 is now available. I still need to create dropsonde and vortex message decoders. I decided to work on the vortex decoder first. I realized it may be the more difficult one to create and the sooner I complete it, the sooner beta testing can begin.