diff --git a/.nojekyll b/.nojekyll index 4df4970..ffbe18e 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -fb2cea08 \ No newline at end of file +7f5e2b21 \ No newline at end of file diff --git a/about.html b/about.html index ead0081..fc7f8e9 100644 --- a/about.html +++ b/about.html @@ -131,6 +131,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/gsi/01-gsi.html b/content/gsi/01-gsi.html index 90a3914..0275954 100644 --- a/content/gsi/01-gsi.html +++ b/content/gsi/01-gsi.html @@ -180,6 +180,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/gsi/02-convencionals.html b/content/gsi/02-convencionals.html index dfb7810..5943037 100644 --- a/content/gsi/02-convencionals.html +++ b/content/gsi/02-convencionals.html @@ -194,6 +194,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/gsi/03-radiances.html b/content/gsi/03-radiances.html index 23890ef..a0164bb 100644 --- a/content/gsi/03-radiances.html +++ b/content/gsi/03-radiances.html @@ -214,6 +214,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/gsi/04-diagfiles.html b/content/gsi/04-diagfiles.html index a9a1e67..4bb05e2 100644 --- a/content/gsi/04-diagfiles.html +++ b/content/gsi/04-diagfiles.html @@ -165,6 +165,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/gsi/05-tutorial.html b/content/gsi/05-tutorial.html index e994443..c68e4d7 100644 --- a/content/gsi/05-tutorial.html +++ b/content/gsi/05-tutorial.html @@ -165,6 +165,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/observations/01-bufr.html b/content/observations/01-bufr.html index 6548699..b9eefbf 100644 --- a/content/observations/01-bufr.html +++ b/content/observations/01-bufr.html @@ -165,6 +165,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/content/observations/02-conv-bufr.html b/content/observations/02-conv-bufr.html new file mode 100644 index 0000000..29469e7 --- /dev/null +++ b/content/observations/02-conv-bufr.html @@ -0,0 +1,874 @@ + + + + + + + + + +DA-documentation - Conventional obs bufr + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    +
    + +
    + +
    + + + + +
    + +
    +
    +

    Conventional obs bufr

    +
    + + + +
    + + + + +
    + + + +
    + + +

    This section covers how to add new surface observations to prepBUFR and how to divide a file into smaller subsets.

    +

    One of the first challenges (besides to learning how to use GSI and deal with bufr files) was to incorporate surface observations from automatic stations to the assimilation. In Argentina there are very few official stations to cover the entire country but there are private automatic station network that could potentially help to improve weather forecasts thank to their higher frequency (10 minutes) and spacial resolution.

    +
    +
    +

    +
    Location of automatic stations (green squares) and conventional stations (orange triangles)
    +
    +
    +

    This requires to incorporate these new observations (available at Garcia et al. 2019) to the prepBUFR (so, be able to write fortran code and understand the bufr format). The 2 programs included here are based on the INPE/CPTEC modules and programs that are part of their operational assimilation system. I was lucky to visit CPTEC in 2019 where I learned how to work with these tools.

    +
    +

    Add observations to a prepBUFR file

    +

    In the previous section I briefly explained how to read and write a bufr file. To add new observations to an existing file (o to a new file!) we use the same functions mentioned here and structure of the routine is also similar. But we also need to figure out how to read the new observations and how to create new reports.

    +

    The code I’m sharing here assume that the new observations are in csv format and has the following look:

    + ++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    LATLONELVSTATIONyyyymmddhhmmssTOBQOBUOBVOBPRSS
    -19.0124-65.29212907“BLV102”2018111120000028.90.006249367199416470-2.8808888888888971950
    -17.4111-66.17442548“BLV124”20181111200000290.00636021217641851-3.27390439689371-3.2739043968937274280
    -16.8378-64.7925195“BLV141”2018111120000032.90.0439748860789023-1.34593847427853-1.3459384742785350240
    -17.9747-67.084057“BLV154”2018111120150019.20.00317768840091676-6.08364406830543-2.5199278817427462030
    -17.6364-67.20033798“BLV157”120181111200000190.00626217287136847-5.34737718159307-5.3473771815930764460
    +
    +
    +
    + +
    +
    +Important +
    +
    +
    +

    If you want to convert the files available here into csv tables, here is an R script to do that.

    +
    +
    +

    The program read the configuration file that define where is the namelist and the prepbufr table and call a function that start the process. The namelist will list the name of the prepbufr to modify and the csv files with the new observations. The most important subroutine is adpsfc_rw_prepbufr():

    +
      +
    1. Reads the bufr file
      2. +
    2. Reads the csv with new observations
      3. +
    3. Decides if the observations fall into the time window
      4. +
    4. Creates a new report type 187
      5. +
    5. Write the bufr file
      6. +
    +
    +
    +

    +
    add_obs rutine flowchart
    +
    +
    +
    +
    +

    Filter observations in a prepBUFR

    +

    When I started using GSI I thought that the bufr files needed to have the observations for the assimilation window only, in other words, that GSI was not capable of filtering and ignoring the observations outside the assimilation window. I know, it does not make sense. Before I realized I was wrong I created a program to read a prepbufr and write a new one only with the observations in a specific time window.

    +

    The general structure is similar to the previous routine. In this case the read_namelist module will read the namelist and list all the available prepbufrs. The magic then happens inside filter_obs_prepbufr() that loops over each observation and check the difference between the observation time and the analysis time. If that difference is less that half the window, the observation is written in the new prepBUFR file.

    +
    +
    +

    +
    prepBURF generation / filter obs rutine flowchart
    +
    +
    +
    +
    +

    The code

    +

    The source code is publicly available in this repository. Each source folder includes a compile file as example of how to compile the program. It needs the bufr and w3lib to work with times.

    +

    The run folder includes all the configuration files, namelists and bufr tables. The can be uses to run the programs using the example observations available in the example_obs folder.

    +

    Finally, if you ever need to do this for many files, the run_bash folder has 2 bash scripts that modify the namelist and run the programs in a loop.

    +
    prepbufr_tools
+├── example_obs
+│   ├── 2018111719.csv
+│   ├── 2018111720.csv
+│   ├── cimap.20181117.t20z.01h.prepbufr.nqc
+│   └── prepbufr.gdas.20181117.t18z.nr.48h
+├── README.md
+├── run
+│   ├── add_obs.conf
+│   ├── namelist_conv.PREPOBS
+│   ├── namelist_conv.PREPRW
+│   ├── prepbufr_gen.conf
+│   └── prepobs_prep.bufrtable
+├── run_bach
+│   ├── run_add_obs.sh
+│   └── run_filter_obs.sh
+├── src_add_obs
+│   ├── compile
+│   ├── m_adpsfc_rw.f90
+│   ├── main_add_obs.f90
+│   ├── m_convobs.f90
+│   ├── m_vars_global.f90
+└── src_filter_obs
+    ├── compile
+    ├── m_filter_obs.f90
+    ├── m_read_namelist.f90
+    ├── m_vars_global.f90
+    └── prepbufr_gen.f90
    +
    +
    +
    + +
    +
    +Important +
    +
    +
    +

    Once again, this routines are based on the INPE/CPTEC modules and programs that are part of their operational assimilation system.

    +
    +
    + + + +
    + +

    References

    +
    +Garcia, Fernando, Juan Ruiz, Paola Salio, Hernan Bechis, and Steve Nesbitt. 2019. “Argentina Mesonet Data. Version 1.1. UCAR/NCAR - Earth Observing Laboratory.” https://doi.org/10.26023/JEXB-W6B6-E310. +
    +
    + +
    + + + + + + \ No newline at end of file diff --git a/content/observations/03-rad-bufr.html b/content/observations/03-rad-bufr.html new file mode 100644 index 0000000..a3f9f6d --- /dev/null +++ b/content/observations/03-rad-bufr.html @@ -0,0 +1,621 @@ + + + + + + + + + +DA-documentation - Radiance obs bufr + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    +
    + +
    + +
    + + + + +
    + +
    +
    +

    Radiance obs bufr

    +
    + + + +
    + + + + +
    + + + +
    + + +

    bufr table for abi

    +

    cloud mask

    +

    routine

    + + + +
    + +
    + + + + + + \ No newline at end of file diff --git a/content/observations/img/add_obs.png b/content/observations/img/add_obs.png new file mode 100644 index 0000000..b880522 Binary files /dev/null and b/content/observations/img/add_obs.png differ diff --git a/content/observations/img/ema.png b/content/observations/img/ema.png new file mode 100644 index 0000000..d5c6b01 Binary files /dev/null and b/content/observations/img/ema.png differ diff --git a/content/observations/img/filter_obs.png b/content/observations/img/filter_obs.png new file mode 100644 index 0000000..6788e50 Binary files /dev/null and b/content/observations/img/filter_obs.png differ diff --git a/contribute.html b/contribute.html index e786a19..a26b596 100644 --- a/contribute.html +++ b/contribute.html @@ -131,6 +131,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/contributors.html b/contributors.html index ca46812..678717b 100644 --- a/contributors.html +++ b/contributors.html @@ -131,6 +131,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/index.html b/index.html index 2bfc9d2..9d00870 100644 --- a/index.html +++ b/index.html @@ -131,6 +131,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/license.html b/license.html index 8f71024..837a4fb 100644 --- a/license.html +++ b/license.html @@ -131,6 +131,14 @@
  • Working with bufr files +
    • +
  • + + Conventional obs bufr +
    • +
  • + + Radiance obs bufr
    • diff --git a/search.json b/search.json index e15b1d7..4c76883 100644 --- a/search.json +++ b/search.json @@ -20,6 +20,27 @@ "section": "", "text": "Contributors to this guide will appear here." }, + { + "objectID": "content/observations/01-bufr.html", + "href": "content/observations/01-bufr.html", + "title": "Working with bufr files", + "section": "", + "text": "The Binary Universal Form for the Representation of meteorological data (BUFR) is a binary data format maintained by the World Meteorological Organization (WMO). According to Wikipedia BUFR was created in 1988 with the goal of replacing the WMO’s dozens of character-based, position-driven meteorological codes, such as SYNOP (surface observations), TEMP (upper air soundings) and CLIMAT (monthly climatological data). BUFR was designed to be portable, compact, and universal. BUFR belongs to the category of table-driven code forms, this mean that the information is codified using specific tables previously defined.\nNCEP converts and archives all observational data received into a BUFR tank and provides several kinds of BUFR files for its global and regional numerical weather forecast systems. These BUFR files are used by GSI as the standard data sources and can be download from Global Data Assimilation System (GDAS) webpage.\nA BUFR file is divided into one or more messages, each message containing one or more subsets and each subset containing one or more data values.\nA message is a continuous binary stream that can be divided into 6 sections:" + }, + { + "objectID": "content/observations/01-bufr.html#how-to-interpret-the-information-in-a-message", + "href": "content/observations/01-bufr.html#how-to-interpret-the-information-in-a-message", + "title": "Working with bufr files", + "section": "How to interpret the information in a message", + "text": "How to interpret the information in a message\nSection 1 of each message with include the relevant information associated to the data: data type and observation date and time. However, the information is encoded using mnemonics. To decode the information in a BUFR file we will need the table that was used to create the file. In the case of prepBUFR files the table has different sections (check the complete table here). In general this tables includes the mnemonics and a short description. The first part of the prepBUFR table includes the mnemonics associated to the data type:\n.------------------------------------------------------------------------------.\n| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- |\n|------------------------------------------------------------------------------|\n| MNEMONIC | NUMBER | DESCRIPTION |\n|----------|--------|----------------------------------------------------------|\n| | | |\n| ADPUPA | A48102 | UPPER-AIR (RAOB, PIBAL, RECCO, DROPS) REPORTS |\n| AIRCAR | A48103 | MDCRS ACARS AIRCRAFT REPORTS |\n| AIRCFT | A48104 | AIREP, PIREP, AMDAR, TAMDAR AIRCRAFT REPORTS |\nAs an example, let say we get the message type ADPUPA. The first part of the table tell us that this message includes upper-air observations. In a following section of the table we find the structure of this type of messages:\n|------------------------------------------------------------------------------|\n| MNEMONIC | SEQUENCE |\n|----------|-------------------------------------------------------------------|\n| | |\n| ADPUPA | HEADR SIRC {PRSLEVEL} <SST_INFO> <PREWXSEQ> {CLOUDSEQ} |\n| ADPUPA | <CLOU2SEQ> <SWINDSEQ> <AFIC_SEQ> <TURB3SEQ> |\nThis tell us that the message has a header (HEADR - report header sequence) among other things. When we search for HEADR inside the table, we will find the content and it definition:\n| HEADR | SID 207003 XOB YOB 207000 DHR ELV TYP T29 TSB ITP SQN |\n| HEADR | PROCN RPT TCOR <RSRD_SEQ> \nAgain, we will need to look each mnemonic to understand their meaning:\n\nSID: STATION IDENTIFICATION\nXOB: LONGITUDE (DEG E)\nYOB: LATITUDE (DEG N)\nDHR: OBSERVATION TIME MINUS CYCLE TIME (HOURS)\nELV: STATION ELEVATION (METERS)\nTYP: PREPBUFR REPORT TYPE\nT29: DATA DUMP REPORT TYPE\nTSB: REPORT SUBTYPE (HAS VARIOUS MEANINGS DEPENDING ON TYPE)\nITP: INSTRUMENT TYPE\nSQN: REPORT SEQUENCE NUMBER\nPROCN: PROCESS NUMBER FOR THIS MPI RUN\nRPT: REPORTED OBSERVATION TIME (HOURS)\nTCOR: INDICATOR WHETHER OBS. TIME IN “DHR” WAS CORRECTED\nRSRD_SEQ: RESTRICTIONS ON REDISTRIBUTION SEQUENCE\n\nEach of this mnemonics are described in an specific section:\n|------------------------------------------------------------------------------|\n| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------|\n|----------|------|-------------|-----|--------------------------|-------------|\n| | | | | |-------------|\n| ACID | 0 | 0 | 64 | CCITT IA5 |-------------|\n| SAID | 0 | 0 | 10 | CODE TABLE |-------------|\n| SID | 0 | 0 | 64 | CCITT IA5 |-------------|\n| SIRC | 0 | 0 | 4 | CODE TABLE |-------------|\n| MSST | 0 | 0 | 3 | CODE TABLE |-------------|\n| ITP | 0 | 0 | 8 | CODE TABLE |-------------|\n| RPT | 5 | 0 | 22 | HOURS |-------------|\n| DHR | 5 | -2400000 | 23 | HOURS |-------------|\nHalf of the information included in the header is not useful from the assimilation point of view but it is good to know how it works. For this example we have the {PRSLEVEL} sequence included. This is the “PRESSURE LEVEL SEQUENCE” for upper-air observations and includes humidity, temperature and wind observations at each pressure level.\n| PRSLEVEL | CAT <P___INFO> <Q___INFO> <T___INFO> <Z___INFO> <W___INFO> |\n| PRSLEVEL | <DRFTINFO> [W1_EVENT] \nA surface station (ADPSFC) will have a different structure but the idea is the same, everything can be decode from the table." + }, + { + "objectID": "content/observations/01-bufr.html#an-introduction-on-how-to-read-and-write-bufr-files", + "href": "content/observations/01-bufr.html#an-introduction-on-how-to-read-and-write-bufr-files", + "title": "Working with bufr files", + "section": "An introduction on how to read and write BUFR files", + "text": "An introduction on how to read and write BUFR files\nWorking with BUFR files requires to work with fortran routines. Fortunately there is a library that contains subroutines, functions and other utilities that can be used to read (decode) and write (encode) data in BUFR: NCEPLIBS-bufr. This library is compiled during the compilation of GSI but can also be used independently.\nHere is a simplified example of a routine to decode a bufr file:\nopen(unit_in,file='sample.bufr',action='read',form='unformatted')\ncall openbf(unit_in,'IN',unit_in)\n \n msg_report: do while (ireadmg(unit_in,subset,idate) == 0)\n \n sb_report: do while (ireadsb(unit_in) == 0)\n\n call ufbint(unit_in,hdr,3,1 ,iret,hdstr)\n call ufbint(unit_in,obs,1,10,iret,obstr)\n\n enddo sb_report\n \n enddo msg_report\n\ncall closbf(unit_in)\n\nThe first level of code open and close the file. It uses the openbf() and closebf()functions from the BUFR library.\nThe second level will read each message. Each loop reads in one message until the last message in the file is reached. It uses the ireadmg() function.\nThe third level will read the reports inside each message using the ireadsb().\nAnd finally inside each report, we access the data values using the ufbint() function. In this example we save the header in formation in one array and the observation information in a second array.\n\nGSI includes some example routines as part ad the bufr tools kit. A complete routine to decode a BUFR file can be found here. The routine to encode a BUFR file will have similar characteristics.\nAnd again, BUFR files will be different for different type of observations and you always need the associated mnemonics table.\n\nDecode prepBUFR files\nThis is not different that we’ve seen previously and the BUFR tools fonder in the GSI code has very good examples to use. But I like to have the data in tidy tables, so I’ve modified that code to decode a prepBUFR file into a table where each row is an observation. This is the routine:\nprogram prepbufr_decode_all_df\n!\n! read all observations out from prepbufr. \n! read bufr table from prepbufr file\n! output in a \"data frameish\" type \n!\n implicit none\n\n integer, parameter :: mxmn=35, mxlv=250\n character(80):: hdstr='SID XOB YOB DHR TYP ELV SAID T29'\n character(80):: obstr='POB QOB TOB ZOB UOB VOB PWO CAT PRSS'\n character(80):: qcstr='PQM QQM TQM ZQM WQM NUL PWQ '\n character(80):: oestr='POE QOE TOE NUL WOE NUL PWE '\n real(8) :: hdr(mxmn),obs(mxmn,mxlv),qcf(mxmn,mxlv),oer(mxmn,mxlv)\n\n INTEGER :: ireadmg,ireadsb\n\n character(8) :: subset\n integer :: unit_in=10,idate,nmsg,ntb\n\n character(8) :: c_sid\n real(8) :: rstation_id\n equivalence(rstation_id,c_sid)\n\n integer :: i,k,iret\n!\n!\n open(24,file='prepbufr.table')\n open(unit_in,file='prepbufr',form='unformatted',status='old')\n call openbf(unit_in,'IN',unit_in)\n call dxdump(unit_in,24)\n call datelen(10)\n nmsg=0\n msg_report: do while (ireadmg(unit_in,subset,idate) == 0)\n nmsg=nmsg+1\n ntb = 0\n\n sb_report: do while (ireadsb(unit_in) == 0)\n ntb = ntb+1\n !write(*,*)'New secction: ',ntb,' in msg ',nmsg\n call ufbint(unit_in,hdr,mxmn,1 ,iret,hdstr)\n call ufbint(unit_in,obs,mxmn,mxlv,iret,obstr)\n call ufbint(unit_in,oer,mxmn,mxlv,iret,oestr)\n call ufbint(unit_in,qcf,mxmn,mxlv,iret,qcstr)\n rstation_id=hdr(1)\n !write(*,*)\n !write(*,'(a14,8f14.1)') c_sid,(hdr(i),i=2,8)\n DO k=1,iret\n write(*,'(a6,i12,a14,6f17.3,9f17.3,7f17.3)') subset,idate,c_sid,(hdr(i),i=2,6),(obs(i,k),i=1,9),(qcf(i,k),i=1,7)\n ENDDO\n enddo sb_report\n \n enddo msg_report\n call closbf(unit_in)\n\nend program\n\n\n\n\n\n\nImportant\n\n\n\nThere are excellent resources for learning how to work with BUFR files from which I have taken many things mentioned here:\n\nBUFR/PrepBUFR User’s Guide from the Developmental Testbed Center.\nBUFR Reference Manual from the ECMWF." + }, { "objectID": "content/gsi/05-tutorial.html", "href": "content/gsi/05-tutorial.html", @@ -189,25 +210,39 @@ "text": "Footnotes\n\n\nThis files need to be downloaded separately as they are to big to be part of the GSI repository. Also the coefficient files can be updated with better approximations over time.↩︎" }, { - "objectID": "content/observations/01-bufr.html", - "href": "content/observations/01-bufr.html", - "title": "Working with bufr files", + "objectID": "content/observations/03-rad-bufr.html", + "href": "content/observations/03-rad-bufr.html", + "title": "Radiance obs bufr", "section": "", - "text": "The Binary Universal Form for the Representation of meteorological data (BUFR) is a binary data format maintained by the World Meteorological Organization (WMO). According to Wikipedia BUFR was created in 1988 with the goal of replacing the WMO’s dozens of character-based, position-driven meteorological codes, such as SYNOP (surface observations), TEMP (upper air soundings) and CLIMAT (monthly climatological data). BUFR was designed to be portable, compact, and universal. BUFR belongs to the category of table-driven code forms, this mean that the information is codified using specific tables previously defined.\nNCEP converts and archives all observational data received into a BUFR tank and provides several kinds of BUFR files for its global and regional numerical weather forecast systems. These BUFR files are used by GSI as the standard data sources and can be download from Global Data Assimilation System (GDAS) webpage.\nA BUFR file is divided into one or more messages, each message containing one or more subsets and each subset containing one or more data values.\nA message is a continuous binary stream that can be divided into 6 sections:" + "text": "bufr table for abi\ncloud mask\nroutine" }, { - "objectID": "content/observations/01-bufr.html#how-to-interpret-the-information-in-a-message", - "href": "content/observations/01-bufr.html#how-to-interpret-the-information-in-a-message", - "title": "Working with bufr files", - "section": "How to interpret the information in a message", - "text": "How to interpret the information in a message\nSection 1 of each message with include the relevant information associated to the data: data type and observation date and time. However, the information is encoded using mnemonics. To decode the information in a BUFR file we will need the table that was used to create the file. In the case of prepBUFR files the table has different sections (check the complete table here). In general this tables includes the mnemonics and a short description. The first part of the prepBUFR table includes the mnemonics associated to the data type:\n.------------------------------------------------------------------------------.\n| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- |\n|------------------------------------------------------------------------------|\n| MNEMONIC | NUMBER | DESCRIPTION |\n|----------|--------|----------------------------------------------------------|\n| | | |\n| ADPUPA | A48102 | UPPER-AIR (RAOB, PIBAL, RECCO, DROPS) REPORTS |\n| AIRCAR | A48103 | MDCRS ACARS AIRCRAFT REPORTS |\n| AIRCFT | A48104 | AIREP, PIREP, AMDAR, TAMDAR AIRCRAFT REPORTS |\nAs an example, let say we get the message type ADPUPA. The first part of the table tell us that this message includes upper-air observations. In a following section of the table we find the structure of this type of messages:\n|------------------------------------------------------------------------------|\n| MNEMONIC | SEQUENCE |\n|----------|-------------------------------------------------------------------|\n| | |\n| ADPUPA | HEADR SIRC {PRSLEVEL} <SST_INFO> <PREWXSEQ> {CLOUDSEQ} |\n| ADPUPA | <CLOU2SEQ> <SWINDSEQ> <AFIC_SEQ> <TURB3SEQ> |\nThis tell us that the message has a header (HEADR - report header sequence) among other things. When we search for HEADR inside the table, we will find the content and it definition:\n| HEADR | SID 207003 XOB YOB 207000 DHR ELV TYP T29 TSB ITP SQN |\n| HEADR | PROCN RPT TCOR <RSRD_SEQ> \nAgain, we will need to look each mnemonic to understand their meaning:\n\nSID: STATION IDENTIFICATION\nXOB: LONGITUDE (DEG E)\nYOB: LATITUDE (DEG N)\nDHR: OBSERVATION TIME MINUS CYCLE TIME (HOURS)\nELV: STATION ELEVATION (METERS)\nTYP: PREPBUFR REPORT TYPE\nT29: DATA DUMP REPORT TYPE\nTSB: REPORT SUBTYPE (HAS VARIOUS MEANINGS DEPENDING ON TYPE)\nITP: INSTRUMENT TYPE\nSQN: REPORT SEQUENCE NUMBER\nPROCN: PROCESS NUMBER FOR THIS MPI RUN\nRPT: REPORTED OBSERVATION TIME (HOURS)\nTCOR: INDICATOR WHETHER OBS. TIME IN “DHR” WAS CORRECTED\nRSRD_SEQ: RESTRICTIONS ON REDISTRIBUTION SEQUENCE\n\nEach of this mnemonics are described in an specific section:\n|------------------------------------------------------------------------------|\n| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------|\n|----------|------|-------------|-----|--------------------------|-------------|\n| | | | | |-------------|\n| ACID | 0 | 0 | 64 | CCITT IA5 |-------------|\n| SAID | 0 | 0 | 10 | CODE TABLE |-------------|\n| SID | 0 | 0 | 64 | CCITT IA5 |-------------|\n| SIRC | 0 | 0 | 4 | CODE TABLE |-------------|\n| MSST | 0 | 0 | 3 | CODE TABLE |-------------|\n| ITP | 0 | 0 | 8 | CODE TABLE |-------------|\n| RPT | 5 | 0 | 22 | HOURS |-------------|\n| DHR | 5 | -2400000 | 23 | HOURS |-------------|\nHalf of the information included in the header is not useful from the assimilation point of view but it is good to know how it works. For this example we have the {PRSLEVEL} sequence included. This is the “PRESSURE LEVEL SEQUENCE” for upper-air observations and includes humidity, temperature and wind observations at each pressure level.\n| PRSLEVEL | CAT <P___INFO> <Q___INFO> <T___INFO> <Z___INFO> <W___INFO> |\n| PRSLEVEL | <DRFTINFO> [W1_EVENT] \nA surface station (ADPSFC) will have a different structure but the idea is the same, everything can be decode from the table." + "objectID": "content/observations/02-conv-bufr.html", + "href": "content/observations/02-conv-bufr.html", + "title": "Conventional obs bufr", + "section": "", + "text": "This section covers how to add new surface observations to prepBUFR and how to divide a file into smaller subsets.\nOne of the first challenges (besides to learning how to use GSI and deal with bufr files) was to incorporate surface observations from automatic stations to the assimilation. In Argentina there are very few official stations to cover the entire country but there are private automatic station network that could potentially help to improve weather forecasts thank to their higher frequency (10 minutes) and spacial resolution.\nThis requires to incorporate these new observations (available at Garcia et al. 2019) to the prepBUFR (so, be able to write fortran code and understand the bufr format). The 2 programs included here are based on the INPE/CPTEC modules and programs that are part of their operational assimilation system. I was lucky to visit CPTEC in 2019 where I learned how to work with these tools." }, { - "objectID": "content/observations/01-bufr.html#an-introduction-on-how-to-read-and-write-bufr-files", - "href": "content/observations/01-bufr.html#an-introduction-on-how-to-read-and-write-bufr-files", - "title": "Working with bufr files", - "section": "An introduction on how to read and write BUFR files", - "text": "An introduction on how to read and write BUFR files\nWorking with BUFR files requires to work with fortran routines. Fortunately there is a library that contains subroutines, functions and other utilities that can be used to read (decode) and write (encode) data in BUFR: NCEPLIBS-bufr. This library is compiled during the compilation of GSI but can also be used independently.\nHere is a simplified example of a routine to decode a bufr file:\nopen(unit_in,file='sample.bufr',action='read',form='unformatted')\ncall openbf(unit_in,'IN',unit_in)\n \n msg_report: do while (ireadmg(unit_in,subset,idate) == 0)\n \n sb_report: do while (ireadsb(unit_in) == 0)\n\n call ufbint(unit_in,hdr,3,1 ,iret,hdstr)\n call ufbint(unit_in,obs,1,10,iret,obstr)\n\n enddo sb_report\n \n enddo msg_report\n\ncall closbf(unit_in)\n\nThe first level of code open and close the file. It uses the openbf() and closebf()functions from the BUFR library.\nThe second level will read each message. Each loop reads in one message until the last message in the file is reached. It uses the ireadmg() function.\nThe third level will read the reports inside each message using the ireadsb().\nAnd finally inside each report, we access the data values using the ufbint() function. In this example we save the header in formation in one array and the observation information in a second array.\n\nGSI includes some example routines as part ad the bufr tools kit. A complete routine to decode a BUFR file can be found here. The routine to encode a BUFR file will have similar characteristics.\nAnd again, BUFR files will be different for different type of observations and you always need the associated mnemonics table.\n\nDecode prepBUFR files\nThis is not different that we’ve seen previously and the BUFR tools fonder in the GSI code has very good examples to use. But I like to have the data in tidy tables, so I’ve modified that code to decode a prepBUFR file into a table where each row is an observation. This is the routine:\nprogram prepbufr_decode_all_df\n!\n! read all observations out from prepbufr. \n! read bufr table from prepbufr file\n! output in a \"data frameish\" type \n!\n implicit none\n\n integer, parameter :: mxmn=35, mxlv=250\n character(80):: hdstr='SID XOB YOB DHR TYP ELV SAID T29'\n character(80):: obstr='POB QOB TOB ZOB UOB VOB PWO CAT PRSS'\n character(80):: qcstr='PQM QQM TQM ZQM WQM NUL PWQ '\n character(80):: oestr='POE QOE TOE NUL WOE NUL PWE '\n real(8) :: hdr(mxmn),obs(mxmn,mxlv),qcf(mxmn,mxlv),oer(mxmn,mxlv)\n\n INTEGER :: ireadmg,ireadsb\n\n character(8) :: subset\n integer :: unit_in=10,idate,nmsg,ntb\n\n character(8) :: c_sid\n real(8) :: rstation_id\n equivalence(rstation_id,c_sid)\n\n integer :: i,k,iret\n!\n!\n open(24,file='prepbufr.table')\n open(unit_in,file='prepbufr',form='unformatted',status='old')\n call openbf(unit_in,'IN',unit_in)\n call dxdump(unit_in,24)\n call datelen(10)\n nmsg=0\n msg_report: do while (ireadmg(unit_in,subset,idate) == 0)\n nmsg=nmsg+1\n ntb = 0\n\n sb_report: do while (ireadsb(unit_in) == 0)\n ntb = ntb+1\n !write(*,*)'New secction: ',ntb,' in msg ',nmsg\n call ufbint(unit_in,hdr,mxmn,1 ,iret,hdstr)\n call ufbint(unit_in,obs,mxmn,mxlv,iret,obstr)\n call ufbint(unit_in,oer,mxmn,mxlv,iret,oestr)\n call ufbint(unit_in,qcf,mxmn,mxlv,iret,qcstr)\n rstation_id=hdr(1)\n !write(*,*)\n !write(*,'(a14,8f14.1)') c_sid,(hdr(i),i=2,8)\n DO k=1,iret\n write(*,'(a6,i12,a14,6f17.3,9f17.3,7f17.3)') subset,idate,c_sid,(hdr(i),i=2,6),(obs(i,k),i=1,9),(qcf(i,k),i=1,7)\n ENDDO\n enddo sb_report\n \n enddo msg_report\n call closbf(unit_in)\n\nend program\n\n\n\n\n\n\nImportant\n\n\n\nThere are excellent resources for learning how to work with BUFR files from which I have taken many things mentioned here:\n\nBUFR/PrepBUFR User’s Guide from the Developmental Testbed Center.\nBUFR Reference Manual from the ECMWF." + "objectID": "content/observations/02-conv-bufr.html#add-observations-to-a-prepbufr-file", + "href": "content/observations/02-conv-bufr.html#add-observations-to-a-prepbufr-file", + "title": "Conventional obs bufr", + "section": "Add observations to a prepBUFR file", + "text": "Add observations to a prepBUFR file\nIn the previous section I briefly explained how to read and write a bufr file. To add new observations to an existing file (o to a new file!) we use the same functions mentioned here and structure of the routine is also similar. But we also need to figure out how to read the new observations and how to create new reports.\nThe code I’m sharing here assume that the new observations are in csv format and has the following look:\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nLAT\nLON\nELV\nSTATION\nyyyymmddhhmmss\nTOB\nQOB\nUOB\nVOB\nPRSS\n\n\n\n\n-19.0124\n-65.2921\n2907\n“BLV102”\n20181111200000\n28.9\n0.00624936719941647\n0\n-2.88088888888889\n71950\n\n\n-17.4111\n-66.1744\n2548\n“BLV124”\n20181111200000\n29\n0.00636021217641851\n-3.27390439689371\n-3.27390439689372\n74280\n\n\n-16.8378\n-64.7925\n195\n“BLV141”\n20181111200000\n32.9\n0.0439748860789023\n-1.34593847427853\n-1.34593847427853\n50240\n\n\n-17.9747\n-67.08\n4057\n“BLV154”\n20181111201500\n19.2\n0.00317768840091676\n-6.08364406830543\n-2.51992788174274\n62030\n\n\n-17.6364\n-67.2003\n3798\n“BLV157”\n120181111200000\n19\n0.00626217287136847\n-5.34737718159307\n-5.34737718159307\n64460\n\n\n\n\n\n\n\n\n\nImportant\n\n\n\nIf you want to convert the files available here into csv tables, here is an R script to do that.\n\n\nThe program read the configuration file that define where is the namelist and the prepbufr table and call a function that start the process. The namelist will list the name of the prepbufr to modify and the csv files with the new observations. The most important subroutine is adpsfc_rw_prepbufr():\n\nReads the bufr file\nReads the csv with new observations\nDecides if the observations fall into the time window\nCreates a new report type 187\nWrite the bufr file\n\n\n\n\nadd_obs rutine flowchart" + }, + { + "objectID": "content/observations/02-conv-bufr.html#filter-observations-in-a-prepbufr", + "href": "content/observations/02-conv-bufr.html#filter-observations-in-a-prepbufr", + "title": "Conventional obs bufr", + "section": "Filter observations in a prepBUFR", + "text": "Filter observations in a prepBUFR\nWhen I started using GSI I thought that the bufr files needed to have the observations for the assimilation window only, in other words, that GSI was not capable of filtering and ignoring the observations outside the assimilation window. I know, it does not make sense. Before I realized I was wrong I created a program to read a prepbufr and write a new one only with the observations in a specific time window.\nThe general structure is similar to the previous routine. In this case the read_namelist module will read the namelist and list all the available prepbufrs. The magic then happens inside filter_obs_prepbufr() that loops over each observation and check the difference between the observation time and the analysis time. If that difference is less that half the window, the observation is written in the new prepBUFR file.\n\n\n\nprepBURF generation / filter obs rutine flowchart" + }, + { + "objectID": "content/observations/02-conv-bufr.html#the-code", + "href": "content/observations/02-conv-bufr.html#the-code", + "title": "Conventional obs bufr", + "section": "The code", + "text": "The code\nThe source code is publicly available in this repository. Each source folder includes a compile file as example of how to compile the program. It needs the bufr and w3lib to work with times.\nThe run folder includes all the configuration files, namelists and bufr tables. The can be uses to run the programs using the example observations available in the example_obs folder.\nFinally, if you ever need to do this for many files, the run_bash folder has 2 bash scripts that modify the namelist and run the programs in a loop.\nprepbufr_tools\n├── example_obs\n│   ├── 2018111719.csv\n│   ├── 2018111720.csv\n│   ├── cimap.20181117.t20z.01h.prepbufr.nqc\n│   └── prepbufr.gdas.20181117.t18z.nr.48h\n├── README.md\n├── run\n│   ├── add_obs.conf\n│   ├── namelist_conv.PREPOBS\n│   ├── namelist_conv.PREPRW\n│   ├── prepbufr_gen.conf\n│   └── prepobs_prep.bufrtable\n├── run_bach\n│   ├── run_add_obs.sh\n│   └── run_filter_obs.sh\n├── src_add_obs\n│   ├── compile\n│   ├── m_adpsfc_rw.f90\n│   ├── main_add_obs.f90\n│   ├── m_convobs.f90\n│   ├── m_vars_global.f90\n└── src_filter_obs\n ├── compile\n ├── m_filter_obs.f90\n ├── m_read_namelist.f90\n ├── m_vars_global.f90\n └── prepbufr_gen.f90\n\n\n\n\n\n\nImportant\n\n\n\nOnce again, this routines are based on the INPE/CPTEC modules and programs that are part of their operational assimilation system." }, { "objectID": "contribute.html", diff --git a/sitemap.xml b/sitemap.xml index 11c008e..6cbdeb8 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,46 +2,54 @@ https://paocorrales.github.io/DA-documentation/about.html - 2024-02-01T23:47:22.957Z + 2024-02-06T00:19:59.503Z https://paocorrales.github.io/DA-documentation/contributors.html - 2024-02-01T23:47:22.965Z + 2024-02-06T00:19:59.515Z + + + https://paocorrales.github.io/DA-documentation/content/observations/01-bufr.html + 2024-02-06T00:19:59.511Z https://paocorrales.github.io/DA-documentation/content/gsi/05-tutorial.html - 2024-02-01T23:47:22.961Z + 2024-02-06T00:19:59.507Z https://paocorrales.github.io/DA-documentation/content/gsi/03-radiances.html - 2024-02-01T23:47:22.957Z + 2024-02-06T00:19:59.507Z https://paocorrales.github.io/DA-documentation/content/gsi/02-convencionals.html - 2024-02-01T23:47:22.957Z + 2024-02-06T00:19:59.507Z https://paocorrales.github.io/DA-documentation/index.html - 2024-02-01T23:47:22.965Z + 2024-02-06T00:19:59.515Z https://paocorrales.github.io/DA-documentation/license.html - 2024-02-01T23:47:22.965Z + 2024-02-06T00:19:59.515Z https://paocorrales.github.io/DA-documentation/content/gsi/04-diagfiles.html - 2024-02-01T23:47:22.961Z + 2024-02-06T00:19:59.507Z https://paocorrales.github.io/DA-documentation/content/gsi/01-gsi.html - 2024-02-01T23:47:22.957Z + 2024-02-06T00:19:59.507Z - https://paocorrales.github.io/DA-documentation/content/observations/01-bufr.html - 2024-02-01T23:47:22.961Z + https://paocorrales.github.io/DA-documentation/content/observations/03-rad-bufr.html + 2024-02-06T00:19:59.511Z + + + https://paocorrales.github.io/DA-documentation/content/observations/02-conv-bufr.html + 2024-02-06T00:19:59.511Z https://paocorrales.github.io/DA-documentation/contribute.html - 2024-02-01T23:47:22.965Z + 2024-02-06T00:19:59.515Z