-
Notifications
You must be signed in to change notification settings - Fork 12
Overview of `make_cube.py`
The make_cube.py script houses two classes: CubeFactory and TicaCubeFactory. Both of these classes serve to generate a cube from a batch of TESS full frame images (FFIs) that correspond to a TESS sector/camera/CCD observation. There are two different productt (FFI) types that you can generate cubes from that are structurally different, which is why there is one class for each FFI type: CubeFactory will allow you to generate a cube from a batch of mission-delivered FFIs (SPOC FFIs) and TicaCubeFactory will allow you to generate a cube from a batch of TESS Image CAlibrator FFIs (TICA FFIs). There are advantages to using each type, which are outlined in the TESSCut API documentation. This wiki page will walk you through the make_cube.py functionality
The CubeFactory class serves to take in a batch of mission-delivered (SPOC) full frame images and generate a 4-dimensional cube with them, which is then used by CutoutFactory to generate a cutout target pixel file for a designated target or set of coordinates. The main method of this class is make_cube, which calls upon the other methods to generate the cube and store it into a FITS file. The order upon which make_cube calls the methods is as follows:
__init___configure_cube_build_info_table_build_cube_file_write_block_write_info_table
The functionality of each of these methods will be discussed briefly in the sections following. Should there be any questions/comments regarding this wiki page, please file an issue under the Issues tab in this repository or contact us through the Archive Help Desk at [email protected].
This is a standard initialization function within a class that instantiates a certain set of variables as part of the class for future calls later in the code. The max_memory argument is the allocated memory (in GB) that can be used when generating the cube. The smaller this number, the longer it will take to generate the cube, but this number will depend on the specifications of your machine. We suggest leaving this at the default 50 GB unless it causes computational errors.
The attributes block_size and num_blocks are dependent on the input max_memory. This is because the FFIs are not read into the cube all at once, but rather in chunks or "blocks". The block size, and therefore the number of blocks, is calculated from the max_memory value. More detail on this will be under _configure_cube, where these attributes are assigned values.
The remaining attributes are things such as keyword headers, or variables that will be called across multiple functions such as cube_file.
This method will determine the block size (a subset of the full FFI) and the number of blocks to iterate through, based on the input max_memory from __init__. The largest possible block size, or max_block_size is determined by taking the max_memory (converted to Bytes from GB) and dividing it by the slice_size, which is the # of columns in the image multiplied by the # of FFIs in the batch * 2 * 4. [VERIFY:] We multiply by 2 because each cube will have 2 "layers" (axes): 1 axis dedicated to the science data and 1 axis dedicated to the error data, and then multiply by 4 because there are approximately 4 bytes per pixel. After this calculation is done, the number of blocks is determined by taking the # of rows in the FFIs and dividing by the slice_size, adding 1, and rounding down to the nearest integer ([VERIFY:] we add 1 because np.int always rounds down).
This is also the method that will assign the template_file variable. This is the FFI whose image header keywords are used as the template to initialize the image header table in the later steps.
Lastly, this method also makes the cube's primary header. To do this, we copy the primary header keywords and keyword values from the first FFI in the stack, to the cube file's primary header. Extra header keywords are added such as the factory keywords, and lastly, some primary header keywords are populated with the keyword values from the last FFI in the stack. These keywords are DATE-END and TSTOP for the SPOC FFIs, and ENDTJD for TICA FFIs.
With the template file that is defined in _configure_cube, this method sets up the columns of the image header table by making a column for each image header keyword in the FFI (and thus all the FFIs) and assigning it the correct data type. If you ever use TicaCubeFactory and receive an Info Warning: Card too long message, this is the piece of code that would be the culprit. There is a keyword in the TICA header called COMMENT which contains a comment on the TICA processing that may have different lengths depending on the FFI. When the template file chosen for that stack contains a COMMENT keyword value string that is shorter than any of the FFIs that follow, this warning message will trigger. It does not appear to clip the COMMENT keyword values, so it is not something to worry about currently.
Once the column objects are defined, they are appended to a cols list which is then turned into an astropy.table.Table object, and assigned to the info_table variable.
This method builds the cube file and writes it into a file with the name designated via the cube_file argument in the call. It writes in the primary header that was created in the _configure_cube step, and initializes the HDU list with just the primary HDU and writes it into the file. It then creates the cube header for the image HDU, first populating the data side of the HDU with an empty cube of size (100, 100, 10, 2), and then populates the header with some keywords defining the dimensions of the cube.
Next, it expands the files size so that it can accommodate all the FFIs that will be part of the cube. To do this, the total cube size (cubesize_in_bytes) must be calculated. The size is calculated by taking the area of any of the FFIs, and multiplying it by 4 which is the estimated byte size of 1 pixel. We add 2880 to this value because FITS requires all blocks to be a multiple of 2880, and then subtract by 1 because [VERIFY:] numpy arrays are zero-indexed. All of that is divided by 2880, and then multiplied by 2880 once again.
The last step in this method is writing a null byte. It takes the total size in bytes of the cube and seek to the end of the file to write a null byte (CUBE.write(b'\0')) because ????
From here we begin the iterating process of writing the FFIs into the cube. The cube created with _build_cube is opened and fed into _write_block along with a few other arguments of interest:
-
start_rowandend_row, which determine the start and end rows that act as the edges of the block, -
cube_hdu, which is the cube created in_build_cube, that is now being populated with the FFIs -
fill_info_table, which is set to the booleanTrue, and triggers that the image header table in the third extension (EXT=2) gets written out, and -
verbose, which will print out updates on the cube-making process when it is set toTrue.
_write_block is called within a for-loop, so it's iterated upon a number of times, depending on the number of blocks that are needed to write out the entirety of each FFI into the cube. As such, the start_row and end_row arguments will change for each iteration. For example: start_row will be 0 for the first iteration, then 0 + size of the first block for the second iteration, and so on, with end_row also being updated accordingly.
This method stores the table into a FITS table. Assigns the correct data type for each column/image header keyword, and builds the table HDU from these columns. The table HDU is then appended to the existing cube file.
The TicaCubeFactory architecture reflects CubeFactory's with some minor but notable differences. In the sections below, we will point out some of the differences that are not as straight-forward. For the ones that are, we will just refer you back to its analogous section under CubeFactory.