Supported Data Formats
Raster formats
The WhiteboxTools library can currently support reading/writing raster data in several common formats.
| Format | Extension | Read | Write |
|---|---|---|---|
| GeoTIFF | *.tif, *.tiff | X | X |
| Big GeoTIFF | *.tif, *.tiff | X | X |
| Esri ASCII | *.txt, *.asc | X | X |
| Esri BIL | *.bil, *.hdr | X | X |
| Esri Binary | *.flt and *.hdr | X | X |
| GRASS ASCII | *.txt, *.asc | X | X |
| Idrisi | *.rdc and *.rst | X | X |
| SAGA Binary | *.sdat and *.sgrd | X | X |
| Surfer ASCII | *.grd | X | X |
| Surfer Binary | *.grd | X | X |
| Whitebox | *.tas and *.dep | X | X |
Throughout this manual code examples that manipulate raster files all use the GeoTIFF format (.tif) but any of the supported file extensions can be used in its place.
The library is primarily tested using Whitebox raster and GeoTIFF data sets and if you encounter issues when reading/writing data in other formats, you should report the issue.
WhiteboxTools is able to read GeoTIFFs compressed using the PackBits, DEFLATE, and LZW methods. Compressed GeoTIFFs, created using the DEFLATE algorithm, can also be output from any tool that generates raster output files by using the --compress_rasters command line flag. If using the Python WhiteboxTools API, you can set the flag by using the set_compress_rasters method (e.g. wbt.set_compress_rasters(True)). The default is False. GeoTIFF output file compression can be set in the WhiteboxTools Runner Multi-band GeoTIFFs are not currently supported.
Please note that there are no plans to incorporate third-party libraries, like GDAL, in the project given the design goal of keeping a pure (or as close as possible) Rust codebase without third-party dependencies. This design greatly simplifies installation of the library but significantly complicates the development by necessitating hand-rolled encoders/decoders for various raster data formats. For more complicated file formats, particularly the ubiquitous GeoTIFF format, it will take time to develop a mature and bug-free encoder/decoder and it is likely that we will only ever be able to support a sub-set of the possible GeoTIFF raster types. It is helpful if users who encounter raster-format related issues report their problems complete with descriptions of the error (What tool was being used? What type of data were input? How far along was the execution before the error was thrown? What was the error message?) and if possible submit a copy of the problematic file.
Vector Formats
At present, there is limited support in WhiteboxTools for working with vector geospatial data formats. The only supported vector format is the ESRI Shapefile. Shapefiles geometries (.shp) and attributes (.dbf) can be read and written. There are plans to support a wider variety of vector file formats in the future.
While the Shapefile format is extremely common, it does have certain limitations for vector representation. For example, owing to their 32-bit indexing, Shapefiles are limited in the number of geometries that can be stored in these files. Furthermore, Shapefiles are incapable of storing geometries of more than one type (point, lines, polygons) within the same file. As such, the vector-related tools in WhiteboxTools also carry these same limitations imposed by the Shapefile format.
Point Cloud (LiDAR) Formats
LiDAR data can be read/written in the common LAS data format. WhiteboxTools also supports the losslessly compressed open-source zLidar data format. zLidar files are often 15-25% of the equivalent LAS file size, making the management of large LiDAR data projects more manageable. WhiteboxTools can also read and write LAS files that have been compressed (zipped with a .zip extension) using the common DEFLATE algorithm. Note that only LAS file should be contained within a zipped archive file. The following is an example of running a LiDAR tool using zipped input/output files:
>>./whitebox_tools -r=LidarTophatTransform -v --wd="/path/to/data/"
-i="input.las.zip" -o="output.las.zip" --radius=10.0
Note that the double extensions (.las.zip) in the above command are not necessary and are only used for convenience of keeping track of LiDAR data sets (i.e. .zip extensions work too). The extra work of decoding/encoding compressed files does add additional processing time, although the Rust compression library that is used is highly efficient and usually only adds a few seconds to tool run times. Zipping LAS files frequently results 40-60% smaller binary files, making the additional processing time worthwhile for larger LAS file data sets with massive storage requirements.
Note: The compressed LiDAR format LAZ and ESRI LiDAR format are not currently supported in WhiteboxTools. The ESRI format is proprietary and closed-source, making it impossible to write an encoder/decoder for this file format. While there is a published paper describing Martin Isenburg's impressive LAZ format, it is not up-to-date with the current data format. Nor does the paper provide as much detailed information regarding the file structure as an open specification document (e.g. the LAS format spec). This makes the open-source code for the LAZ reader/writer the only means of interrogating the current structure of these files, which makes it difficult to natively support LAZ. For further information, see here.