BIL - A Block I/O Layer for Analysis and Visualization Applications
Please click on DOWNLOAD to get the full source, released under the LGPL license.
2. SynopsisAs a team effort by SciDAC Ultravis, we have produced an easy to use library called BIL (Block I/O Layer). The BIL library is a wrapper around parallel I/O systems, and implements the current "best practice" of using collective methods to flexibly access scientific datasets sized at terabytes or more. BIL is specifically designed for one general use case -- situations where the user wishes to load a distributed array (blocks) of their dataset across processes. The dataset can be any dimensionality, a single process can request multiple blocks from the same file, and the blocks may even be stored across multiple files. In comparison with parallel data access using conventional POSIX I/O or naive methods with MPI-I/O, BIL can often deliver orders of magnitude performance increases, and thus greatly benefit parallel visualization of 4D multivariate data. Reading of raw and netCDF file format is supported in the current release and HDFx formats will be included in future releases. Writing data will also be supported in future releases as well.
3. Using BILA short manual about BIL and the API can be downloaded here.
For advanced implementation details, comments, questions, and bug reporting, please email Wes Kendall at kendall AT eecs dot utk dot edu.
If you plan to use this interface in your work, we would love to hear about it. We also ask that you would reference "Terascale Data Organization for Discovering Multivariate Climatic Trends", Wesley Kendall, Markus Glatter, Jian Huang, Tom Peterka, Rob Latham, and Rob Ross, Proceedings of Supercomputing '09, Portland OR, Nov. 2009, a paper that used the initial workings of the BIL interface on over a terabyte of data.