- #1
DrFurious
- 23
- 0
Hello all,
I have a question concerning the best way to access random elements in a large file. I have a single data file of numbers, 12G in size with dimensions 1024x768x16384. My goal is to, as quickly as possible, load, say 16384 of those numbers into memory. Assume these numbers are not aligned on the disk.
What is the fastest way of doing this? I've tried using HDF5, lseek-ing with my own program in C, etc, but the best I can do is about 3 minutes for this amount. Does anyone have ideas on the fastest approach?
I have a question concerning the best way to access random elements in a large file. I have a single data file of numbers, 12G in size with dimensions 1024x768x16384. My goal is to, as quickly as possible, load, say 16384 of those numbers into memory. Assume these numbers are not aligned on the disk.
What is the fastest way of doing this? I've tried using HDF5, lseek-ing with my own program in C, etc, but the best I can do is about 3 minutes for this amount. Does anyone have ideas on the fastest approach?