HDF5 for efficient I/O
Fast Containers
© Lead Image © Kirsty Pargeter, 123RF.com
HDF5 is a flexible, self-describing, and portable hierarchical filesystem supported by a number of languages and tools, with the ability to run processes in parallel.
Input/output operations are a very important part of many applications, sometimes involving a huge amount of data and a large number of reads and writes. Therefore, applications can use a very significant portion of their total run time to perform I/O, which becomes critical in Big Data, machine learning, and high-performance computing (HPC).
In a previous article [1], I discussed options for improving I/O performance, focusing on parallel I/O. One of the options mentioned was to use a high-level library to perform the I/O. A great example of such a library is the Hierarchical Data Format (HDF) [2], a standard library used primarily for scientific computing.
In this article, I introduce HDF5 and focus on the concepts and its strengths in performing I/O; then, I look at some simple Python and Fortran code examples, before ending with an example of parallel I/O with HDF5 and Fortran.
[...]
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
System76 Announces the New Thelio Prime Desktop
The makers of the Thelio desktop line and Pop!_OS have released a powerful small form factor PC.
-
A Must See KDE Documentary
It doesn't matter what desktop environment you prefer, this KDE documentary should top your list of things to watch.
-
New KORA Linux Distro to Harness the Power of Local AI
The development team behind CyanogenMod, Android, and Playtron unveils KORA, a Linux distribution built specifically for AI nerds.
-
Epic Games Targets Linux for New Launcher
The Epic Games new launcher is still in preview, but it looks to land on several Linux distributions.
-
Fedora 45 Now Available
The latest iteration of Fedora has arrived with Gnome 51 and KDE Plasma 6.8.
-
Linux 7.4 is "Much Healthier" Thanks to AI
At nearly 42 million lines, Linux 7.4 is the largest merge in the kernel’s history.
-
Working Exploit for a Linux Kernel Flaw Made Public
Even before a bug received its CVE identifier or an upstream fix, the local privilege escalation vulnerability exploit found its way into the wild.
-
r/opensource Bans AI-Assisted and AI-Related Projects
The de facto subreddit for all things open source has not only banned AI-related projects but also users who "don't speak English."
-
Bottlenecks Discovered by Meta Engineers Lead to Scalability Improvements
Thanks to a small group of engineers at Meta, the Linux kernel module kernfs has been improved, which has led to some serious gains.
-
PwnEscape Poses VM Vulnerability
A guest-to-host escape flaw in Linux KVM, named PwnEscape, might be yet another means of attacking Linux.
