Programs need to update files. Although most programmers know that unexpected things can happen while performing I\/O, I often see code that has been written in a surprisingly na\u00efve way. In this article, I would like to share some insights on how to improve I\/O reliability in Python code.<\/p>\n
Consider the following Python snippet. Some operation is performed on data coming from and going back into a file:<\/p>\n
with open(filename) as f:\r\n input = f.read()\r\noutput = do_something(input)\r\nwith open(filename, 'w') as f:\r\n f.write(output)<\/pre>\n<\/div>\n<\/div>\nPretty simple? Probably not as simple as it looks at the first glance. I often debug applications that show strange behaviour on production servers. Here are examples of failure modes I have seen:<\/p>\n
Nothing of what follows is really new. My goal is to present common approaches and techniques to Python developers who are less experienced in system programming. I will provide code examples to make it easy for developers to incorporate these approaches into their own code.<\/p>\n
In the broadest sense, reliability means that an operation is performing its required function under all stated conditions. With regard to file updates, the function in question is to create, replace or extend the contents of a file. It might be rewarding to seek inspiration from database theory here. The ACID<\/a> properties of the classic transaction model will serve as guidelines to improve reliability.<\/p>\n To get started, let\u2019s see how the initial example can be rated against the four ACID properties:<\/p>\n If we would be able to gain all four ACID properties, we would have come a long way towards increased reliability. But this requires significant coding effort. Why reinvent the wheel? Most database systems already have ACID transactions.<\/p>\n Reliable data storage is a solved problem. If you need reliable storage, use a database.<\/strong> Chances are high that you will not do it by yourself as good as those who have been working on it for years if not decades. If you do not want to set up a \u201cbig\u201d database server, you can use sqlite<\/a> for example. It has ACID transactions, it\u2019s small, it\u2019s free, and it\u2019s included in Python\u2019s standard library<\/a>.<\/p>\n The article could finish here. But there are valid reasons not to use a database. They are often tied to file format<\/em> or file location<\/em> constraints. Both are not easily controllable with database systems. Reasons include:<\/p>\n …and so on. You get the point.<\/p>\n If we are set out to implement reliable file updates on our own, there are some programming techniques to consider. In the following, I will present four common patterns of performing file updates. After that, I will discuss what steps can be taken to establish ACID properties with each file update pattern.<\/p>\n<\/div>\n Files can be updated in a multitude of ways, but I see at least four common patterns. These will serve as a basis for the rest of this article.<\/p>\n This is probably the most basic pattern. In the following example, hypothetical domain model code reads data, performs some computation, and re-opens the existing file in write mode:<\/p>\n A variant of this pattern opens the file in read-write mode (the \u201cplus\u201d modes in Python), seeks to the start, issues an explicit truncate()<\/cite> call and rewrites the contents:<\/p>\n An advantage of this variant is that we open file only once and keep it open all the time. This simplifies locking for example.<\/p>\n<\/div>\n Another widely used pattern is to write new contents into a temporary file and replace the original file after that:<\/p>\n This method is more robust against errors than the truncate-write<\/em> method. See below for a discussion of atomicity and consistency properties. It is used by many applications.<\/p>\n These first two patterns are so common that the ext4 filesystem in the Linux kernel even detects them<\/a> and fixes some reliability shortcomings automatically. But don\u2019t depend on it: you are not always using ext4, and the administrator might have disabled this feature.<\/p>\n<\/div>\n The third pattern is to append new data to an existing file:<\/p>\n This pattern is used for writing log files and other cumulative data processing tasks. Technically, its outstanding feature is its extreme simplicity. An interesting extension is to perform append-only updates during regular operation and to reorganize the file into a more compact form periodically.<\/p>\n<\/div>\n Here we treat a directory as logical data store and create a new uniquely named file for each record:<\/p>\n This pattern shares its cumulative nature with the append<\/em> pattern. A big advantage is that we can put a little amount of metadata into the file name. This can be used, for example, to convey information about the processing status. A particular clever implementation of the spooldir<\/em> pattern is the maildir<\/a> format. Maildirs use a naming scheme with additional subdirectories to perform update operations in a reliable and lock-free way. The md<\/a> and gocept.filestore<\/a> libraries provide convenient wrappers for maildir operations.<\/p>\n If your file name generation is not guaranteed to give unique results, there is even a possibility to demand that the file must be actually new. Use the low-level os.open()<\/cite> call with proper flags:<\/p>\n After opening the file with O_EXCL<\/cite>, we use os.fdopen<\/cite> to convert the raw file descriptor into a regular Python file object.<\/p>\n<\/div>\n<\/div>\n In the following, I will try to enhance the file update patterns. Let\u2019s see what we can do to meet each ACID property in turn. I will keep this as simple as possible, since we are not planning to write a complete database system. Please note that the material presented in this section is not exhaustive, but it may give you a good starting point for your own experimentation.<\/p>\n The write-replace<\/strong> pattern gives you atomicity for free since the underlying os.rename()<\/cite> function is atomic<\/a>. This means that at any given point in time, any process sees either the old or the new file. This pattern has a natural robustness against write errors: if the write operation triggers an exception, the rename operation is never performed and thus, we are not in the danger of overwriting a good old file with a damaged new one.<\/p>\n The append<\/strong> patterns is not atomic by itself, because we risk to append incomplete records. But there is a trick to make updates appear atomic: Annotate each written record with a checksum. When reading the log later on, discard all records that do not have a valid checksum. This way, only complete records will be processed. In the following example, an application makes periodic measurements and appends a one-line JSON record each time to a log. We compute a CRC32 checksum of the record\u2019s byte representation and append it to the same line:<\/p>\n This example code simulates the measurements by creating a random value every second.<\/p>\n To process the log file, we read one record per line, split off the checksum, and compare it to the read record:<\/p>\n Now we simulate a truncated write by chopping the last line:<\/p>\n When the log is read, the last incomplete line is rejected:<\/p>\n The checksummed log record approach is used by a large number of applications including many database systems.<\/p>\n Individual files in the spooldir<\/strong> can likewise feature a checksum in each file. Another, probably easier, approach is to borrow from the write-replace<\/em> pattern: first write the file aside and move it to its final location afterwards. Devise a naming scheme that protects work-in-progress files from being processed by consumers. In the following example, all file names ending with .tmp<\/tt> are ignored by readers and are thus safe to use during write operations:<\/p>\n At last, truncate-write<\/strong> is non-atomic. I am sorry that I am not able to offer you an atomic variant. Right after performing the truncate operation, the file is nulled and no new content has been written yet. If a concurrent program reads the file now or, worse yet, an exception occurs and our program gets aborted, we see neither the old nor the new version.<\/p>\n<\/div>\n Most things I have said about atomicity can be applied to consistency as well. In fact, atomic updates are a prerequisite for internal consistency. External consistency means to update several files in sync. As this cannot easily be done, lock files can be used to ensure that read and write access do not interfere. Consider a directory where files need to be consistent with each other. A common pattern is to designate a lock file, which controls access for the whole directory.<\/p>\n Example writer code:<\/p>\n Example reader code:<\/p>\n This method only works if we have control over all readers. Since there may be only one writer active at a time (the exclusive lock is blocking all shared locks), the scalability of this method is limited.<\/p>\n To take it one step further, we can apply the write-replace<\/strong> pattern to whole directories. This involves creating a new directory for each update generation<\/em> and changing a symlink once the update is complete. For example, a mirroring application maintains a directory of tarballs together with an index file, which lists file name, file size, and a checksum. When the upstream mirror gets updated, it is not enough to implement an atomic file update for every tarball and the index file in isolation. Instead, we need to flip both the tarballs and the index file at the same time to avoid checksum mismatches. To solve this problem, we maintain a subdirectory for each generation and symlink the active generation:<\/p>\n Here, the new generation 484 is in the process of being updated. When all tarballs are present and the index file is up to date, we can switch the current<\/tt> symlink with a single, atomic os.symlink()<\/cite> call. Other applications see always either the complete old or the complete new generation. It is important that readers need to os.chdir()<\/cite> into the current<\/tt> directory and refer to files without their full path names. Otherwise, there is a race condition when a reader first opens current\/index.json<\/tt> and then opens current\/a.tgz<\/tt>, but in the meanwhile the symlink target has been changed.<\/p>\n<\/div>\n Isolation means that concurrent updates to the same file are serializable<\/em> \u2014 there exists a serial schedule that gives the same results as the parallel schedule actually performed. \u201cReal\u201d database systems use advanced techniques like MVCC<\/a> to maintain serializability while allowing for a great degree of parallelism. Back on our own, we better use locks to serialize file updates.<\/p>\n Locking truncate-write<\/strong> updates is easy. Just acquire an exclusive lock prior to all file operations. The following example code reads an integer from a file, increments it, and updates the file:<\/p>\n Locking updates using the write-replace<\/strong> pattern can be tricky. Using a lock the same way as in truncate-write<\/em> can lead to updates conflicts. A na\u00efve implementation could look like this:<\/p>\n What is wrong with this code? Imagine two processes compete to update a file. The first process just goes ahead, but the second process is blocked in the fcntl.flock()<\/cite> call. When the first process replaces the file and releases the lock, the already open file descriptor in the second process now points to a \u201cghost\u201d file (not reachable by any path name) with old contents. To avoid this conflict, we must check that our open file is still the same after returning from fcntl.flock()<\/cite>. So I have written a new LockedOpen<\/cite> context manager to replace the built-in open<\/cite> context. It ensures that we actually open the right file:<\/p>\n Locking append<\/strong> updates is as easy as locking truncate-write<\/em> updates: acquire an exclusive lock, append, done. Long-running processes, which leave a file permanently open, may need to release locks between updates to let others in.<\/p>\n The spooldir<\/strong> pattern has the elegant property that it does not require any locking. Again, it depends on using a clever naming scheme and a robust unique file name generation. The maildir specification<\/a> is a good example for a spooldir design. It can be easily adapted to other cases, which have nothing to do with mail.<\/p>\n<\/div>\n Durability is a bit special because it depends not only on the application, but also on OS and hardware configuration. In theory, we can assume that os.fsync()<\/cite> or os.fdatasync()<\/cite> calls do not return until data has reached permanent storage. In practice, we may run into several problems: we may be facing incomplete fsync implementations or awkward disk controller configurations, which never give any persistence guarantee. A talk from a MySQL dev<\/a> goes into great detail of what can go wrong. Some database systems like PostgreSQL even offer a choice of persistence mechanisms<\/a> so that the administrator can select the best suited one at runtime. The poor man\u2019s option although is to just use os.fsync()<\/cite> and hope that it has been implemented correctly.<\/p>\n With the truncate-write<\/strong> pattern, we have to issue an fsync after finishing write operations but before closing the file. Note that there is usually another level of write caching involved. The glibc buffer<\/em> holds back writes inside the process even before they are passed to the kernel. To get the glibc buffer empty as well, we have to flush()<\/cite> it before fsync\u2019ing:<\/p>\n Alternatively, you can invoke Python with the -u<\/strong> flag to get unbuffered writes for all file I\/O.<\/p>\n I prefer os.fdatasync()<\/cite> over os.fsync()<\/cite> most of the time to avoid synchronous metadata updates (ownership, size, mtime, …). Metadata updates can result in seeky disk I\/O, which slows things down quite a bit.<\/p>\n Applying the same trick to write-replace<\/strong> style updates is only half of the story. We make sure that the newly written file has been pushed to non-volatile storage before replacing the old file, but what about the replace operation itself? We have no guarantee that the directory update is performed right on. There are lengthy discussions on how to sync a directory update<\/a> on the net, but in our case (old and new file are in the same directory) we can get away with this rather simple solution:<\/p>\n We open the directory with the low-level os.open()<\/cite> call (Python\u2019s built-in open()<\/cite> does not support opening directories) and perform a os.fsync()<\/cite> on the directory\u2019s file descriptor.<\/p>\n Persisting append<\/strong> updates is again quite similar to what I have said about truncate-write<\/em>.<\/p>\n The spooldir<\/strong> pattern has the same directory sync problems as the write-replace<\/em> pattern. Fortunately, the same solution applies here as well: first sync the file, then sync the directory.<\/p>\n<\/div>\n<\/div>\n\n
Use a database system if you can<\/h2>\n
\n
File update patterns<\/h2>\n
Truncate-Write<\/h3>\n
with open(filename, 'r') as f:\r\n model.read(f)\r\nmodel.process()\r\nwith open(filename, 'w') as f:\r\n model.write(f)<\/pre>\n<\/div>\n<\/div>\n
with open(filename, 'a+') as f:\r\n f.seek(0)\r\n model.input(f.read())\r\n model.compute()\r\n f.seek(0)\r\n f.truncate()\r\n f.write(model.output())<\/pre>\n<\/div>\n<\/div>\n
Write-Replace<\/h3>\n
with tempfile.NamedTemporaryFile(\r\n 'w', dir=os.path.dirname(filename), delete=False) as tf:\r\n tf.write(model.output())\r\n tempname = tf.name\r\nos.rename(tempname, filename)<\/pre>\n<\/div>\n<\/div>\n
Append<\/h3>\n
with open(filename, 'a') as f:\r\n f.write(model.output())<\/pre>\n<\/div>\n<\/div>\n
Spooldir<\/h3>\n
with open(unique_filename(), 'w') as f:\r\n f.write(model.output())<\/pre>\n<\/div>\n<\/div>\n
fd = os.open(filename, os.O_WRONLY | os.O_CREAT| os.O_EXCL, 0o666)\r\nwith os.fdopen(fd, 'w') as f:\r\n f.write(...)<\/pre>\n<\/div>\n<\/div>\n
Applying ACID properties to file updates<\/h2>\n
Atomicity<\/h3>\n
with open(logfile, 'ab') as f:\r\n for i in range(3):\r\n measure = {'timestamp': time.time(), 'value': random.random()}\r\n record = json.dumps(measure).encode()\r\n checksum = '{:8x}'.format(zlib.crc32(record)).encode()\r\n f.write(record + b' ' + checksum + b'\\n')<\/pre>\n<\/div>\n<\/div>\n$ cat log\r\n{\"timestamp\": 1373396987.258189, \"value\": 0.9360123151217828} 9495b87a\r\n{\"timestamp\": 1373396987.25825, \"value\": 0.40429005476999424} 149afc22\r\n{\"timestamp\": 1373396987.258291, \"value\": 0.232021160265939} d229d937<\/pre>\n<\/div>\nwith open(logfile, 'rb') as f:\r\n for line in f:\r\n record, checksum = line.strip().rsplit(b' ', 1)\r\n if checksum.decode() == '{:8x}'.format(zlib.crc32(record)):\r\n print('read measure: {}'.format(json.loads(record.decode())))\r\n else:\r\n print('checksum error for record {}'.format(record))<\/pre>\n<\/div>\n<\/div>\n$ cat log\r\n{\"timestamp\": 1373396987.258189, \"value\": 0.9360123151217828} 9495b87a\r\n{\"timestamp\": 1373396987.25825, \"value\": 0.40429005476999424} 149afc22\r\n{\"timestamp\": 1373396987.258291, \"value\": 0.23202<\/pre>\n<\/div>\n$ read_checksummed_log.py log\r\nread measure: {'timestamp': 1373396987.258189, 'value': 0.9360123151217828}\r\nread measure: {'timestamp': 1373396987.25825, 'value': 0.40429005476999424}\r\nchecksum error for record b'{\"timestamp\": 1373396987.258291, \"value\":'<\/pre>\n<\/div>\nnewfile = generate_id()\r\nwith open(newfile + '.tmp', 'w') as f:\r\n f.write(model.output())\r\nos.rename(newfile + '.tmp', newfile)<\/pre>\n<\/div>\n<\/div>\n
Consistency<\/h3>\n
with open(os.path.join(dirname, '.lock'), 'a+') as lockfile:\r\n fcntl.flock(lockfile, fcntl.LOCK_EX)\r\n model.update(dirname)<\/pre>\n<\/div>\n<\/div>\n
with open(os.path.join(dirname, '.lock'), 'a+') as lockfile:\r\n fcntl.flock(lockfile, fcntl.LOCK_SH)\r\n model.readall(dirname)<\/pre>\n<\/div>\n<\/div>\n
mirror\r\n|-- 483\r\n| |-- a.tgz\r\n| |-- b.tgz\r\n| `-- index.json\r\n|-- 484\r\n| |-- a.tgz\r\n| |-- b.tgz\r\n| |-- c.tgz\r\n| `-- index.json\r\n`-- current -> 483<\/pre>\n<\/div>\n
Isolation<\/h3>\n
def update():\r\n with open(filename, 'r+') as f:\r\n fcntl.flock(f, fcntl.LOCK_EX)\r\n n = int(f.read())\r\n n += 1\r\n f.seek(0)\r\n f.truncate()\r\n f.write('{}\\n'.format(n))<\/pre>\n<\/div>\n<\/div>\ndef update():\r\n with open(filename) as f:\r\n fcntl.flock(f, fcntl.LOCK_EX)\r\n n = int(f.read())\r\n n += 1\r\n with tempfile.NamedTemporaryFile(\r\n 'w', dir=os.path.dirname(filename), delete=False) as tf:\r\n tf.write('{}\\n'.format(n))\r\n tempname = tf.name\r\n os.rename(tempname, filename)<\/pre>\n<\/div>\n<\/div>\nclass LockedOpen(object):\r\n\r\n def __init__(self, filename, *args, **kwargs):\r\n self.filename = filename\r\n self.open_args = args\r\n self.open_kwargs = kwargs\r\n self.fileobj = None\r\n\r\n def __enter__(self):\r\n f = open(self.filename, *self.open_args, **self.open_kwargs)\r\n while True:\r\n fcntl.flock(f, fcntl.LOCK_EX)\r\n fnew = open(self.filename, *self.open_args, **self.open_kwargs)\r\n if os.path.sameopenfile(f.fileno(), fnew.fileno()):\r\n fnew.close()\r\n break\r\n else:\r\n f.close()\r\n f = fnew\r\n self.fileobj = f\r\n return f\r\n\r\n def __exit__(self, _exc_type, _exc_value, _traceback):\r\n self.fileobj.close()<\/pre>\n<\/div>\n<\/div>\n
def update(self):\r\n with LockedOpen(filename, 'r+') as f:\r\n n = int(f.read())\r\n n += 1\r\n with tempfile.NamedTemporaryFile(\r\n 'w', dir=os.path.dirname(filename), delete=False) as tf:\r\n tf.write('{}\\n'.format(n))\r\n tempname = tf.name\r\n os.rename(tempname, filename)<\/pre>\n<\/div>\n<\/div>\nDurability<\/h3>\n
with open(filename, 'w') as f:\r\n model.write(f)\r\n f.flush()\r\n os.fdatasync(f)<\/pre>\n<\/div>\n<\/div>\n
os.rename(tempname, filename)\r\ndirfd = os.open(os.path.dirname(filename), os.O_DIRECTORY)\r\nos.fsync(dirfd)\r\nos.close(dirfd)<\/pre>\n<\/div>\n<\/div>\n