Synchronize among files

The file-based synchronization is based on the package fasteners. In this case, the buffer is implemented by shared log files. The synchronization is implemented by the file-based locks. When users need to synchornize the messages from different programs on the same device, it is sutiable to use LineFileBuffer.

In the following example, we provide two scripts (writer.py and reader.py). The reader should be launched before the writer starts. Then the messages from the writer would be catched by the reader.

Writer

writer.py

import os
import time
from contextlib import redirect_stdout, redirect_stderr
import syncstream

os.makedirs('logs', exist_ok=True)  # Create the log folder.
buffer = syncstream.LineFileBuffer('logs/message', maxlen=10, tmp_id=os.getpid())
with redirect_stdout(buffer), redirect_stderr(buffer):
    for i in range(10):
        time.sleep(1.0)
        print('Line', 'buffer', 'new', i, end='\n')
buffer.new_line()  # A final signal

Reader

reader.py

import os
import shutil
import time
import syncstream

# Initialization.
shutil.rmtree('logs')
os.makedirs('logs', exist_ok=True)
buffer = syncstream.LineFileBuffer('logs/message', maxlen=10, tmp_id=os.getpid())

while True:  # Wait until getting 10 messages.
    time.sleep(1)
    messages = buffer.read()
    if len(messages) >= 10:
        break

# Show results.
for mitem in messages:
    print(mitem)

In the above example, the writer and reader are launched by two different python programs. Both the reader and the writer do not know the running details from each other. The only synchronized thing is the messages shared by the files. The file locks could ensure that the programs would not cause synchronization issues.

The same instance LineFileBuffer is used for both the writer and the reader. Different from the LineBuffer or LineProcBuffer, the reader need to call the method buffer.read() reapeatly. Each time the method is called, the records in the file would be iterated for once. The reader would keep scanning the file for every one second until 10 lines are collected.

danger

The only unsafe issue is the terminate of the program. Some of the lock files would be preserved after the program is finished. To avoid the interference from the previous lock files, we suggest to remove all lock files before the reader starts.