An adventure with the asynchronous multi-threaded time-series data processing in Python over Kubernetes

In this post, I would like to describe the beginning of my great adventure with Python-based Non-SQL asynchronous multi-threaded near-real-time time-series data processing over Kubernetes-orchestrated infrastructure. This post aims to provide an introduction — and maybe an inspiration — for beginners and intermediate users who look for a fundamental grasp of the matter. I know how annoying and tiresome testing different orchestration configurations may be — as well, I am aware of the complexity of the task at hand. Nevertheless, please do not be sad — after taking the first steps, everything will become more intuitive.

My main goal was to develop an efficient application to collect the WebSocket-originated messages from at least eight cryptocurrencies exchanges with (on average) 120 cryptos per exchange in milliseconds intervals. Afterwards, this information should be validated, formatted and formulated for mathematical calculations — everything in near-real-time precision.

I used to be a front-end developer focused on React and node.js. However, after a couple of weeks of digging for a solution, I realised one should not reinvent the wheel and therefore switched to the Cryptohooper platform. However, my endeavour remains, and I would like to help others make the first steps.

Now, after completing my Kubernetes training and acquiring a higher degree of Python software development concepts, I still find the codebase valuable.

In order to start our experiment, please fork the Wunder repository. Its name is somehow misleading — as “Wunder” stands for “a miracle” in German, I find it quite amusing now.

git clone git@github.com:pequx/wunder.git wunder

Please follow the repository README for installation instructions.

After completing the setup, we should see our infrastructure in the Kubernetes.

Please note I decided to skip Helm and focus on old fashioned Make-files based orchestration.

Now we have a functional system. In order to collect some data, please ssh the backend and execute a command like:

git clone git@github.com:pequx/wunder.git wunder

Now we have a functional system. In order to collect some data, please ssh the backend and execute a command like:

$ cli.py --debug hundi.action.markets
$ cli.py --debug hundi.action.ticker

In my next post, I will explain in detail what is going on there.

Happy hacking ^^