An adventure with the asynchronous multi-threaded time-series data processing in Python over Kubernetes
In this post, I would like to describe the beginning of my great adventure with Python-based Non-SQL asynchronous multi-threaded near-real-time time-series data processing over Kubernetes-orchestrated infrastructure. This post aims to provide an introduction — and maybe an inspiration — for beginners and intermediate users who look for a fundamental grasp of the matter. I know how annoying and tiresome testing different orchestration configurations may be — as well, I am aware of the complexity of the task at hand. Nevertheless, please do not be sad — after taking the first steps, everything will become more intuitive.
My main goal was to develop an efficient application to collect the WebSocket-originated messages from at least eight cryptocurrencies exchanges with (on average) 120 cryptos per exchange in milliseconds intervals. Afterwards, this information should be validated, formatted and formulated for mathematical calculations — everything in near-real-time precision.
I used to be a front-end developer focused on React and node.js. However, after a couple of weeks of digging for a solution, I realised one should not reinvent the wheel and therefore switched to the Cryptohooper platform. However, my endeavour remains, and I would like to help others make the first steps.
Now, after completing my Kubernetes training and acquiring a higher degree of Python software development concepts, I still find the codebase valuable.
In order to start our experiment, please fork the Wunder repository. Its name is somehow misleading — as “Wunder” stands for “a miracle” in German, I find it quite amusing now.
git clone git@github.com:pequx/wunder.git wunder
Please follow the repository README for installation instructions.
After completing the setup, we should see our infrastructure in the Kubernetes.
Please note I decided to skip Helm and focus on old fashioned Make-files based orchestration.
Now we have a functional system. In order to collect some data, please ssh the backend and execute a command like:
git clone git@github.com:pequx/wunder.git wunder
Now we have a functional system. In order to collect some data, please ssh the backend and execute a command like:
$ cli.py --debug hundi.action.markets$ cli.py --debug hundi.action.ticker
In my next post, I will explain in detail what is going on there.
Happy hacking ^^