Tracking Cryptocurrency Exchange Trades with Google Cloud Platform in Real-Time

“Tracking Cryptocurrency Exchange Trades with Google Cloud Platform in Real-Time”

Pengantar

Dunia keuangan saat ini kompleks, dan teknologi lama yang digunakan untuk membangun saluran data keuangan tidak mengikuti. Dengan berbagai bursa keuangan yang beroperasi di seluruh dunia dan permintaan pengguna global, saluran data ini harus cepat, andal, dan dapat diskalakan.

Saat ini, menggunakan pendekatan ekonometrik—menerapkan model ke data keuangan untuk meramalkan tren masa depan—tidak berfungsi untuk prediksi keuangan waktu nyata. Dan data yang lama, tidak akurat, atau dari satu sumber tidak diterjemahkan menjadi data yang dapat diandalkan untuk digunakan oleh lembaga keuangan.

Praktikum

Task 1. Create your lab resources

• In Cloud Shell, run the following command:

gcloud beta compute instances create crypto-driver \
--zone=us-east1-c \
--machine-type=n1-standard-1 \
--subnet=default \
--network-tier=PREMIUM \
--maintenance-policy=MIGRATE \
--service-account=$(gcloud iam service-accounts list --format='value(email)' --filter="compute") \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--image-family=debian-11 \
--image-project=debian-cloud \
--boot-disk-size=20GB \
--boot-disk-type=pd-standard \
--boot-disk-device-name=crypto-driver

Connect to the instance via SSH

• In the Cloud Platform Console, on the Navigation menu, click Compute Engine > VM Instances.
• For the instance called crypto-driver , click SSH.
• Run the following commands to install all the necessary tools (such as java, git, maven, pip, python and cloud bigtable command line tool cbt):

sudo -s
apt-get update -y

sudo apt install python3-pip -y
sudo pip3 install -U virtualenv
virtualenv -p python3 venv

source venv/bin/activate

sudo apt -y --allow-downgrades install openjdk-11-jdk git maven google-cloud-sdk=349.0.0-0 google-cloud-sdk-cbt=349.0.0-0

• Now create the Bigtable resource. The first gcloud command will enable the required Bigtable and Dataflow API in the project. The next command will create a Bigtable Cluster called “cryptorealtime-c1” with one instance called “cryptorealtime”. The instance type is Development, and therefore it will be a one node instance. And finally, using the cbt command you are creating a table called “cryptorealtime” with one column family called “market” in the Bigtable instance.

export PROJECT=$(gcloud info --format='value(config.project)')
export ZONE=$(curl "http://metadata.google.internal/computeMetadata/v1/instance/zone" -H "Metadata-Flavor: Google"|cut -d/ -f4)
gcloud services enable bigtable.googleapis.com \
bigtableadmin.googleapis.com \
dataflow.googleapis.com \
--project=${PROJECT}
gcloud bigtable instances create cryptorealtime \
    --cluster=cryptorealtime-c1 \
    --cluster-zone=${ZONE} \
    --display-name=cryptorealtime \
    --cluster-storage-type=HDD \
    --instance-type=DEVELOPMENT
cbt -instance=cryptorealtime createtable cryptorealtime families=market

• Run the following command to create a bucket:

gsutil mb -p ${PROJECT} gs://realtimecrypto-${PROJECT}

• Now clone the application source code repository:

git clone https://github.com/GoogleCloudPlatform/professional-services

cd professional-services/examples/cryptorealtime
mvn clean install

• Once the build is finished, start the pipeline:

./run.sh ${PROJECT} \
cryptorealtime gs://realtimecrypto-${PROJECT}/temp \
cryptorealtime market

• Wait a couple of minutes, then run the following to observe the incoming trades by peeking into Bigtable. You can do this by using Cloud Bigtable CLI tool (called cbt). If the pipeline is successfully executing, you should see new data appearing in the cryptorealtime table.

cbt -instance=cryptorealtime read cryptorealtime

Task 2. Examine the Dataflow pipeline

• In the Cloud Platform Console, on the Navigation menu, click Dataflow.
• Click the name of the existing pipeline.

Task 3. Visualizing the data

• Go back to the SSH session and run the following command to open firewall port 5000 for visualization:

gcloud compute --project=${PROJECT} firewall-rules create crypto-dashboard \
--direction=INGRESS \
--priority=1000 \
--network=default \
--action=ALLOW \
--rules=tcp:5000 \
--source-ranges=0.0.0.0/0 \
--target-tags=crypto-console \
--description="Open port 5000 for crypto visualization tutorial"

• Now link the VM with the firewall rule:

gcloud compute instances add-tags crypto-driver --tags="crypto-console" --zone=${ZONE}

• Next, navigate to the frontend directory:

cd frontend/

pip install -r requirements.txt

pip uninstall  Flask Jinja2

pip install Flask Jinja2

python app.py ${PROJECT} cryptorealtime cryptorealtime market

• Open another SSH session and run the following command to find your external IP address for the crypto-driver instance:

gcloud compute instances list --format='value(EXTERNAL_IP)' --filter="name:crypto-driver"

• Copy the EXTERNAL IP address displayed to use for the next command.

http://<external-ip>:5000/stream

Task 4. Clean up

• You can stop the pipeline in either the Console or in the SSH session. It takes a few minutes with either method.

gcloud dataflow jobs cancel \
$(gcloud dataflow jobs list \
--format='value(id)' \
--filter="name:runthepipeline*" \
--region="us-central1")

• Inside the SSH session run the following commands to empty and delete the bucket:

export PROJECT=<Your Project ID>
gsutil -m rm -r gs://realtimecrypto-${PROJECT}/*
gsutil rb gs://realtimecrypto-${PROJECT}

• Inside the SSH session run the following command to delete the Bigtable instance:

gcloud bigtable instances delete cryptorealtime

• If prompted, Do you want to continue (Y/n), press Y.
• Close the SSH console.
• In the Cloud Platform Console, on the Navigation menu, click Compute Engine > VM Instances.
• Check the box next to the crypto-driver instance then click Delete, then Delete again to confirm your action.

Penutup

Sahabat Blog Learning & Doing demikianlah penjelasan mengenai Tracking Cryptocurrency Exchange Trades with Google Cloud Platform in Real-Time. Semoga Bermanfaat . Sampai ketemu lagi di postingan berikut nya.