https://steemitimages.com/0x0/http://cryptogab.com/wp-content/uploads/2014/09/iiskg2.jpg # TL;DR # I wanted to share with you all how to begin building an open-source platform for analyzing cryptocurrency market data. If you aren't a fan of getting your trading advice from platforms derived from traditional markets and are curious about building your own tools for forecasting and investing, I hope you'll read further and comment. All feedback is appreciated. # Free Datahz? # There are many APIs to choose from with more detailed options ... often with a price tag. I chose this API for a few reasons: * Free (for now?) * Historical Data for Many Alt-Coins * Easy of Use! (Thanks to http://coinmarketcap.northpole.ro) # Setup # https://steemitimages.com/0x0/http://4.bp.blogspot.com/-1zFG4UY4gKg/UpzI685WlKI/AAAAAAAAIco/IHY2I3bmHeU/s1600/to+the+moon+meme+2.jpg ### Get VBox ### https://www.virtualbox.org/wiki/Downloads ### Get ELK VM ### https://bitnami.com/redirect/to/151919/bitnami-elk-5.4.1-0-linux-debian-8-x86_64.ova ### Config ELK VM ### _Change Adapter 1 to Host-Only_ ![1.png](https://steemitimages.com/DQmVJFgxN3miT8qVEoTjJZ2wEtFZT8DjL4CRWE6HPD8xLSx/1.png) _Enable Adapter 2 to NAT_ ![2.png](https://steemitimages.com/DQmRJTMCEBYYz8SzpQaPpcfJS9oh62xUz4c5m5NzPMikyj1/2.png) _For Improved Performance, Adjust RAM/Processors/Etc._ ![3.png](https://steemitimages.com/DQmRksixxw8nkhxFMCFhN2WGCpkb5s6Esnd48JooG9x354k/3.png) _Login with Defaults_ **bitnami/bitnami** (Change password) ### Enable ssh (this can be skipped but, I preferred ssh for this) ### sudo rm -f /etc/ssh/sshd_not_to_be_run sudo systemctl enable ssh sudo systemctl start ssh ### Install python + pip + elasticsearch.py ### sudo apt-get upgrade -y python-dev libssl-dev git-dev sudo python < <(curl -s https://bootstrap.pypa.io/get-pip.py) sudo pip install elastic search ### Check IP Address ### sudo ifconfig eth0 Link encap:Ethernet HWaddr de:ad:be:ef:ca:fe inet addr:192.168.100.101 Bcast:192.168.100.255 Mask:255.255.255.0 # Now we are ready to get going ... # https://steemitimages.com/0x0/http://3.bp.blogspot.com/-xpJ5piAA6vQ/UpzI6GvaoXI/AAAAAAAAIcg/IIbdxoSQt8s/s1600/to+the+moon+meme.jpg The basic workflow is like so: 1. Fetch JSON data from API 1. Parse JSON data 1. Configure elasticsearch index mapping to use the JSON object timestamp 1. Send result of parsed JSON data to elasticsearch This script can be saved to to something like chistory.py: #Imports import sys import datetime import requests import json from elasticsearch import Elasticsearch #Variables server = 'localhost' arg1 = sys.argv[1] arg2 = sys.argv[2] doctype = arg2.lower() id = arg1 #Send data to ELK function def sendtoelastic(data): try: es = Elasticsearch(server) es.index(index=id, doc_type=doctype, body=data) except Exception as e: print 'Error sending to elasticsearch: {0}'.format(str(e)) pass #Send settings to ELK function def settingselastic(data): try: es = Elasticsearch(server) es.indices.create(index=id, ignore=400, body=data) except Exception as e: print 'Error sending to elasticsearch: {0}'.format(str(e)) pass #Build request, download and load JSON data payload = {'coin': arg2, 'period': arg1} r = requests.get('http://coinmarketcap.northpole.ro/history.json?', params=payload) h = r.text d = json.loads(h)['history'] #Send settings settings = '''{"mappings": {''' + '"' + arg1 + '"' + ''': {"properties": {"timestamp": {"type": "date","format": "epoch_second"}}}}}''' settingselastic(settings) #Send data for i in d.items(): for l in i: if type(l) is dict: sendtoelastic(json.dumps(l)) ### **Usage:** ### python chistory.py 2017 BTC python chistory.py 2017 BTC;python chistory.py 2017 ETH;python chistory.py 2017 DOGE python chistory.py 2016 BTC;python chistory.py 2016 ETH;python chistory.py 2016 DOGE ### **Coins:** ### A current list of coins can be found [here](http://coinmarketcap.northpole.ro/coins.json) ### **Dumping Index:** ### If for any reason we aren't happy with the results, we can dump elastic like this: curl -XDELETE 'http://localhost:9200/_all' # Haz Datahs ... Now What? # https://steemitimages.com/0x0/https://pbs.twimg.com/media/CEWFP8oWIAAFoaG.jpg Here's a quick how-to visualize your data in Kibana with a few line graphs which may be of interest ... * Open your browser and go to http://192.168.100.101/elk (use IP address from above) (user/bitnami) * Create Index "201*" ### Create Visualization ### **__Change.1h x Date__** 1. Click the "Visualize" Button 1. Click "Create New Visualization" 1. Click "Basic Charts > Line" 1. Select Index "201*" 1. Click Drop Down Button for Y-Axis 1. Select Aggregation > "Sum" 1. Select Field > "change1h" ![5.png](https://steemitimages.com/DQmSnBy9EYyfB6fJxDLsBAsPugPckrveMVU19mtnT9Nhsh4/5.png) 1. Click "Select Buckets Type" > X-Axis 1. Select Aggregation > "Date Histogram" 1. Select Field > "timestamp" 1. Select Interval > "Daily" ![6.png](https://steemitimages.com/DQmUsaBEpDfHZRmJXU727yx8iAXpAzty2FoDJNGVdi8XwS2/6.png) 1. Click "Add Sub-Buckets" > "Split Series" 1. Sub-Aggregation > "Filters" 1. Filter 1 > "btc" 1. Filter 2 > "eth" 1. Filter 3 > "doge" ![7.png](https://steemitimages.com/DQmReuqK4DhELJAyaQyhsV4ueFiN9yuQeTRJDd1hxnvAAav/7.png) 1. Save **__Change.24h x Date__** 1. Change Y-Axis Field > "change24h" 1. Save 1. Change name Change.24h 1. Check "Save as a New Visualization" Option 1. Save **__Volume.EUR x Date__** 1. Change Y-Axis Field > "volume24.eur" 1. Save 1. Change name Volume.EUR 1. Check "Save as a New Visualization" Option 1. Save ### Create a Dashboard ### 1. Select Dashboard Button 1. Create New Dashboard 1. Add Visualizations to Dashboard ![4.png](https://steemitimages.com/DQmcX1wK8BPKWKrHjvzZi6tiNfqocR8aejn9DJdjQrfpTz5/4.png) # On APIs # This is just a basic example of leveraging an API for analytics. The basic mechanics ought be able to be applied to other APIs of interest. If you'd like to see more API analytics, follow and up vote. I hope this helps someone! If you made it this far ... Thanks!