Data Engineering for Data Scientists: Checklist

It is always good for a data scientist to understand more about data engineering, especially the blueprint of a fully productionized data platform.

There are several things to get into:

• Connection to Data Sources
  • Connect to DB
  • Connect to Streaming Data
    • Message Queues
  • Connect to Website
    • Scraping
      • [[Node Crawler]] Node Crawler Write a crawler using nodejs
    • API
  • Other Data Services
• [[Data Storage]] Data Storage Storing big data
  • Data Lake
  • [[Data Warehouse]] Data Warehouse Take care of your data and your data will show you its power.
  • Message Queues
• [[Data Processing]] Data Processing Processing Data is essential.
  • Streaming
  • Batch Processing
• Data Buffer
  • Cache:
    • Redis
      • [[Basics of Redis]] Basics of Redis Redis is an in-memory nosql data structure server
  • Message Queues
    • Kafka
    • AWS Kinesis
• Using Data
  • Query Data
  • Visualization
  • Analysis and Model Building
• [[Scale Up]] Scale Up scale up your services
  • Scale up storage
  • Scale up node
  • Parallel