Teko's Blog

Data Scraping With Access Tokens Using Node JS & Express & MongoDB
#nodejs #expressjs #mongodb #restapi #api #scraping #data
It was challenging to create a data scraping using Node JS as a scraping platform.
I had completed the project with 100% that the client has been satisfied
I created and Developed new methods to enhance the quality of the project and bring the data scraping to ultimate speed.

• Creating a parallel HTTP request that handles multiple requests.
• Creating multi tread Node JS microservice, that can run multiple nodes working together to scrap data and store it in the database
• Storing Data to MongoDB with full CURD controls and handlers
• Creating Internal Restful API to manage Scraping data and request information for MongoDB
• Access Tokens manager and store to can scrap pages that need logins credentials
• Can control the loop timer to speed up the scrap or slow down
• Can control how many links will scrap every loop