I am looking for a solution architect for consulting on what tool to use to setup an appropriate backend stack for my project.
The project is a SAAS product. A few tasks required to run:
- Thousands of page to scrape each day using headless browser.
- Custom ML models training and to put in production.
Must have good knowledge of:
- Cloud & Baremetal providers : AWS, Google, Scaleway, Hetzner
- OS : Linux/Unix
- Container : Docker, Kubernetes
- Infrastructure as Code : Terraform, Cloudformation
- Automation Process : Ansible, Chef, Puppet
- Git : Github, Gitlab
- Build systems
- Cost optimization (Preemptible / spots instances)
- Performance Optimization
- Load Balancing
- Audits : Security, Cost, Backup, Updates and Patches
- Database : Postgresql, Cockroachdb, TiDb
- Distributed storage
- Data persistence
- Multi cloud, multi providers
- Networking security
- Devops, gitops, best practises
For now with my current research I tend towards:
- Infrastructure as code with Terraform.
- Private, self hosted GitLab for git and CI/CD
- Kubernetes cluster (with a few nodes always up on Hetzner) and the rest dynamically provisionned on AWS using spot instances
- Postgresql or Cockroabdb
I need you to help me clarify some things and choose the correct solution.
1h or two by phone or chat to answer my questions.
30 freelancers are bidding on average $34/hour for this job
Terraform approach is good but I would recommend using Microservices architecture. There are many other software which helping in making a scalable Web Crawling Application which I'll discuss after hire.