I need a AWS expert for a couple of serverless applications.
Attached please find process flow & a sample of data.
The project is to create data bucket that can collect any number of inbound requests of an undetermined velocity and filesize. For now the region will be AWS AU-SYD only, with intention of similar data collection points being created later through cloning the setup into the other regions. All with their own Queue and Database.
The objectives of this project are to:
1) Always be online
2) Be able to be regionalised
3) Accept 3 formats of data (for now, JSON/XML,CSV)
4) Store the data immediately so it can not be lost (SQS ? )
5) Process at pace the SQS pendings,
a) any failing parsing should be thrown to a Dead Letter SQS
b) any pass should be inserted into an RDS, PER mapped ID. In the following format:
(Table: Nerve)[UniqID, TIMESTAMP, SOURCE, IP]
(Table: NerveData)[[login to view URL], DATA]
6) The Dead Letter SQS should be able to be transferred back into the original queue for re-attempt
7) The Dead Letter queue should have rolling backups to RDS or similar, before deleting after 7 days
Once this part of the system is working, there will be more work for then processing the data that is in RDS in a separate routine that will extract each collected data in RDS and convert it into a Table per "ID". If you inspect the sample files each file contains 3 columns (this may vary) and one is identifier for a IoT device [ ID _ BATCHNO _ VALUE ].
Some files can contain more than 1 devices data, and the format may be subject to change at a later date so I am trying to separate the collection and parsing into separate independently scalable process. And the first logical one to build is the receiver.
We can get to that later but if you understand all the above process or have suggestions on ways to improve the mechanism. It needs to be really, really solid and save all data, always.
After the setup is done Id like a quick walk through. I have experience with AWS EB and RDS but not SQS/Lambda and would appreciate some skilled feedback if any improvements can be made to the model.