SEC338: How FINRA builds serverless data-masking pipelines across accounts

FINRA ingests up to 100 billion trading data records daily that need to be obfuscated securely when moving them. This data replication and obfuscation supports production data analysis, setup environments for user acceptance testing (UAT), and troubleshooting production issues. With the automated, one-click solution, FINRA teams can leverage the pipelines and within hours (versus days) get data for analysis, development, and testing. The solution is serverless, leveraging AWS Step Functions, AWS Lambda, Amazon ECS/AWS Fargate, Amazon SNS, and Amazon SQS. FINRA plans to open source this by the end of this year so attendees will be able to adopt the solution.