I have 5+ years of hands-on building experience in Data Engineering with clients in the UK, HK, & India. I build large scale end-to-end data pipe lines, with experience in batch as well as streaming data. I develop automated data processing systems for data modeling, mining and production while also being scalable and efficient.
My experience gave me a decent knowledge of Pipeline development, infrastructure, and scaling. I’ve used a wide range of technologies:
- Batch Processing (Data warehouse, ETL)
- Apache Hive, AWS Redshift & Sprectrum, Google BigQuery, SAP IQ, Apache Spark, MySQL, KSQL
- Data Ingestion & Stream Processing
- Structured Streaming, K Streams, Apache Kafka, Google PubSub, Google Cloud Dataflow(Apache Beam)
- Workflow/Pipeline Automation
- Apache Oozie, Apache Airflow, Cloud Composer, Cron jobs
- Data Architecture and Infra management(GCP & AWS)
- K8s Orchestration, Docker containers, AWS EMR, Google Dataproc, AWS S3 and Google Cloud Storage, AWS EC2 and GCP Compute Engine
- Data Visualization
- D3.js & Highcharts.js, Tableau
- Data Science and AI
- Familiarity with the fundamentals of machine learning
- Software Engineering
- MEAN Stack(Mongo Express Angular Nodejs)
I am interested in desinging data architecture and have been learning/building/contributing such skill sets at work but I have lot more to learn and I am going strong.
I love riding motor-cycles, hit me up if you want to plan a trip. I also like traveling to places and to learn about different culture and languages.
I am happy to contribure on Open Source projects, do let me know if see an opportunity.