1. What strategies have you implemented to harness the potential of innovative big data analytics techniques to fulfil your business requirements?
We have developed a modern data analytics platform leveraging cloud-based technologies. This hybrid onpremises and cloud-based infrastructure provides the scalability and flexibility needed for advanced analytics.
For example, we migrated our data warehouse into a large data lake, allowing queries across petabytes of financial data. We also optimised our workloads across a unified analytics engine for risk modelling and trade analysis, reducing run times by over 40%. Our data scientists can now train machine learning models faster using compute provisioned through data science workbenches.
Establishing strong data governance and talent practices has been equally important. To maintain regulatory compliance, we formulated policies for data access, encryption, and cross-border data flows. Our training programmes also allow the whole organisation to become proficient in Structure Query Language (SQL) and dashboarding tools to self-serve insights.
The combination of in-house infrastructure, skilled talent and external capabilities allows us to execute modern analytics while maintaining data security and control. We continue to evaluate new technologies like data fabric and auto-machine learning to further our analytics maturity.
2. If your company can give consent for this, can you share your experiences from one of the projects that you were recently involved in?
To manage the volume of data and complexity of analytics that has grown within the bank, we recently enhanced our data platform to increase our productivity and optimise our data pipeline.
A key focus was optimising the data jobs that consumed the most resources and took the longest execution time. These revolved around the machine learning models that powered our risk management and trading analytics applications. By leveraging features like caching, broadcast variables, and partition pruning, we reduced shuffling operations and memory usage significantly
Additionally, we tuned over 100 different configuration parameters across the execution, serialisation, memory management and hardware provisioning layers. This exercise of performance tuning was tedious but helped unlock massive gains. Finally, we also redesigned certain jobs to avoid unnecessary stages.
“The combination of in-house infrastructure, skilled talent and external capabilities allows us to execute modern analytics while maintaining data security and control”
The impact was effective —our core analytics applications now run over ten times faster while consuming 25% less cluster resources. These performance gains have been crucial to scaling up the data science workloads as we build newer AI-driven features that will make our platforms more resilient.
3. What are some of the challenges in implementing big data analytics that make current services unable to provide an optimal solution?
Talent and technology gaps can make implementation challenging. Advanced techniques like machine learning require scarce specialised skills. Existing tools also have limitations in enabling productionscale deployments. Close collaboration across teams is key to overcoming these obstacles.
4. How do you see the future of big data analytics evolving in your industry?
I see AI and machine learning being applied across the finance function - from client analysis and risk management to back-office automation with embedded Generative Capabilities. Cloud platforms will enable access to more data sources and computing power. Regulation will also push financial institutions to leverage data and analytics further for transparency and reporting.
5. What advice would you give to other organisations leveraging big data analytics to drive innovation and improve their competitive edge?
Start small, focus on business needs, and make sure you have the right people involved. A crawl, walk, run approach allows you to demonstrate quick wins, build internal capabilities and gain stakeholder buy-in for larger initiatives. Finally, balance innovation with resiliency to manage technology risks effectively.
