Big Data Architecture for Enterprise
We were asked to help turn a high-level vision into reality for a large retail bank. The bank knew that the analytical and front-end demands of the project were such that their existing tools could not support the vision: a step change in the quality of service offered to the customer.
HTI Labs was engaged to define the architecture for the data tiers of this "24-7", enterprise-scale platform that would need to serve millions of customers across an array of devices. The client wanted to use our experience of the latest Big Data platforms alongside our ability to define and implement technical architecture appropriate for large enterprises.
The platform was required to provide the following:
- A world-class permissioning model to ensure confidentiality of data
- A flexible delivery model which would allow for frequent changes to the platform
- Access to the data by a varied set of trusted application development teams
- The ability to accept real-time updates of transactional data
- A horizontally scalable architecture and infrastructure
- Access to data and analytics in milliseconds
- Storage for a growing set of data
- Complex analytics applied across the data, often defined by other teams in a variety of languages (eg Python, R)
- Full operational requirements including data recovery, fail-over and low-tier environments
- Most critically, uncompromisable security.
It is envisaged that the landscape of consumer financial services will change dramatically over the next few years. This platform would have to adapt to make our client the leader not just next year but for many to come.
HTI Labs' focus to date
HTI Labs was assigned these initial responsibilities:
- Gathering data requirements
- Data modelling
- Definition of the data tier technical architecture
- Definition of the data technologies
- Physical implementation of the data model
- Data governance and management of change
- Delivery of the analytics platform
- Integration with the wider platform
- Non-functional testing of the data tier
- Scaling of the data platform.
In addition to this, our team has:
- Been asked to define pieces of the architecture relating to application development and user interface (UI)
- Defined the technical architecture for the infrastructure tier
- Helped with overall resolution of platform issues and risks
- Communicated to the wider team the processes for gathering more data on to the platform
- Reviewed the architecture with the client.
The technical architecture has been developed based on the following high-level technologies:
- Amazon Web Services (AWS) – for provision of the infrastructure and a number of the technical services
- HP Vertica
- Amazon Relational Database Services (RDS)
- Layer 7
- OData (Open Data Protocol)
Please contact us to understand our experiences with these technologies.
The client continues to use HTI Labs for new challenges on the platform as they prepare to go live with the initial services and develop a "road map" for the future. We’re excited to be part of their journey!
At HTI Labs we have developed our own way of working focused on the pillars of innovation, technical delivery, business analysis & management and our unique assets. We're happy to work with clients and share this approach with them. We show you more here...