10 tips to implement a data warehouse for a bank in 9 months
Western Union completes 29 transactions per second on average. The received data is being integrated for statistical modeling and predictive analytics into a unified platform. Like Western Union, thousands of other financial institutions and banks gather large data stores in a data warehouse for bank to use those in a variety of ways: to detect fraud, improve transaction processing and customer experience, boost trade execution, and gain a competitive advantage.
Today, banks come up to data processing with a special responsibility in the face of growing cases of sponsoring terrorism and money laundering.
Data is the new gold
The gold that should be stored and used respectively to bring its true value to your company. Being driven by new demands of local regulators, central banks, and ministries of finance, banks are those who are most in need of proper data storage.
Although a lot of banks are shifting from traditional data warehousing to Hadoop- or Spark-powered platforms for data analytics, there are still those who analyze information in monthly batches, which signifies 30-day trends can be lost.
All data should be managed fast and reliably, in a centralized place. Banks do succeed in reducing costs and skill gaps by acquiring cloud data processing services. However, neither Big Data technology nor cloud computing is no longer a competitive differentiator. Here is where banks are losing sight of what’s important. Banks should focus on accelerating data processing and turning the findings into actions to enhance customer service and customer experience, rethink risk measurement means, and handle fraud.
You can access and use data better by going to the cloud or implementing predictive modeling but that’s not enough. It would help if you comprehend the real power of data and data warehousing; thus, rethink the way you process, store, analyze it. You should create the storage from the ground up that fully matches with your business logic, your customer buyer journeys, transactions, payments, trade, marketing, and many more. Below are 10 tips to help you build that storage in the shortest time possible.
10 data warehouse implementation tips
Read more about how to implement data warehouse in our case study. This case is a good data warehouse implementation example that demonstrates our expertise and experience in custom development for banks. We’ll be glad to share more details at the meet.
1. Understand the project goal and consider the time
The duration of a corporate data warehouse development project can take up from 2 to 5 years on average. Is it too long, isn’t it? Have you ever thought that when your DWH is developed, many things will be irrelevant? What if we would say that it’s possible to complete a DWH-related project in just 9 months? We have seen in practice that this is quite possible!
To shorten the duration of the project development, consider the project goal and values first. Articulate these things precisely and clearly so that everyone understands what outcomes you expect. The rest but not the least factors affecting the project duration include the appropriately chosen technology, the project roadmap, and the right sequence of jobs to be done, the allocated skills and resources. Critical data like Golden Source data and dictionaries should also be taken into account.
2. Analyze your existing information systems
Data warehouse implementation can’t go without asking the right questions and analyzing information.
Before proceeding with the development, you should take an inventory of all data assets. Driven by a “data first” mindset, you can reach project goals faster and stress-free. The success of the project depends heavily on data quality. To make decisions and handle reporting, managers need to have complete and reliable data and knowledge bases. Thus, it is critical to analyze existing information systems, take inventory of data sources, define the global dictionaries, and the rules of building information links across various systems.
3. Make a rational choice of the technology platform
The analysis of existing systems, as well as a forecast of growth for the next 3-5 years, are important factors affecting the project duration. However, you should not overlook choosing the right technology platform and approach to data warehouse development.
Depending on your goals, you will have to choose between single-tier, two-tier, and three-tier architectures, the latter being the most widely used. Let’s say you need to delete redundant data and minimize the stored information, single-tier architecture will be your fit. In case you need to divide the data warehouse and available data assets, select the two-tier architecture. Ultimately, a three-tier architecture for data warehousing is the most popular as it has a multi-modular structure allowing for better flexibility and data management. Note that insufficient elaboration of the architecture or its redundancy can also negatively shift the timing and significantly increase the cost of the project.
Many DWH experts recommend starting with the most important sources used by business executives for decision-making. These can be customer relationship management systems, accounting software, various BI and reporting systems, etc. With the help of in-depth analysis, DWH implementation specialists discover all the ways information is collected in your organization.
The next critical step of the data warehouse design and implementation is understanding how data users collect and process the information. Once analysts make that clear for themselves, they formalize their understanding with all the project stakeholders to avoid ambiguity.
4. Establish effective interaction between business departments
Each data warehousing module development and implementation requires involvement from a particular business department or mutual engagement of several departments. Ineffective interaction between the departments, misunderstanding the importance of data warehousing, high workload often become major things distracting from providing timely data or acceptance testing feedback. This is clearly an issue for all stakeholders to take up, as ineffective collaboration leads to launch delays.
End to end data warehouse implementation requires the participation of many stakeholders in gathering requirements, defining the criteria of successful project outcomes, acceptance testing, users training. So, you must be prepared to spend some time on these activities. The time needed should be defined beforehand. A professional DWH vendor will inform you about the time needed from your side. It may sound like “a couple of hours per week for requirements gathering”, for instance.
5. Initiate data standardization procedures and data processing flow
If you are about to create a data warehouse from the ground up, there’s no doubt that something wrong is going with your data. You might definitely think of a centralized place to manage your data, if you face a variety of troubles: incorrect data format, poor description, missing data, different data formats in various departments, and many more. Let’s see how it works in practice: your front office system doesn’t capture the last name of the lead, whereas your digital marketing system requires the last name to be fed into a mandatory field. The gap between various data formats leads to poor data quality and the impossibility of making reliable data-driven decisions.
6. Audit all existing information systems to identify the real source of data
Most legacy banking systems were purchased from vendors. Let’s see, a bank would buy systems like an IBM zSeries mainframe, often referred to as monolithic systems, that had only one purpose, e.g., which was to support a deposit system. Buying one system per one purpose was a standard practice earlier for banks, which led to vertical silos and meant that each of those systems was written in different software generations. At the same time, it led to the interoperable complex web, which meant that systems were built and deployed first with no integration in mind. Consequently, it’s too hard to identify the origination of data sources.
7. Seek the support and collaboration of the executive management
Any enterprise-level project may fail without appropriate support and involvement from the executives’ side. Besides ensuring effective cross-department collaboration, executives are able to provide business sponsorship in data warehouse projects by communicating the value of data for the business area he or she owns. Business C-level representatives should also provide their requirements, e.g., what data they would like to feed to the analytical systems, what sources will data come from, what decisions it will support, and many more.
8. Use Agile methodology for data warehouse development projects
Waterfall is not a good fit for a data warehouse project. Consider Agile or Scrum to carry out the project jobs iteratively. For business intelligence projects, it’s impossible to gather and formalize all the requirements at once. You are losing time if a business analyst spends weeks asking users what they expect to get from the system to be developed. They don’t know what they want. Seriously. We recommend moving progressively and get feedback on what’s been released in sprints. This approach allows for getting timely feedback and managing change requests more effectively.
Usually, our team provides a comprehensive overview of DWH-related project management methodology to a customer. The overview is a part of a technical proposal; it contains helpful guidelines for data warehouse implementation, project time frame, roadmap, communication plan, team composition.
9. Prepare to embrace changes
Changes to data warehousing are inevitable. Eventually, you will have to make changes initiated by internal or external influencers, e.g., regulations or a new data management policy. Your ETL processes and toolsets should be established with possible changes in mind. Do comprehensive research to identify the maximum possible scenarios and adapt your system to those beforehand.
Any change is challenging, and although proven practices for data warehousing implementation will help minimize those, you must be prepared.
10. Consider powerful memory devices for the highest performance
According to Oracle, input/output operations are often a bottleneck for large queries. Oracle paper says that modern hardware platforms can reduce latency and eliminate I/o operations. Such memory devices as DRAM and Flash can significantly improve performance.
Understanding the DWH project challenges and following the tips mentioned above, you can shorten the project development life cycle from a couple of years to 9 months. Surely, the timeline will also depend on how accurately you adhere to the project strategy, which is crucial for any IT project in general, and a data warehouse project in particular. Further, the implementation of recommendations requires close collaboration with a software vendor with strong data warehousing expertise for banks.
How DICEUS can help you
The DICEUS team will be glad to help you build the most effective architecture to manage your historical and real-time data and offers the following:
- Development and maintenance of a DWH project strategy
- Data warehouse implementation plan
- Assistance in identifying the roles and responsibilities of a DWH environment
- Collection of business requirements
- Data modeling
- Assistance in selecting infrastructure tools
- DWH custom development and ETL program creation
- Integration testing and data loading
- Data warehouse for bank support
The complete list of DWH-related services is not limited to the points mentioned above and can vary concerning the complexity and specifics of your project.
Our team will devise an individual approach to your DWH project, including all required steps to implement data warehouse, timeline, budget, team composition. Data warehouse implementation timeline significantly depends on the scope and complexity of your existing and awaited storage solutions. It may take you from weeks to months to develop and launch a data warehouse. For our recent banking project, we spent 9 months. More accurate estimation is available upon your request and based on requirements and the preliminary discovery phase.
More insights in the article “Leveraging enterprise data warehouse to facilitate your business efficiency”
