Last month, Strands engineers attended JBCNConf, the first big Java convention in Barcelona. The conference was a fantastic opportunity for us to exchange knowledge with our peers in other sectors and learn about the newest tools and techniques that enable us get the most out of Java for big data analytics and machine learning.
Many exciting workshops took place over the two days. Some of the more interesting topics covered were:
- Microservices as an alternative to designing multimodule systems
- Using Docker for software packaging and delivery
- Advanced automated testing approaches
- Parallel data processing with Hadoop and YARN
- Teamwork strategies
MicroservicesIs your domain composed of a bunch of separate modules that have to work together? Microservices (MS) take modularity to the next level. First introduced by Martin Fowler, MS architecture consists of building small systems with a very specific functionality (API) and independent development cycles.
This approach makes sense for big systems where many subsystems interact with each other, and allows a team of developers to work on a specific part in parallel with other teams working on others. Since the different subsystems are totally decoupled, they can be mantained and deployed independently from one another.
For example, let's say we want to build a shopping cart app with 3 entities: Users, Products and Stock. A typical monolithic approach would be to build the app as a whole system that manages all 3 entities. In MS however, we would build a separate service for each entity to be processed: a Users service, a Products service and a Stock service.
DockerSoftware deployment can become a problem when you have to run the same app on different platforms. Different hardware architectures, operating systems, or different configurations can make running an application that is required to work on top of many different environment become overly complex - something we see a lot in FinTech when working on legacy core banking systems. A piece of software might have been battle tested on Linux systems, but imagine needing to run it on Windows machines too - you're right, a tsunami of issues is likely coming your way!
Docker was created in 2013 to overcome this problem. Think of it like standard containers used in shipping, since they are used to move goods of any shape or size via different modes of transportation without ever changing the container. Docker is similar in that it isolates application enviroments from the outside world, meaning that they can run in any enviroment simply by placing them in a system that uses a Docker engine.
Docker containers may sound similar to Virtual Machines and they certainly are - but one important thing makes them unique: they are much lighter since they don't contain an entire operating system, just the libraries and configuration needed by the apps.
Automated testing is both loved and hated by many developers. But everyone who has done test-driven development (TDD) knows that one has much more to gain than to lose from it. Not only does it help us instantly detect errors caused by our latest changes, but it can also enable the definition of new functionalities before writing a single line of code.
At JBCNConf we were shown some scenarios (e.g. stock trading) in which automated testing is a must and saw it applied as we never thought possible. Thousands of tests were applied every time a change was committed and issues were fixed by the whole team upon detection before they continued with their tasks. No more than one failing commit was permitted.
Since Fintech is a similarly demanding domain, these real life examples had us thinking about new ways to continiously improve our automated testing practices by not only adding more tests but also different varieties of testing: unit, integration, acceptance and also performance tests at different levels. This would create a powerful shield for added security against potential system flaws.
HADOOP AND YARN
Hadoop is an open-source framework that makes distributed data processing and storage a lot easier. It assumes a storage part (HDFS) and a processing part (Map-Reduce), so when a massive file has to be processed it is distributed among the nodes whithin a cluster and the different parts are processed in parallel.
Apache Hadoop's MapReduce and HDFS components were based on Google's Map-Reduce research as well as Google File System. Although it has been around for several years, new challenges made it reach new heights, evolving into what is now known as YARN. JBCNConf showed us that YARN could be an excellent tool to help us optimize data processing, especially when it comes to dealing with complex flows.
JBCNConf was not only a good place to share technical experiences but also to learn how other software companies work effectively in teams. Here are some of the key drivers of successful agile teams:
- At daily standup meetings, avoid talking about useless/endless topics
- Periodic Swarm Meetings that get everybody to discuss every new feature
- Pair or tandem programming, so developers can share shortcuts with each other ;)
- Rotating teams enable everyone to get to know every parts of the system
By keeping up with the latest in software development, we continue to translate this knowledge to help banks stay at the forefront of FinTech innovation. We've already started applying these new tools and techniques to maximize the value we get from Java, and are looking forward to next year's conference to learn even more!