Architecture and Data Blog

Thoughts about intersection of data, devops, design and software architecture

Publish data models in CI Pipeline

Enable wider usage of data models by publishing the latest models on team Wikis

Many a times ER models are created by the data team and are not shared outside of the data team generally for the lack of tools licenses, since its not feasible for the entire team to purchase licenses for the ER modelling tools such as Erwin Data Modeller or Er Studio

Posted by Pramod Sadalage on Saturday, September 16, 2023

SSL Connection to AWS Aurora

Creating secure connection to the AWS aurora involves multiple steps

On a recent project we had to connect to AWS Aurora postgres 10.6 version of the database in SSL mode using JDBC and Java 11 JRE. When the Aurora cluster is setup, we can force all connections to use SSL by using the options group settings (forceSSL=true), establishing secure connection from the application to the database is not as easy as it looks.

Posted by     Pramod Sadalage on Monday, July 15, 2019

Static Analysis of PL/SQL code

Applying modern development practices to database code

In all new development and sometimes during legacy codebase modernization, developers tend to add code quality checks and static analysis of codebase such as style checks, bug finders, cyclomatic complexity checking etc. into the CI/CD pipeline. When we inherit a codebase that has much PL/SQL and there is a desire to put the PL/SQL code base through the same types of code analysis, what options does a developer/dba have?

Posted by Pramod Sadalage on Tuesday, August 1, 2017

Using liquibase to load data and ignore some columns

Loading data into tables is needed many times on projects to load test, Liquibase provides a method to load data into tables with lots of customization. In the example shown below, I’m loading zip code data with the following column layout

"Zipcode","ZipCodeType","City","State","LocationType","Lat","Long","Location","Decommisioned","TaxReturnsFiled","EstimatedPopulation","TotalWages"

Posted by Pramod Sadalage on Monday, May 15, 2017

Synonyms as abstraction layer

In many development shops, developers are not allowed to access the database schema directly, and are not allowed to create tables, indexes, views etc, instead are given access via a different schema that allows SELECT, UPDATE and DELETE access on data. The general reason is to avoid developers creating database objects without

Posted by Pramod Sadalage on Friday, April 15, 2016

Using rake and activerecord to generate boilerplate DB Code

Generating code that is necessary for standard code

IN many projects, there are tables which need default audit columns such as Created_By, Created_Date, Modified_By, Modified_date and other columns that need to be updated every time some actions are done against the tables and/or columns. This type of functionality can be implemented using triggers.

Posted by Pramod Sadalage on Monday, July 20, 2015

Migrations in NoSQL databases

In relational database usage the pattern of migrations is well understood and has gained widespread acceptance. Frameworks such as DBDeploy, DBMaintain, MyBatis migrations, Flyway, Liquibase, Active Record Migrations and many others. These tools allow to migrate the database and maintain the version history of the database in the database. With the rise of NoSQL Databases and their adoption in development teams we are faced with the problem of migrations in NoSQL databases.

Posted by Pramod Sadalage on Tuesday, October 14, 2014

10 node mongodb ReplicaSet on a Single Machine"

While doing evalauation of NoSQL databases, we had a 10 node riak cluster and wanted check how a similar setup would work with mongodb. So started to setup a 10 node mongodb cluster. Since this was for initial spikes, we decided to set this up on a single machine as with the other test setup using Riak. Before I explain how we setup 10 node mongodb ReplicaSet, let me talk about replica sets.

Posted by Pramod Sadalage on Monday, June 9, 2014

10 node Riak cluster on a single machine

When trying to evaluate NoSQL databases, its usually better to try them out. While trying them out, its better to use them with multiple node configurations instead of running single node. Such as clusters in Riak or Replica-set in mongodb maybe even a sharded setup. On our project we evaluated a 10 node Riak cluster so that we could experiment with N, R and W values and decide which values where optimal for us.

Posted by Pramod Sadalage on Thursday, April 25, 2013

Backup in mongodb replica-set configurations

How to ensure you have a backup in mongodb with regards to replica-sets

There are multiple ways to take backups of mongodb is different configuraitions, one of the configuration that I have been involved recently is replica-sets. When mongodb is running in replica-set configuration, there is a single primary node and multiple secondary nodes. To take backup of the replica-set we can either do a mongodump of one of the nodes or shutdown one of the secondary nodes and take file copies, since in a replica-set all nodes have the same data (except arbiter).

Posted by Pramod Sadalage on Thursday, December 27, 2012

avatar

Developer, Data Architect, Data Specialist, Author and Thoughtworker

FEATURED TAGS
agile analytics bdd cd data database dba design developer devops evolutionary-design framework index lessons migration news nosql oracle practices refactoring ruby security testing