[{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/agile/","section":"Tags","summary":"","title":"Agile","type":"tags"},{"content":"Developer, Data Architect, Data Specialist, Author and Thoughtworker. Thoughts about the intersection of data, devops, design and software architecture.\n","date":"24 September 2026","externalUrl":null,"permalink":"/","section":"Architecture and Data Blog","summary":"Developer, Data Architect, Data Specialist, Author and Thoughtworker. Thoughts about the intersection of data, devops, design and software architecture.\n","title":"Architecture and Data Blog","type":"page"},{"content":"When application has been running in production or uat for some time there are some bugs that get reported because of bad data quality. This data is either generated by the application or is created by data migration scripts, data conversion scripts. The application suffers with non-deterministic errors especially is some specific environments while similar errors are not noticed in other environments.\nHow # When application bugs or stack traces are reported, the data team should investigate along with the developers on the root cause of the bug. Some bugs are caused because the application is not expecting data in the given format, some bugs are caused by the application not expecting null values in the database and other times the data is not conforming to referential integrity rules as they are not defined in the database.\nData format # Many times when data is loaded in the database using data conversion, data upload scripts there is mismatch with the type of formatting that is needed by the application and the type of formatting done by the data upload/conversion scripts. In collaboration with the developers the data team should introduce the data migration scripts that format the data in the correct way. Database refactoring pattern such as [http://databaserefactoring.com/IntroduceCommonFormat.html](Introduce Common Format) as a way to fix the formatting errors across all rows instead of fixing one occurrence of the error.\nUnexpected null values # In situations when the application is expecting the data attribute to not contain null values, the application code does not know how to deal with data where null value is found. These cases lead to null pointer exceptions, when there are these kinds of bugs found, the data team should try to isolate the reason why the data attribute is set to null, what all parts of the application or other scripts put data in this attribute and can this attribute be made non nullable.\nIn some situations we can in fact make the column non nullable by applying the [http://databaserefactoring.com/MakeColumnNonNullable.html](Make Column Non Nullable) database refactoring pattern. In this pattern, the data team finds all the rows where the column has null values and working with the business finds data for those rows, applies the data fixes, when the column is made non-nullable make sure the application testing is done to ensure it works with the non-nullable column and move these changes in to production.\nPersistence frameworks driving database design # Some persistence frameworks such as https://hibernate.org/orm/ https://blog.mybatis.org their usage my sometimes make the team to skip setting up foreign keys as the persistence frameworks may insert data out of order resulting in integrity constraint violations. The lack of foreign keys exposes the database for bad data to be entered either by the application or other data import programs.\nOther situations where the primary key or unique key is being generated by some other means and the value may not be available burning the initial insert resulting in unique constraint violations, removing the primary key or unique key may not be the right option as this will lead to bad data getting into system.\nThe data team can pair with the developers and introduce DEFERRED constraints which instruct database to check for constraints at commit time instead of immediately after the insert, delete or update statement.\nCREATE TABLE payment ( paymentid NUMBER NOT NULL, paymentnumber VARCHAR2(128), customerid NUMBER NOT NULL, CONSTRAINT pk_payment PRIMARY KEY (paymentid) ); ALTER TABLE payment ADD (CONSTRAINT fk_payment_customer FOREIGN KEY (customerid) REFERENCES customer DEFERRABLE INITIALLY DEFERRED ) ; Setting up the constraints this way allows for the database integrity to be maintained and at the same time allows the application developers to function without re-writing or making major changes the applications persistence layer, thus improving productivity of the team.\nMaintaining good data quality, allows for the developers to not code defensively, such as checking for null on columns that are not supposed to be null, parent rows existing when child records are found. Having better data quality also reduces null pointer exceptions errors and improves the quality perception of the application.\nOthers # Similarly other data quality issues such as missing foreign keys or non standardized data can be fixed by applying [http://databaserefactoring.com/AddForeignKey.html](Add foreign key), [http://databaserefactoring.com/ApplyStandardCodes.html](Apply Standard Code) by the data team in collaboration with the developers by discovering these patterns easily and apply corrections that fix the root cause of the problem instead of fixing symptoms caused by bad data.\n","date":"24 September 2026","externalUrl":null,"permalink":"/post/collaborate_to_improve_data_quality/","section":"Posts","summary":"When application has been running in production or uat for some time there are some bugs that get reported because of bad data quality. This data is either generated by the application or is created by data migration scripts, data conversion scripts. The application suffers with non-deterministic errors especially is some specific environments while similar errors are not noticed in other environments.\n","title":"Collaborate to improve data quality","type":"post"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/dba/","section":"Tags","summary":"","title":"DBA","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/post/","section":"Posts","summary":"","title":"Posts","type":"post"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/practices/","section":"Tags","summary":"","title":"Practices","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/quality/","section":"Tags","summary":"","title":"Quality","type":"tags"},{"content":"","date":"24 September 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"Lots of time hand coded SQL is written in the applications for performance sake, getting data using complex SQL out of the tables. As the database structure changes it becomes really hard to find out all the SQL that needs to be changed. In other cases SQL is generated based on certain conditions in the code and its hard to find if the generated SQL is valid after database changes are done.\nUsually the problem surfaces when table structures are asked to be changed and team members reply by saying I will have to find all the places where this is used\nHow # Every project tends to have custom SQL in the form of either SQL in the client code such as Java or database code written in the form of stored procedures, views, triggers, packages and functions. As part of the iterative development cycles parts of the code and the database structure are bound to change.\nCreating a suite of tests that can invoke the relevant code so that it can be verified that the code is valid with respect to syntax and also is functionally correct provides a safety to confidently make the changes required.\nLets look at Java code that is generating a SELECT statement based on the parameters it receives\nsqlString.append(\u0026#34;SELECT customerId, name, customernumber FROM customer WHERE 1 = 1 \u0026#34;); if (countryId != null) { sqlString.append(\u0026#34; AND countryId = \u0026#34;).append(countryId); } if (regionId != null) { sqlString.append(\u0026#34; AND regionId = \u0026#34;).append(regionId); } if (salesPersonId != null) { sqlString.append(\u0026#34; AND salesPersonId = \u0026#34;).append(salesPersonId); } sqlString.append(\u0026#34; ORDER BY customerId\u0026#34;); As shown in the code snippet above, its hard to see the resulting SELECT generated, this SELECT statement generation can be verified by executing the method using techniques described in Behavior Driven Database Development as shown in the code snippet below\nCustomerSearchBuilder searchBuilder = new CustomerSearchBuilder(); List\u0026lt;Customer\u0026gt; nullParameters = searchBuilder.searchCustomersFor(null, null, null); assertEquals(4, nullParameters.size()); List\u0026lt;Customer\u0026gt; noSalesPerson = searchBuilder.searchCustomersFor(1L, 1L, null); assertEquals(1, noSalesPerson.size()); List\u0026lt;Customer\u0026gt; noCountry = searchBuilder.searchCustomersFor(null, 1L, 99L); assertEquals(1, noCountry.size()); List\u0026lt;Customer\u0026gt; noRegion = searchBuilder.searchCustomersFor(1L, null, 99L); assertEquals(1, noRegion.size()); List\u0026lt;Customer\u0026gt; customers = searchBuilder.searchCustomersFor(1L, 1L, 99L); assertEquals(1, customers.size()); Similar situations are also encountered when using other SQL constructs like INSERT code below shows the insert statement being used by the java code\ntry { stmt = DB.prepare(\u0026#34;insert into customer\u0026#34; + \u0026#34;(customerid,customernumber,name,taxlocation) \u0026#34; + \u0026#34;values (?, ?, ?, ?)\u0026#34;); stmt.setLong(1, customerToInsert.getCustomerId()); stmt.setLong(2, customerToInsert.getCustomerNumber()); stmt.setString(3, customerToInsert.getName()); stmt.setString(4, customerToInsert.getTaxLocation()); stmt.execute(); } catch (SQLException exc) { DB.HandleDBException(exc); } finally { DB.cleanup(stmt); } The INSERT statement should be executed by the test method using a test customer, which should be removed after the test is done. The code below shows the test code.\npublic void testCustomerGatewayInsert() throws Exception { Customer insertCustomer = createTestCustomer(\u0026#34;Mike\u0026#34;, customerIdToUse); customerGateway.insert(insertCustomer); Customer foundCustomer = customerGateway.findByCustomerPOID(customerIdToUse); assertEquals(\u0026#34;Incorrect Customer ID\u0026#34;, insertCustomer.getCustomerId(), foundCustomer.getCustomerId()); cleanupTestDataForCustomerId(insertCustomer.getCustomerId()); } This method of automated execution of all the hand written SQL helps to execute all the database code used by the application. These kinds of tests are known as Integration Testing, if anything changes in the database all of the Database Integration Tests can be run to find out what portions of the application are affected, these integration tests can also be run as part of continuous integration build as described in Continuous Integration.\n","date":"23 September 2025","externalUrl":null,"permalink":"/post/automated_custom_sql_testing/","section":"Posts","summary":"Lots of time hand coded SQL is written in the applications for performance sake, getting data using complex SQL out of the tables. As the database structure changes it becomes really hard to find out all the SQL that needs to be changed. In other cases SQL is generated based on certain conditions in the code and its hard to find if the generated SQL is valid after database changes are done.\n","title":"Automated testing of custom sql","type":"post"},{"content":"","date":"23 September 2025","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"Devops","type":"tags"},{"content":"When accessing the database using stored procedures for basic Create, Read, Update and Delete (CRUD) functions, or when you want to write triggers that capture the before and after values in tables, or when you want to create Plain Old Java Objects (POJO\u0026rsquo;s) that match the database objects, writing these by hand takes a lot of effort and also since the requirements are changing in an agile projects at a frequent rate, there will be design changes to meet the requirement changes, so the triggers, CRUD stored procedures or Data Access Objects (DAO) are going to be out of data, instead of hand coding, its better to generate the code, using the metadata of the database\nHow # When writing code, there needs to be a way to make sure that, there is no effort spent writing code that keeps changing constantly because its tightly coupled to other objects under development, data access layer is one example of this\nGiven a customer table in the script below and the need to write Create, Read, Update and Delete (CRUD) stored procedures, to access the database.\nCREATE TABLE customer ( customerid NUMBER(18,0) NOT NULL, name VARCHAR2(64) NOT NULL, isactive NUMBER(1) DEFAULT 1 NOT NULL, CONSTRAINT pk_customer PRIMARY KEY (customerid) ); The create customer stored procedure\nCREATE OR REPLACE PROCEDURE Customer_Create (inName Customer.Name%type, inIsActive Customer.isActive%type) AS BEGIN IF inName IS NULL THEN raise_application_error(-21003, \u0026#39;Name cannot be null\u0026#39;); END IF; IF inIsActive IS NULL THEN raise_application_error(-21002, \u0026#39;isActive Cannot be null\u0026#39;); END IF; INSERT INTO Customer (CustomerId,Name,isActive) VALUES (Sequence_Customer.nextval,inName,inIsActive); EXCEPTION WHEN others THEN raise_application_error(-21111, sqlerrm); END; read customer stored procedure\nCREATE OR REPLACE PROCEDURE GetCustomerInformation ( inCustomerId IN Customer.CustomerId%type, customerRecord OUT SYS_REFCURSOR) AS BEGIN OPEN customerRecord FOR SELECT CustomerId, Name, isActive FROM Customer WHERE CustomerId = inCustomerId; EXCEPTION WHEN no_data_found THEN raise_application_error(-21021, \u0026#39;CustomerId not found\u0026#39;); WHEN others THEN raise_application_error(-21000, sqlerrm); end; / Update\nCREATE OR REPLACE PROCEDURE UpdateCustomer ( inCustomerId Customer.CustomerId%type, inName Customer.Name%type, inIsActive Customer.isActive%type) AS BEGIN IF inName IS NULL THEN raise_application_error(-21011, \u0026#39;Name cannot be null\u0026#39;); END IF; IF inCustomerId IS NULL THEN raise_application_error(-21010, \u0026#39;CustomerId cannot be null\u0026#39;); END IF; BEGIN SELECT CustomerID FROM Customer WHERE CustomerID = inCustomerId; EXCEPTION WHEN no_data_found THEN raise_application_error(-21012, \u0026#39;CustomerId not found\u0026#39;); WHEN others THEN raise_application_error(-21000, sqlerrm); END; UPDATE Customer SET Name=inName WHERE CustomerID = inCustomerID; IF sql%rowcount=0 THEN raise_application_error(-21012, \u0026#39;CustomerId not found\u0026#39;); END IF; EXCEPTION WHEN others THEN raise_application_error(-21000, sqlerrm); END; / and now Delete\nCREATE OR REPLACE PROCEDURE DeleteCustomer ( inCustomerId Customer.CustomerId%type) AS BEGIN IF inCustomerId IS NULL THEN raise_application_error(-21020, \u0026#39;CustomerId cannot be null\u0026#39;); END IF; DELETE FROM Customer WHERE CustomerID = inCustomerID; IF sql%rowcount=0 THEN raise_application_error(-21021, \u0026#39;CustomerId not found\u0026#39;); END IF; EXCEPTION WHEN others THEN raise_application_error(-21000, sqlerrm); end; / procedures are shown. Hand coding these stored procedures every time the table changes is really error prone and time consuming. To avoid the errors, its best to generate these stored procedures using the metadata of the database from the USER_TABLES,USER_CONSTRAINTS, USER_TAB_COLUMNS and USER_CONS_COLUMNS in the Oracle database as the code fits a template. Stored procedures can be used to generate this code as shown below\nCREATE OR REPLACE PROCEDURE InformationGenerator(inTableName VARCHAR2) AS BEGIN write(\u0026#39;CREATE OR REPLACE PROCEDURE get\u0026#39;||inTableName||\u0026#39;Information (\u0026#39;); write(\u0026#39;in\u0026#39;||getPrimaryKeyColumn(inTableName)||\u0026#39; \u0026#39;||inTableName||\u0026#39;.\u0026#39;||getPrimaryKeyColumn(inTableName)||\u0026#39;%type,\u0026#39;); write(inTableName||\u0026#39;Record OUT SYS_REFCURSOR)\u0026#39;); write(\u0026#39;AS\u0026#39;); write(\u0026#39;BEGIN\u0026#39;); write(\u0026#39;\tOPEN \u0026#39;||inTableName||\u0026#39;Record FOR SELECT \u0026#39;||getColumnsFor(inTableName)); write(\u0026#39; FROM \u0026#39;||inTableName); write(\u0026#39; WHERE \u0026#39;||getPrimaryKeyColumn(inTableName)||\u0026#39; = in\u0026#39;||getPrimaryKeyColumn(inTableName)||\u0026#39;;\u0026#39;); write(\u0026#39;EXCEPTION\u0026#39;); write(\u0026#39; WHEN no_data_found THEN\u0026#39;); write(\u0026#39; raise_application_error(-21021,\u0026#39;|| \u0026#39;\u0026#39;\u0026#39;Search key cannot be null\u0026#39;\u0026#39;);\u0026#39;); write(\u0026#39; WHEN others THEN\u0026#39;); write(\u0026#39; raise_application_error(-21000, sqlerrm);\u0026#39;); write(\u0026#39;END;\u0026#39;); write(\u0026#39;/\u0026#39;); END; / This InformationGenerator procedure generates the Read customer stored procedure. Once the generator code is written, all that needs to be done is use it on all the tables of the project as shown below\nDECLARE BEGIN FOR tablesList IN (SELECT table_name FROM user_tables) LOOP InformationGenerator(tablesList.table_name); END LOOP; END; / Whenever a table changes all that needs to be done is run the code generator and get up-to-date code. This generated code can then be checked in or distributed as an artifact of the Continuous Integration instance of your team. The same code generator can also be done using a scripting language like ruby as shown below.\ndef generateInformationProcedure(tableName) writeLine(\u0026#34;CREATE OR REPLACE PROCEDURE get\u0026#34;+tableName+\u0026#34;Information (\u0026#34;) writeLine(\u0026#34;in\u0026#34;+getPrimaryKeyColumn(tableName)+\u0026#34; \u0026#34;+tableName+\u0026#34;.\u0026#34;+getPrimaryKeyColumn(tableName)+\u0026#34;%type,\u0026#34;); writeLine(tableName.capitalize+\u0026#34;Record OUT SYS_REFCURSOR)\u0026#34;); writeLine(\u0026#34;AS\u0026#34;); writeLine(\u0026#34;BEGIN\u0026#34;); writeLine(\u0026#34;\tOPEN \u0026#34;+tableName.capitalize+\u0026#34;Record FOR SELECT \u0026#34;+getColumnsFor(tableName)); writeLine(\u0026#34; FROM \u0026#34;+tableName); writeLine(\u0026#34; WHERE \u0026#34;+getPrimaryKeyColumn(tableName)+\u0026#34; = in\u0026#34;+getPrimaryKeyColumn(tableName)+\u0026#34;;\u0026#34;); writeLine(\u0026#34;EXCEPTION\u0026#34;); writeLine(\u0026#34; WHEN no_data_found THEN\u0026#34;); writeLine(\u0026#34; raise_application_error(-21021,\u0026#39;Search key cannot be null\u0026#39;);\u0026#34;); writeLine(\u0026#34; WHEN others THEN\u0026#34;); writeLine(\u0026#34; raise_application_error(-21000, sqlerrm);\u0026#34;); writeLine(\u0026#34;END;\u0026#34;); writeLine(\u0026#34;/\u0026#34;); writeLine(\u0026#34;\u0026#34;); end @connection = OCI8.new(schemaName, dbPassword,dbName) @proceduresFile = File.open(\u0026#39;crudProcedures.sql\u0026#39;, \u0026#39;w\u0026#39;) @connection.exec(\u0026#34;SELECT table_name FROM user_tables\u0026#34;) do | row | generateInformationProcedure(row[0]) end @connection.logoff ","date":"17 May 2025","externalUrl":null,"permalink":"/post/generate_boiler_plate_code/","section":"Posts","summary":"When accessing the database using stored procedures for basic Create, Read, Update and Delete (CRUD) functions, or when you want to write triggers that capture the before and after values in tables, or when you want to create Plain Old Java Objects (POJO’s) that match the database objects, writing these by hand takes a lot of effort and also since the requirements are changing in an agile projects at a frequent rate, there will be design changes to meet the requirement changes, so the triggers, CRUD stored procedures or Data Access Objects (DAO) are going to be out of data, instead of hand coding, its better to generate the code, using the metadata of the database\n","title":"Generate boiler plate code","type":"post"},{"content":"Traditionally the data-team is used to sitting in their own area and working for many project teams by handling requests either via a ticketing system or vi email. The hand-over of work or throwing of work over the wall creates knowledge silos and inefficiencies.\nHow # There multiple ways teams can use shared resources from the data team, which also depends on the size of the development team and the size of the data team\nSmall team # In a small team of developers, when the tasks are data related we may have a single person focusing on database related tasks or we may have single dba doing all these tasks for the team. While doing these tasks its better for the data person to pair with the rest of the developers.\nThe data person should be involved while discussion about features are happening, get involved in white boarding solutions. After the discussions tasks such as designing new tables, indexes, views, stored procedures, triggers for the features under development should be done while pairing with the developers.\nOperational tasks such as automating monitoring, writing scripts to take backups, performance tuning queries and optimizing index usage or utilities to put test data or take a subset of the production database. The data person should not be doing all these tasks by themselves as it will create silos of information and during times when they are not able to work, brings the team to a halt. The data person should also be pairing to spread knowledge to the team about doing specific tasks with the database and learn from the developers, how the database is used?\nLarge team # In a large team, if there is a single data person, many tasks may have to be automated such as Sandbox Creation or Publish Data Dictionary, pairing with developers to automate these tasks helps the developers self service much of their database needs. The data person should then be pairing with developers for database design, table design, index design and coding stored procedures, views etc along with the developers. This pairing helps the developers understand the data architecture and makes for a productive team instead of productive silos of knowledge.\nMultiple teams # In situations where we have multiple teams and single data person, effective rotation of the data person becomes critical for the success of the teams. The data person should not be targeting to slice the time equally among the teams but look for effective use of the time on the teams. Since each team is going through different development phases they may not need the same attention from the data person.\nWhen the data person is with a given team they should pair with the team members to accomplish the task at hand. They should also enable the team members to pull them into a quick conversation/discussion as and when needed, so that the team does not experience wait times\nMultiple teams and data team # In many companies we have a shared database support teams comprizing of DBA, data modellers, data architects, data developers and others. In such a situation its difficult to allocate the data team members to development teams by role during a iteration.\nAllows data specialists to be utilized across multiple projects The figure above shows a team of specialists in data, providing service to large number of project teams. When the teams need data related service they should ask for help from the shared data team and then pair with the data team member. This data team member after returning back to the shared team should share the knowledge of the work done on the project team so that anyone from the data team is able to help the project team the next time instead of being dependent on the same individual.\nData team members going from one project team to another also allows for them to share knowledge about what other project teams are doing and talk to them about best practices, design decisions being taken and data entities and attributes available from the other project teams.\n","date":"1 August 2024","externalUrl":null,"permalink":"/post/pair-with-developers/","section":"Posts","summary":"Traditionally the data-team is used to sitting in their own area and working for many project teams by handling requests either via a ticketing system or vi email. The hand-over of work or throwing of work over the wall creates knowledge silos and inefficiencies.\n","title":"Data specialists should pair with developers","type":"post"},{"content":"Many a times ER models are created by the data team and are not shared outside of the data team generally for the lack of tools licenses, since its not feasible for the entire team to purchase licenses for the ER modelling tools such as Erwin Data Modeller or Er Studio\nHow # In an agile development environment as the requirements are evolving over the duration of the project, the database design is being evolved along with the business requirements. While the database design is being changed the data team generally maintains a data model, its share this data model with the rest of the development team for better collaboration and knowledge sharing.\nThe data team working with the developers change the database using migration scripts, while doing the changes to the database the data model can be updated to ensure both are synchronized. The step to update the data model may seem redundant or unnecessary sometimes, but maintaining a data model provides an easy way to communicate about the database design and allows the entire team to find entities(tables) and attributes(columns) and how they related to each other. This allows the team to use the database without creating duplicate attributes and use them where available.\nVersion control data model # Along with the database change scripts and other scripts about the database, the data team should also be putting data models under version control, this allows the whole team access to the data model, see changes being made and be able to view the data model and understand how the tables and columns are related, what all tables and columns are already available to be used. [[ermodel]] .Data model version controlled along with the rest of the database artifacts image::chapter4/ERModel.png[]\nPublish ER Models on project wiki # Many a times, it not possible to buy licenses to ER modelling tools for the entire team, in this scenario the data team can start publishing data model printouts, generally this was done using large printouts pinned to walls, these large printouts get out of date immediately after they are printed and mislead the team.\nIt is better to publish pdf versions of the ER model on the internal project wiki and kept up to date whenever changes happen, this allows for the entire team to have a view at the data model and kept abreast with the changes happening in the data model. The same wiki based data models can help downstream consumers of the database to understand the data model and use it in an effective way\nAccess to data models is simplified Shown above is an example of a project wiki page where the data team split the data model into smaller subject areas and saved the model as pdf files and can be viewed by the whole team. This page also shows the last date the data model pdf\u0026rsquo;s where updated so that the team is aware when the last update happened.wha\n","date":"16 September 2023","externalUrl":null,"permalink":"/post/publish-data-models/","section":"Posts","summary":"Many a times ER models are created by the data team and are not shared outside of the data team generally for the lack of tools licenses, since its not feasible for the entire team to purchase licenses for the ER modelling tools such as Erwin Data Modeller or Er Studio\n","title":"Publish data models in CI Pipeline","type":"post"},{"content":"","date":"4 December 2020","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"4 December 2020","externalUrl":null,"permalink":"/tags/trust/","section":"Tags","summary":"","title":"Trust","type":"tags"},{"content":"Search engines have been great to find information on the internet with their ease of use and the ability to find cross linked content with specific keywords. This ease of use is being exploited by scammers and imposters.\nFake or spoofed websites are being setup for reputable companies (especially mass consumer products), and the links are being gamed so that the phishing websites appear as top hits. Here is an example of a google search for fixing errors with remote scanning on canon printer canon scanner errors\nYou can see the search results and the first result is from a website canonprintersupport247.com when you click on this link, you can see that its not from official Canon company or someone trust worthy as the address in the Contact US link is suspect\nCalling the number listed, is picked up by a human immediately without any company announcements or menu options etc.\nWho should be responsible for sanitizing the search results, how can the search engines ensure trust worthy results are shown? could there be ML/AI algorithms that can provide a trust score (similar to email spam filters), maybe a score right after the link to the website, this will stop people from gaming the link travesal gaming.\n","date":"4 December 2020","externalUrl":null,"permalink":"/post/trusted-search-results/","section":"Posts","summary":"Search engines have been great to find information on the internet with their ease of use and the ability to find cross linked content with specific keywords. This ease of use is being exploited by scammers and imposters.\n","title":"Trusting search results from Google \u0026 Others","type":"post"},{"content":"","date":"15 July 2019","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"Aws","type":"tags"},{"content":"","date":"15 July 2019","externalUrl":null,"permalink":"/tags/cloud/","section":"Tags","summary":"","title":"Cloud","type":"tags"},{"content":"","date":"15 July 2019","externalUrl":null,"permalink":"/tags/developer/","section":"Tags","summary":"","title":"Developer","type":"tags"},{"content":"On a recent project we had to connect to AWS Aurora postgres 10.6 version of the database in SSL mode using JDBC and Java 11 JRE. When the Aurora cluster is setup, we can force all connections to use SSL by using the options group settings (forceSSL=true), establishing secure connection from the application to the database is not as easy as it looks.\nHere are the steps we took to make this work.\nAWS provides certificates that you can download Certs. These cannot be used directly. Convert the .pem file downloaded to a .der file using openssl openssl x509 -outform der -in your-cert.pem -out your-cert.crt Copy the converted .der file to $JAVA_HOME/lib/security folder Now import the .der file using the keytool command. keytool -importcert -file $JAVA_HOME/lib/security/rds-combined-ca-bundle.crt -cacerts -storepass mypassword alias awsaurora -noprompt After this step the certificate is in the JVM SSL Factory, the JVM has access to the certificate,\nSince there is a bug in the postgres JDBC driver (it does not access the default Java SSL Factory), we have to provide that in the JDBC connection string as shown below. jdbc:postgresql://server-url:5432/database-name?currentSchema=application-schema\u0026amp;ssl=true\u0026amp;sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory The setting ssl=true turns on SSL connection and sslfactory=org.postgresql.ssl.DefaultJavaSSLFactory tells the JDBC driver where to look for the certificate. Its always a good practice to set the currentSchema to some value as the default will be public which is not a good idea, in the above setting we have currentSchema=application-schema\n","date":"15 July 2019","externalUrl":null,"permalink":"/post/ssl-aws-aurora/","section":"Posts","summary":"On a recent project we had to connect to AWS Aurora postgres 10.6 version of the database in SSL mode using JDBC and Java 11 JRE. When the Aurora cluster is setup, we can force all connections to use SSL by using the options group settings (forceSSL=true), establishing secure connection from the application to the database is not as easy as it looks.\n","title":"SSL Connection to AWS Aurora","type":"post"},{"content":"","date":"1 August 2017","externalUrl":null,"permalink":"/tags/oracle/","section":"Tags","summary":"","title":"Oracle","type":"tags"},{"content":"In all new development and sometimes during legacy codebase modernization, developers tend to add code quality checks and static analysis of codebase such as style checks, bug finders, cyclomatic complexity checking etc. into the CI/CD pipeline. When we inherit a codebase that has much PL/SQL and there is a desire to put the PL/SQL code base through the same types of code analysis, what options does a developer/dba have?\nThere are some options we can explore such as\nPMD ClearSQL PL/SQL Cop Toad for Oracle This example shows PL/SQL cop, PL/SQL Cop provides, code checkstyle like checks, code quality checks with McCabe\u0026rsquo;s cyclomatic complexity and the Halstead metrics, find bugs equivalent checks. PL/SQL cop works on the command line or can be integrated into Sonar cube in the build pipeline or Continuous Integration pipeline.\ntvdcc.sh path=code excel=false html=true cleanup=true Using the above command, PL/SQL cop checks all PL/SQL code in the code folder and provides output in html format. The summary stats provided are Each file gets a detailed analysis of the output, along with code excerpts for which the exception is being raised. Using static analysis tools for PL/SQL code provides the team with confidence of the state of the code base and ensures that all code is checked and verified. It also ensures that the PL/SQL code is put through the same build pipeline that other parts of the application are being put through.\n","date":"1 August 2017","externalUrl":null,"permalink":"/post/static-analysis-of-plsql-code/","section":"Posts","summary":"In all new development and sometimes during legacy codebase modernization, developers tend to add code quality checks and static analysis of codebase such as style checks, bug finders, cyclomatic complexity checking etc. into the CI/CD pipeline. When we inherit a codebase that has much PL/SQL and there is a desire to put the PL/SQL code base through the same types of code analysis, what options does a developer/dba have?\n","title":"Static Analysis of PL/SQL code","type":"post"},{"content":"","date":"12 June 2017","externalUrl":null,"permalink":"/tags/design/","section":"Tags","summary":"","title":"Design","type":"tags"},{"content":"","date":"12 June 2017","externalUrl":null,"permalink":"/tags/framework/","section":"Tags","summary":"","title":"Framework","type":"tags"},{"content":"In this blog post, I will discuss ID generation techniques using the Object Relation Mapping frameworks such as Hibernate, Toplink, ActiveRecord, Entity Framework. When using hibernate or any other ORM mapping framework. There is a need to generate primary key values for the \u0026ldquo;id\u0026rdquo; columns. These values can be generated by using IDENTITY, SEQUENCE or TABLE strategies. Generating custom values for primary keys or other values is a topic for another blog post.\nIn this blog post the examples are using Hibernate, JPA, Java8 and Oracle12c.\nTABLE Id generation strategy: # When using this strategy, a table in the database is used to store the current id value and the next value is generated and written back to the table.\nCREATE TABLE hibernate_sequences( sequence_name VARCHAR2(40) NOT NULL, next_val NUMER(18) NOT NULL ) The above table is used by hibernate to generate next_val for sequence_name specified in the entity mapping, if no sequence_name is specified then default is used.\nThe ID generation involves a transaction for its own use and care has to be taken when multiple processes are generating ID\u0026rsquo;s, and involves locking the ROW in the table. This strategy is the least efficient and should be avoided.\nIDENTITY Id generation strategy: # Identity columns in databases like MySQL, MSSQL, PostgreSQL and Oracle 12c are used to auto generate numeric values for a number column. There can be only one identity column in a table, the column has to be not-null and cannot accept any other default values. Identity columns are generally used as synthetic primary key columns. Shown below is an example of creating a table with identity column id in Oracle12c.\nCREATE TABLE person ( id NUMBER(18) GENERATED ALWAYS AS IDENTITY INCREMENT BY 1 START WITH 1 NOT NULL, name VARCHAR2(40) NOT NULL ); ALTER TABLE person ADD CONSTRAINT pk_person PRIMARY KEY (id) ; The id column from the person table can now be mapped to the Person entity. The generation type we are using is IDENTITY and hibernate now understands that the database will provide the value for id at insert time.\n@Id @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; Mapping in Hibernate is simple, without having to specify any sequence names, All inserts into the table automatically use the same method for getting ID\u0026rsquo;s either through the application or directly into the database by other users.\nSince ID\u0026rsquo;s are assigned at insert time, JDBC batching does not work when using identity columns. In Oracle12c dropping of tables does not drop the sequence associated with the identity column leaving stale sequences in the database. In a Parent and Child relationship table such as Person and all their Addresses table, the child.parent_id column cannot be \u0026ldquo;not null\u0026rdquo; as hibernate (the ORM framework) cannot get the ID of the parent before the parent is inserted, hence it does these operations\ninsert into parent_table insert into child_table select parentId from parent_table select childId from child_table update child table with parentId for childId. Identity ID generation does not support preallocation, so requires a select after each insert, increasing the trips to the database and degrading insert performance.\nSEQUENCE Id generation strategy: # Sequences in Oracle, Postgres and SQLServer 2012 are database objects that generate unique numbers and are generated independent of transactions. Sequences improve the concurrency of number generation. Sequences can be increment by any number, can start at any number, can cycle the numbers after maximum number is reached, have upper limit on the number generated and can cache certain number of values. Shown below is an example of creating sequence in Oracle12c.\nCREATE SEQUENCE seq_person START WITH 10000 INCREMENT BY 1; CREATE TABLE PERSON ( id NUMBER(18) NOT NULL, NAME VARCHAR2(40) ); ALTER TABLE PERSON ADD CONSTRAINT PK_PERSON PRIMARY KEY (ID) ; Using the sequence above and the table definition, we can map the table with the object and use the SEQUENCE GenerationType and the sequence name seq_person. When a new Person is created, hibernate will get the next_id from the sequence and assign it to id attribute on the object.\n@Id @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = \u0026#34;person_generator\u0026#34;) @SequenceGenerator(name=\u0026#34;person_generator\u0026#34;, sequenceName = \u0026#34;seq_person\u0026#34;) private Long id; Sequences provide a easy way to create an object to provide unique numbers and are not bound by transaction boundaries. Various configuration options, such as start number, increment interval, cache size, max number, min number, cycle the numbers etc.\nORM frameworks can use batch operations for inserts as they can get the sequence values for ObjectID\u0026rsquo;s without inserting the data in the tables.\nApplications have to remember to use the correct sequence to insert the data in the table, hibernate mapping is somewhat verbose. Other utilities have to remember to use the correct sequence when inserting data directly into the database.\nRecommendation: # Based on the above observations and usage:\nIf the database provides features to use sequences they should be used. In databases that do not provide sequences identity columns have to be used.\nIf the database provides both identity and sequence features. The decision depends on the amount of batch operations being performed and the need to provide consistent primary key generation for new data from the application and other utilities loading data directly in the database.\n","date":"12 June 2017","externalUrl":null,"permalink":"/post/id-generation/","section":"Posts","summary":"In this blog post, I will discuss ID generation techniques using the Object Relation Mapping frameworks such as Hibernate, Toplink, ActiveRecord, Entity Framework. When using hibernate or any other ORM mapping framework. There is a need to generate primary key values for the “id” columns. These values can be generated by using IDENTITY, SEQUENCE or TABLE strategies. Generating custom values for primary keys or other values is a topic for another blog post.\n","title":"ID generation with ORM's Table, Sequence or Identity strategy","type":"post"},{"content":"","date":"15 May 2017","externalUrl":null,"permalink":"/tags/database/","section":"Tags","summary":"","title":"Database","type":"tags"},{"content":"Loading data into tables is needed many times on projects to load test, Liquibase provides a method to load data into tables with lots of customization. In the example shown below, I\u0026rsquo;m loading zip code data with the following column layout\n\u0026#34;Zipcode\u0026#34;,\u0026#34;ZipCodeType\u0026#34;,\u0026#34;City\u0026#34;,\u0026#34;State\u0026#34;,\u0026#34;LocationType\u0026#34;,\u0026#34;Lat\u0026#34;,\u0026#34;Long\u0026#34;,\u0026#34;Location\u0026#34;,\u0026#34;Decommisioned\u0026#34;,\u0026#34;TaxReturnsFiled\u0026#34;,\u0026#34;EstimatedPopulation\u0026#34;,\u0026#34;TotalWages\u0026#34; The alphanumeric data is enclosed in \u0026quot; and separated by , the first line in the data file is a row of headers for the columns of data. A snapshot of the data in the file is shown below\n\u0026#34;02108\u0026#34;,\u0026#34;STANDARD\u0026#34;,\u0026#34;BOSTON\u0026#34;,\u0026#34;MA\u0026#34;,\u0026#34;PRIMARY\u0026#34;,42.35,-71.06,\u0026#34;NA-US-MA-BOSTON\u0026#34;,\u0026#34;false\u0026#34;,2348,3312,388783474 \u0026#34;02109\u0026#34;,\u0026#34;STANDARD\u0026#34;,\u0026#34;BOSTON\u0026#34;,\u0026#34;MA\u0026#34;,\u0026#34;PRIMARY\u0026#34;,42.35,-71.06,\u0026#34;NA-US-MA-BOSTON\u0026#34;,\u0026#34;false\u0026#34;,2966,4145,284385612 \u0026#34;02110\u0026#34;,\u0026#34;STANDARD\u0026#34;,\u0026#34;BOSTON\u0026#34;,\u0026#34;MA\u0026#34;,\u0026#34;PRIMARY\u0026#34;,42.35,-71.06,\u0026#34;NA-US-MA-BOSTON\u0026#34;,\u0026#34;false\u0026#34;,2950,4313,231268950 \u0026#34;02111\u0026#34;,\u0026#34;STANDARD\u0026#34;,\u0026#34;BOSTON\u0026#34;,\u0026#34;MA\u0026#34;,\u0026#34;PRIMARY\u0026#34;,42.35,-71.06,\u0026#34;NA-US-MA-BOSTON\u0026#34;,\u0026#34;false\u0026#34;,2964,4467,315024986 The standard dataset for the zipcode file has more than 45k rows. These rows can be loaded using the loaddata command of liquibase. We can ignore some of the columns in the datafile if we are not interested in the data.\n\u0026lt;changeset author=\u0026#34;pramod\u0026#34; id=\u0026#34;45\u0026#34;\u0026gt; \u0026lt;loadData file=\u0026#34;free-zipcode-database-Primary.csv\u0026#34; quotChar=\u0026#34;\u0026amp;quot;\u0026#34; seperator=\u0026#34;,\u0026#34; tableName = \u0026#34;zipcode\u0026#34; \u0026gt; \u0026lt;column header=\u0026#34;zipcode\u0026#34; name=\u0026#34;zipcode\u0026#34; type=\u0026#34;string\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;ZipCodeType\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;City\u0026#34; name=\u0026#34;city\u0026#34; type=\u0026#34;string\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;State\u0026#34; name=\u0026#34;state_code\u0026#34; type=\u0026#34;string\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;LocationType\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;Lat\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;Long\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;Location\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;Decommisioned\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;TaxReturnsFiled\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;EstimatedPopulation\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;column header=\u0026#34;TotalWages\u0026#34; type=\u0026#34;skip\u0026#34;/\u0026gt; \u0026lt;/loadData\u0026gt; \u0026lt;/changeset\u0026gt; As seen above we are ignoring some of the data in the datafile using the skip type and we are mapping the header column names to columns the table by specifying the header and name attributes of the column element.\nThis is a much more efficient method of loading data instead of creating SQL statements or going to external commands such as sql loader or other database specific tools as the loadData command can be wrapped up in the developer tools and pipelines.\n","date":"15 May 2017","externalUrl":null,"permalink":"/post/using-liquibase-to-load-data-and-ignore-some-columns/","section":"Posts","summary":"Loading data into tables is needed many times on projects to load test, Liquibase provides a method to load data into tables with lots of customization. In the example shown below, I’m loading zip code data with the following column layout\n\"Zipcode\",\"ZipCodeType\",\"City\",\"State\",\"LocationType\",\"Lat\",\"Long\",\"Location\",\"Decommisioned\",\"TaxReturnsFiled\",\"EstimatedPopulation\",\"TotalWages\"","title":"Using liquibase to load data and ignore some columns","type":"post"},{"content":"","date":"13 July 2016","externalUrl":null,"permalink":"/tags/data/","section":"Tags","summary":"","title":"Data","type":"tags"},{"content":" In todays environment, identity theft and related crimes are frequent and common place. Most of these incidents take place because our identity can be easily stolen will little bits of information such as name, address, birthdate, social security number or maybe just your card number.\nWhat if, credit file or credit report or credit score even credit cards had two factor authentication, just like we have, two factor authentication on gmail, chase and many other services.\nIn the diagram above when an Individual Citizen makes an application to get a loan or rent an apartment, this action creates the following flow of information\nThe application the individual fills, has personal identifying information and is submitted to the institutions providing the loan or the rental. These institutions in turn make a request to the credit rating agencies with all the personal information. The credit agencies, using the personal information look up the credit history and calculate the score for the individual and return the response with the credit history and/or score. The institutions now can determine based on the credit history and/or score, if to proceed with the transaction and determine the rate of interest or other parameters. In the whole chain on events there is an implicit assumption that the individual making the request is the owner of the personal information being submitted, when the individual making the request is not the owner of the personal information identity theft credit card theft take place.\nAs a technology person that has grown to like two factor authentication I wonder why cannot it be applied to transactions involving credit card, credit files, credit history and identity related information. In the diagram above when an Individual Citizen makes an application, the credit agencies or the credit information requesting organizations can/should get conformation from the Individual Citizen via an alternate method such as SMS, EMAIL, Smart Phone App, Phone call or any other mode of communication.\nOnce the Individual Citizen approves the request as valid and initiated by them, the request should be fulfilled with valid data, if the Individual Citizen rejects the request as invalid or notifies that it was not initiated by them then the request can be rejected without putting the Individual Citizens information at risk.\nTwo factor authentication is very beneficial for the financial and other organizations as it increases the trust level of the transaction and reduces their losses because of stole identity. The Two factor authentication service can be implemented by either an independent organization or the credit rating agencies with enough independent information about the individual people.\n","date":"13 July 2016","externalUrl":null,"permalink":"/post/two-factor-authentication-to-authorize-credit/","section":"Posts","summary":" ","title":"Two factor authentication to authorize credit\"","type":"post"},{"content":"Many projects need addition of identical columns to all the tables created by the project. Audit columns are an example of such a requirement. The requirement is to add columns such as created_by, created_date, modified_by and modified_date to all the tables, these columns store, who created the row, when the row was created, who modified the row last and when was it modified. created_by and created_date are required to be present when the row is inserted and thus are required to be not nullable. Adding these columns to each and every table is a lot of work for developers.\nWhen creating the new migration, we would add all the required columns for every migration we create as shown below\nclass CreateCustomer \u0026lt; ActiveRecord::Migration def change execute \u0026#39;CREATE SEQUENCE SEQ_CUSTOMER\u0026#39; create_table :customer, primary_key: \u0026#39;customer_id\u0026#39;, sequence_name: \u0026#39;seq_customer\u0026#39;, id: false do |t| t.integer :customer_id, limit: 8 t.string :name, null: false t.string :email, limit: 200 t.datetime :date_joined, null: false t.string :active_flag, default: \u0026#39;Y\u0026#39;, limit: 1 t.string :created_by, null: false t.datetime :created_date, null:false t.string :modified_by, null:true t.datetime :modified_date, null:true end execute \u0026#39;ALTER TABLE CUSTOMER MODIFY DATE_JOINED DEFAULT SYSDATE\u0026#39; end end Adding the four columns for every table is repetitive and could be easily forgotten, creating follow-on migrations and data loss. Adding a ActiveRecord method like audit_columns that automatically adds the default columns when a new table is being created saves the developers from remembering about the four columns and also enables them to have a standard definition of these columns.\nmodule ActiveRecordTableDefinitionExtension extend ActiveSupport::Concern included do def audit_columns(options={}) column(:created_by, :string, null: false) column(:created_date, :datetime, null:false) column(:modified_by, :string) column(:modified_date, :datetime ) end end end Once the ActiveSupport method is defined, we can add the new customer table and reference the audit_columns method, which will automatically add the four columns when the migration is run\nclass CreateCustomer \u0026lt; ActiveRecord::Migration def change execute \u0026#39;CREATE SEQUENCE SEQ_CUSTOMER\u0026#39; create_table :customer, primary_key: \u0026#39;customer_id\u0026#39;, sequence_name: \u0026#39;seq_customer\u0026#39;, id: false do |t| t.integer :customer_id, limit: 8 t.string :name, null: false t.string :email, limit: 200 t.datetime :date_joined, null: false t.string :active_flag, default: \u0026#39;Y\u0026#39;, limit: 1 t.audit_columns end execute \u0026#39;ALTER TABLE CUSTOMER MODIFY DATE_JOINED DEFAULT SYSDATE\u0026#39; end end We still have to run the DEFAULT SYSDATE command as a raw sql as we want the default to be a function and not a literal date. Similar techniques can be applied when other types of columns are required to be present on a large number of tables.\n","date":"25 April 2016","externalUrl":null,"permalink":"/post/automatically-adding-columns-to-rails-migrations/","section":"Posts","summary":"Many projects need addition of identical columns to all the tables created by the project. Audit columns are an example of such a requirement. The requirement is to add columns such as created_by, created_date, modified_by and modified_date to all the tables, these columns store, who created the row, when the row was created, who modified the row last and when was it modified. created_by and created_date are required to be present when the row is inserted and thus are required to be not nullable. Adding these columns to each and every table is a lot of work for developers.\n","title":"Automatically adding columns to Rails migrations","type":"post"},{"content":"In many development shops, developers are not allowed to access the database schema directly, and are not allowed to create tables, indexes, views etc, instead are given access via a different schema that allows SELECT, UPDATE and DELETE access on data. The general reason is to avoid developers creating database objects without\nIn oracle this is implemented using a schema to hold all the objects, e.g. ecommercedb and the application (code, developers) get access to the ecommercedb schema using the ecommercedb_rw schema that has SELECT, UPDATE and DELETE privileges on the objects in the ecommercedb schema, in production developers get access to the ecommercedb schema via the ecommercedb_read schema, which has SELECT only access to the ecommercedb schema.\nIn the above setup, the database objects in the base schema ecommercedb can be accessed using the schema name as a prefix as shown below.\nSELECT customer.id, customer.name, customer.status .... FROM ecommercedb.Customer as customer The name of the schema owning the objects is spread all over the SQL being used in application and this is trouble as it does not provide the flexibility to change the schema names and it also makes the database environment not flexible to allow more than one application environment in the same database. Substituting the schema name with an variable is an acceptable solution, but does get more complex as more schemas are added to the mix.\nA much more elegant solution is to use SYNONYMS, which allow to create alternate names to database objects that belong to other schemas or even to the same schema. In our ecommercedb example, create a synonym in ecommercedb_rw and ecommercedb_read schemas that are pointing to the ecommercedb schema as shown below\nCREATE SYNONYM CUSTOMER for ecommercedb.CUSTOMER After the above step, we can change our sql as shown below and we no longer need to reference the schema name in the SQL, thus allowing us to have as many application environments on the same database and also let each database server decide how they want to name the database schema.\nSELECT customer.id, customer.name, customer.status .... FROM customer Manually creating and maintaining all these SYNONYMS in version control is too much work and should be automated using a build task, an example using Rake is shown below\nnamespace :db do desc \u0026#39;Generate Synonyms for all tables\u0026#39; task synonym: :environment do base_schema = \u0026#39;ECOMMERCEDB\u0026#39; read_role = \u0026#39;ECOMMERCEDB_READ\u0026#39; read_write_role = \u0026#39;ECOMMERCEDB_RW\u0026#39; grants_dir = create_folder(\u0026#39;grants\u0026#39;) synonym_dir = create_folder(\u0026#39;synonyms\u0026#39;) grantsfile = File.open(\u0026#34;#{grants_dir.getwd()}grants.sql\u0026#34;, \u0026#39;w\u0026#39;) synonymfile = File.open(\u0026#34;#{synonym_dir.getwd()}synonym.sql\u0026#34;,\u0026#39;w\u0026#39;) sql = \u0026#39;SELECT table_name as tablename FROM user_tables\u0026#39; tables = ActiveRecord::Base.connection.exec_query(sql) tables.each do |table| name = table[\u0026#39;tablename\u0026#39;] grantsfile.puts \u0026#34;GRANT SELECT ON #{name} TO #{read_role}, #{read_write_role};\u0026#34; grantsfile.puts \u0026#34;GRANT UPDATE,INSERT,DELETE ON #{name} TO #{read_write_role};\u0026#34; synonymfile.puts \u0026#34;CREATE OR REPLACE SYNONYM #{name} FOR #{base_schema}.#{name};\u0026#34; end grantsfile.close() synonymfile.close() puts \u0026#34;Generated grants, synonyms for tables\u0026#34; end end The synonym rake task above generates the GRANTS and the SYNONYMS for all the tables in the base ecommercedb, similar scripts can be generated for the rest of the database objects such as sequences, views, materialized views, functions and stored procedures\n","date":"15 April 2016","externalUrl":null,"permalink":"/post/synonyms-as-abstraction-layer/","section":"Posts","summary":"In many development shops, developers are not allowed to access the database schema directly, and are not allowed to create tables, indexes, views etc, instead are given access via a different schema that allows SELECT, UPDATE and DELETE access on data. The general reason is to avoid developers creating database objects without\n","title":"Synonyms as abstraction layer","type":"post"},{"content":"","date":"16 August 2015","externalUrl":null,"permalink":"/tags/bdd/","section":"Tags","summary":"","title":"BDD","type":"tags"},{"content":"When Behavior Driven Development BDD was introduced, some of the key principles were\nRequirements are behavior, Provides \u0026ldquo;ubiquitous language\u0026rdquo; for analysis, Acceptance criteria should be executable. Design constraints should be made into executable tests. All of these principles can be applied to database development. When interacting with the database, we tend to assume certain behavior of the database. Some of this is universal, like when a row is inserted in a table, the same row can later be retrieved. There are other behaviors of the database on every project that are not that universal, like Person table should have at least firstname or lastname. This behavior changes based on the functionality being developed and thus needs to be properly specified and executed. The database lends itself very well to the new way of thinking in the BDD space, where the behavior of the objects is considered. BDD is similar to describing requirements in code.\nHere we are going to see how the BDD techniques can be applied to database development and how these techniques can be used to develop and design databases, in an iterative and incremental way. Lets call these technique as Behavior Driven Database Development (BDDD).\nWhile designing database objects, we are expecting these objects to behave in a certain fashion and we tend to rely on this behavior. Let\u0026rsquo;s say we make a column NOT NULLABLE. We assume that the database server will throw an exception when a NULL value in inserted in this column, making the same column NULLABLE later, can alter this behavior of the database. When the behavior of the database is changed this way, all the assumptions that the application made about the database NOT allowing NULL values in the column are no longer true. To avoid these kinds of mistakes in assumptions of behavior, we can test the database behavior to assert that the database does throw an exception when NULL values are put in column.\nIn this blog article, lets talk about\nDesign a Table Design a Primary Key Design a Not Null Column Design a Constraint on a Column Design a Foreign Key Constraint Design a Sequence for Object ID Design a Unique Index Lets assume we are building a Movie Rental system to be deployed at stores (not many of them are around nowadays). The store wants to track what DVD’s they have and who has rented the DVD’s. For simplicity sake lets assume we are using Java, Hibernate, Oracle and JUnit as some of the technologies. You can substitute these technologies with any others you like.\nDesign a Table # When starting to work on the feature \u0026ldquo;As a Store Manager I should be able to select a DVD to rent\u0026rdquo;. The attributes of the movie Object and movie table have to be decided first. Once you decide the attributes, these attributes need to be mapped in Hibernate mappings.Start with a test first that tries to create a domain object, save the domain object and fetch the domain object back from the database. Let\u0026rsquo;s see the test\n@Test public void ShouldBeCreatedAndSavedSuccessfully() { Movie movie = new Movie(); movie.setName(name); saveDomainObject(movie); assertNotNull(\u0026#34;Insert failed\u0026#34;, movie); } The above test, just verifies that it can create a Movie domain object and then save it in the database, the behavior we are driving out here is that a valid Movie domain object can be saved. At this stage the database script for Movie table looks like\nCREATE TABLE movie ( movieid NUMBER(18), name VARCHAR2(64) ); Design a Primary Key # The next task we want do is to make sure the MOVIE table has a Primary Key and there is a value assigned to the primary key when the MOVIE domain object is saved. We can also check the Hibernate mapping behavior to make sure that the correct SEQUENCE is used to assign the next ID to the MOVIE object. The test looks like.\n@Test public void shouldAssignPrimaryKeyValuesFromSequence() { Movie movie = createAndSave(\u0026#34;PKASSIGNED\u0026#34;); Long currentPKValue = getCurrentValueForSequence(\u0026#34;S_MOVIE\u0026#34;); assertNotNull(\u0026#34;Movie should have ID\u0026#34;, movie.getId()); assertEquals(\u0026#34;ID should be same\u0026#34;,movie.getId(),currentPKValue); } The above test shouldAssignPrimaryKeyValuesFromSequence checks the behavior of the Hibernate mapping, the Primary Key constraint and makes sure that the correct sequence S_MOVIE is used to populate the ID value for the Movie object. At this stage the database script for the Movie table and S_MOVIE sequence looks like\nCREATE TABLE movie ( movieid NUMBER(18), name VARCHAR2(64), CONSTRAINT pk_movie PRIMARY KEY (movieid) ); CREATE SEQUENCE s_movie; Design a Not Null Column # The next feature to work on is \u0026ldquo;As a Internal User I should be able to assign a year the movie was made\u0026rdquo;. Lets also say one of the requirements is to make sure that every movie has to have the year it was made. We will use Make Column Non Nullable database refactoring on the database. Starting with the test as shown below.\n@Test public void shouldNotAllowNullYear() { Movie movie = new Movie(); movie.setName(name); try { saveDomainObject(movie); fail(); } catch (ConstraintViolationException e) { assertContains(e,\u0026#34;ORA-01400: cannot insert NULL\u0026#34;); } } As can be seen in the above example, the behavior of the database to throw an exception when one of the rules set on the table is not satisfied. If this assumption That the database does not allow NULL values in the year column is enforced by the test ShouldNotAllowNullYear, when this assumption on the database is changed this test will fail. During refactoring of the database, these tests help to enforce the assumptions made on the database. After this stage the database table looks like\nCREATE TABLE movie ( movieid NUMBER(18), name VARCHAR2(64), year VARCHAR2(4) NOT NULL, CONSTRAINT pk_movie PRIMARY KEY (movieid) ); Design a Constraint on a Column # The next feature to work on is The store does not carry any movies made before 1999. Lets start with Introduce Column Constraint database refactoring to create a column level constraint on the YEAR column so that it does not allow any value less than 1999.\n@Test public void shouldNotAllowMovieYearBeforeYear1999() { try { createAndSave (\u0026#34;1998\u0026#34;, \u0026#34;YearBefore\u0026#34;); fail(\u0026#34;Movie year is before 1999\u0026#34;); } catch (Exception e) { assertContains(e,\u0026#34;CHK_MOVIEYEAR_GT_1998\u0026#34;); } } The behavior of the database to disallow values before 1999 is asserted by the test \u0026ldquo;ShouldNotAllowMovieYearBeforeYear1999\u0026rdquo;. At this stage the database script looks like\nCREATE TABLE movie ( movieid NUMBER(18), name VARCHAR2(64), year VARCHAR2(4) NOT NULL, CONSTRAINT pk_movie PRIMARY KEY (movieid), CONSTRAINT chk_movieyear_gt_1998 CHECK( year \u0026gt; ‘1998’) ); Design a Foreign Key Constraint # The next feature to work on is Movie has details about itself that need to persisted. To store the details about the movie we will Introduce New Table MovieDetail with a MovieId on the MovieDetail table. So effectively we will have a collection of MovieDetail objects on the Movie object. Having a MovieDetail object created without the MovieId on would create dirty data in the MovieDetail table, so we will Add Foreign Key Constraint on the MoveDetail table. Having a NULL MoveId would also invalidate the MoveDetail object, since a MovieDetail cannot exist without Movie, making the MoveDetail.MovieID not null.\nThis assertion can be done by the domain object, but I have seen over years of consulting at many different companies that the database gets used eventually without the domain layer (Reporting, Data Extract, Data Import etc.), so it’s a better to move these kinds of constraints on the database. The following test will ensure that the database behavior matches what we expected.\n@Test public void shouldNotAllowMovieDetailsToExistWithOutMovie() { Movie movie = createAndSave(\u0026#34;DoomsDay\u0026#34;); MovieDetail detail = new MovieDetail(); detail.setDescription(\u0026#34;DoomsDayMovie\u0026#34;); detail.setMovie(movie); detail.setUrl(\u0026#34;www.doomsday.com\u0026#34;); saveDomainObject(detail); try { removeDomainObject(movie); fail(); } catch (Exception e) { assertContains(e, \u0026#34;FK_MOVIEDETAIL_MOVIE\u0026#34;); } } The above test tries to delete the Movie object from the database when it has the MoveDetail as its child, this forces the database to raise a Foreign Key violation, which is verified by the test, similarly we can write a test where the MoveDetail.MovieId is null.\n@Test public void shouldNotAllowMovieDetailsToExistWithNullMovie() { MovieDetail detail = new MovieDetail(); detail.setDescription(\u0026#34;DoomsDayMovie\u0026#34;); detail.setMovie(null); detail.setUrl(\u0026#34;http://doomsday.com\u0026#34;); try { saveDomainObject(detail); fail(); } catch (Exception e) { assertContains(e, \u0026#34;cannot insert NULL into\u0026#34;); } } At this stage the database script looks like\nCREATE TABLE moviedetail ( moviedetailid NUMBER(18), movieid NUMBER(18) NOT NULL, description VARCHAR2(4000), url VARCHAR2(400), CONSTRAINT pk_moviedetail PRIMARY KEY (moviedetailid) ); ALTER TABLE moviedetail ADD CONSTRAINT fk_moviedetail_movie FOREIGN KEY (movieid) REFERENCES movie; Design a Sequence for ObjectId # The next feature to work on is a Technical Story to reduce database trips to get the ObjectID for every new Object created. To reduce the number of database trips for every object created, we can create a sequence (Oracle database specific), which returns values in Increments of 1000. Then we can make the application return one ID at a time from the value that was returned by the database, reducing the number of round trips to the database from the application. When the application exhausts the 1000 IDs it asks the database for the next 1000.\nWhich means that we are relying on the database behavior to return values in increments of 1000. The following test will ensure that the database behavior matches what we expected.\n@Test public void shouldIncrementIdBy1000() { Long firstValue = getNextValueForSequence(\u0026#34;S_MOVIE\u0026#34;); Long secondValue = getNextValueForSequence(\u0026#34;S_MOVIE\u0026#34;); assertIncrementsBy1000(firstValue, secondValue); } The above test tries to get the next value from the \u0026ldquo;S_MOVIE\u0026rdquo; sequence and compare if the consecutive values returned have incremented by 1000. At this stage the database script looks like\nCREATE SEQUENCE s_movie START WITH 1 INCREMENT BY 1000; Design a Unique Index # The next feature to work on is \u0026ldquo;The system should not allow duplicate Movie.Name in the system\u0026rdquo;, to implement this feature we use the Introduce Index refactoring, the behavior of the database ensures that the Movie.Name cannot be duplicated.\n@Test public void shouldNotAllowDuplicateNames() throws Exception { Movie movie = createAndSave(\u0026#34;NODUPE\u0026#34;); try { Movie dupe = createAndSave(\u0026#34;NODUPE\u0026#34;); fail(\u0026#34;Duplicate Movie name is allowed\u0026#34;); } catch (ConstraintViolationException e) { assertContains(e, \u0026#34;UIDX_MOVIE_NAME\u0026#34;); } } The above test tries to persist the Movie object with duplicate names and expects that the database throws a Unique Index violation. At this stage the database script looks like\nCREATE UNIQUE INDEX uidx_movie_name ON movie (name); Conclusion # These behavior specifications in code make sure that the database provides the specified behavior for the application and that the database cannot be changed inadvertently. These kinds of tests are also important if there is a need to have multiple database compatibility in your application code base. These kinds of tests are also really useful if you are depending on some other applications database and are expecting certain behavior.\n","date":"16 August 2015","externalUrl":null,"permalink":"/post/behavior-driven-database-development/","section":"Posts","summary":"When Behavior Driven Development BDD was introduced, some of the key principles were\nRequirements are behavior, Provides “ubiquitous language” for analysis, Acceptance criteria should be executable. Design constraints should be made into executable tests. ","title":"Behavior Driven Database Development","type":"post"},{"content":"","date":"20 July 2015","externalUrl":null,"permalink":"/tags/ruby/","section":"Tags","summary":"","title":"Ruby","type":"tags"},{"content":"IN many projects, there are tables which need default audit columns such as Created_By, Created_Date, Modified_By, Modified_date and other columns that need to be updated every time some actions are done against the tables and/or columns. This type of functionality can be implemented using triggers.\nWhy have triggers? when the application can do the updating, this is a good argument, but like all application databases eventually other users, applications and scripts will get access to the applications database and end up wanting to read from the database and write to the database. During these times its better to have triggers updating the data independent of the application to ensure audit columns and other columns are updated with appropriate values.\nWe can implement the triggers either by hand coding or by generating them using the database metadata and some build time scripting. Hand coding these triggers is mundane work and error prone, especially when we need to add a new column that needs to added to all the triggers.\nIn the ruby code shown below, we are using active record connection to get the database metadata with a list of tables and then generate triggers to intercept the UPDATE command on all the tables in our application and if the Modified_By or Modified_Date columns are null then we are populating them with the current database user logged in USER and the current system datetime SYSDATE.\nnamespace :db do desc \u0026#39;Generate Triggers for all auditable tables\u0026#39; task triggers: :environment do sql = \u0026#39;SELECT table_name tablename FROM user_tables\u0026#39; tables = ActiveRecord::Base.connection.exec_query(sql) tables.each do |table| name = table[\u0026#39;tablename\u0026#39;] trigger_name = \u0026#34;TRG_#{name.truncate(19, omission: \u0026#39;\u0026#39;)}_UPDATE\u0026#34; File.open(\u0026#34;generated/#{trigger_name}.sql\u0026#34;, \u0026#39;w\u0026#39;) do |triggerfile| triggerfile.puts \u0026#34;CREATE OR REPLACE TRIGGER #{trigger_name}\u0026#34; triggerfile.puts \u0026#34;BEFORE UPDATE ON #{name} FOR EACH ROW\u0026#34; triggerfile.puts \u0026#39;BEGIN\u0026#39; triggerfile.puts \u0026#39; IF :new.modified_by IS NULL\u0026#39; triggerfile.puts \u0026#39; THEN\u0026#39; triggerfile.puts \u0026#39; :new.modified_by := USER;\u0026#39; triggerfile.puts \u0026#39; END IF;\u0026#39; triggerfile.puts \u0026#39; IF :new.modified_date IS NULL\u0026#39; triggerfile.puts \u0026#39; THEN\u0026#39; triggerfile.puts \u0026#39; :new.modified_date := SYSDATE;\u0026#39; triggerfile.puts \u0026#39; END IF;\u0026#39; triggerfile.puts \u0026#39;END;\u0026#39; triggerfile.puts \u0026#39;/\u0026#39; end end puts \u0026#34;Generated triggers for tables\u0026#34; end end This trigger generating code can be run as part of the CI build and the generated triggers can be put in as part of the database artifacts that are generated. This generation of triggers makes it very easy to accommodate database refactoring as now we don\u0026rsquo;t need to delete trigger script file or add new trigger script file when tables are DROPPED or CREATED.\nWhen new columns such as Created_By and Created_Date need to be tracked or we need to track new DML actions such as INSERT or DELETE we just need to change the trigger generation code and all the triggers are updated without the need to hand edit each trigger script file.\n","date":"20 July 2015","externalUrl":null,"permalink":"/post/using-rake-and-activerecord-to-generate-boilerplate-db-code/","section":"Posts","summary":"IN many projects, there are tables which need default audit columns such as Created_By, Created_Date, Modified_By, Modified_date and other columns that need to be updated every time some actions are done against the tables and/or columns. This type of functionality can be implemented using triggers.\n","title":"Using rake and activerecord to generate boilerplate DB Code","type":"post"},{"content":"In every enterprise and every project we end up having multiple environments, especially the database side of the enterprise tends to stick around for a longer period of time and has much more dependencies or application integration as opposed to application urls etc. Given this, how to name the servers, databases and schemas becomes a very important decision, do these names provide for an easy way to use the application and not make it harder or the developers to access the database.\nAssuming we are using Oracle for our database and we have five environments other than the local developer workstations. Environments such as integration, development, qa, uat, production, how do we name the servers, database instances, schemas so that they are easy to understand and use.\nServer naming conventions are generally defined and should have an easy way to identify the environment they are associated with just by looking at the name, an example would be to end the server name with a suffix for the environment procyon-i for integration, procyon-d for development, procyon-q for qa, procyon-u for uat and procyon-p for production (where procyon is name of a star, used as a servername). Instead of suffix, we could prefex the environment to the servername either way the idea is have the server name easily identify the environment.\nInstance naming should follow similar convention where the name of the instance clearly shows the environment, if an instance is dedicated specifically for the application, the instance name can be same as application along with the environment suffix or prefix such as battani where battan is the database instance name and i is the integration environment, so we would have battand, battanq, battanu and battanp for development, qa, uat and production respectively. Once the instance name is setup, we can decide what the schema names are going to be, if the application is just using one schema on the instance then the schema name can be same as the application, if there are multiple instances of the application connecting to the same database instance, then we could have schema names with the purpose they are supposed to serve, such as willapa_trunk, willapa_release etc for each of the schemas.\nIn some situations we could have the need for one application instance to connect to multiple database schemas on the same instance then we should have the purpose of the schema also in the schema name such as trunk_sales, trunk_catalog etc so that its clear about what function they are serving. Following a fixed convention in the environment helps the development team to rely on predictable naming conventions thus avoiding creating excessive application configurations. The naming convention also helps the development and ops team figure out which environment is having problems based on servername, instancename or schemaname.\n","date":"10 June 2015","externalUrl":null,"permalink":"/post/database-naming-conventions-in-different-environments/","section":"Posts","summary":"In every enterprise and every project we end up having multiple environments, especially the database side of the enterprise tends to stick around for a longer period of time and has much more dependencies or application integration as opposed to application urls etc. Given this, how to name the servers, databases and schemas becomes a very important decision, do these names provide for an easy way to use the application and not make it harder or the developers to access the database.\n","title":"Database naming conventions in different environments","type":"post"},{"content":"Many of the projects we end up working on are replacing existing systems with existing data either wholly or in part. In all of the above projects we end up writing data migration or data conversion code to move the data from legacy systems to the new systems. Many stake holders of the project such as business users, project managers, business analysts really care about the data conversion scripts and the quality of the conversion. Since this conversion is business entity related and matters a lot as future business/functionality depends on the data being logically equivalent to the legacy system.\nOver the years we have found many techniques that help in testing the quality of the data conversion. Here are the 8 techniques that encompass our learnings when converting data over from legacy databases.\nStart data conversion work earlier # We have found that converting data earlier in the project life cycle is very helpful even when our data model is not stable yet. This is useful in many ways\nIt helps by forcing the developers to think and account for data in the legacy database that may not be covered by specs provided by the business analysts Business analysts can communicate with the business users using real data from the legacy system that the business is used to seeing and understands the data. Business users when using the new system before it goes live can understand how their new system behaves with legacy data and business entities such as Customers, Products etc. Giving the business users familiarity with the system and easing their transition to the new system. When its time to go-live we have converted the data so many times that its no longer a surprise, as all the bugs, data weirdness have been found and dealt with. Automated compare of data # We compare the data from the legacy database with the new application database that is being developed. This comparison can be automated using sql that creates logically equivalent objects from both databases.\nI blogged about using Automated Data Compare of comparing data. We could also use DiffKit which is an open source framework that lets you compare databases, excel spreadsheets, flat files or custom formats on the legacy database side with databases, excel spreadsheets, flat files or custom formats with the new databases. Or use frameworks that export the database into xml, yml etc formats such as yaml_db Dealing with duplicates # In legacy databases, as the system gets used over many years some business entities may get duplicated and during data conversion will get merged/collapsed into a single entity. So while comparing data we should remember to do UNIQUE or DISTINCT on the SQL we use to compare the data. In some cases we may end up normalizing some of the data and end up with multiple rows for something that was represented using one row in the legacy database.\nDealing with magic values # Systems use magic values to represent data or state of certain business process, such as NULL, 0, N/A, Nil and other strings. When converting we cannot convert these values as-is, we need to understand what do they mean, are these values being converted to mean something else on the application front end? are we using the same logic to convert the data?\nDealing with required data not existing # Some data elements maybe required in the new application that do not exist in the legacy database, we cannot just put null values or some made up data but have to carefully consider what kind of data is needed based on the properties of the new application and the domain entity being converted.\nDealing with string length, number precision # Some string data maybe longer in the legacy database than that which can be accepted by the new application database, we should raise exceptions for these conditions and fix the new application appropriately. Similar conditions can also apply for numeric columns such as ids, amounts, quantities etc, where we may loose precision of numbers being saved.\nSoft deleted rows and their children # In some systems rows are not physically deleted but are marked as deleted using flags or boolean values also known as Soft Delete. When converting this data to the new system, we should remember to not convert these rows if the new system does not use soft deletes. At the same time, how do we deal with children\u0026rsquo;s of rows that have been soft deleted?\nPerformance of the data conversion # How much time do we have to convert the data? can the data conversion code restart from the start when some unexpected error is reported and the process dies. What if we don\u0026rsquo;t have time to do a clean cut over from the old system? how do we convert data over a period of time while the old (legacy) and the new systems are both in production? These requirements have to taken into consideration when writing the data conversion.\n","date":"23 January 2015","externalUrl":null,"permalink":"/post/testing-migration-of-data-from-legacy-systems/","section":"Posts","summary":"Many of the projects we end up working on are replacing existing systems with existing data either wholly or in part. In all of the above projects we end up writing data migration or data conversion code to move the data from legacy systems to the new systems. Many stake holders of the project such as business users, project managers, business analysts really care about the data conversion scripts and the quality of the conversion. Since this conversion is business entity related and matters a lot as future business/functionality depends on the data being logically equivalent to the legacy system.\n","title":"8 Techniques for testing migration of data from legacy systems","type":"post"},{"content":"","date":"23 January 2015","externalUrl":null,"permalink":"/tags/evolutionary-design/","section":"Tags","summary":"","title":"Evolutionary-Design","type":"tags"},{"content":"","date":"23 January 2015","externalUrl":null,"permalink":"/tags/migration/","section":"Tags","summary":"","title":"Migration","type":"tags"},{"content":"","date":"23 January 2015","externalUrl":null,"permalink":"/tags/testing/","section":"Tags","summary":"","title":"Testing","type":"tags"},{"content":"In relational database usage the pattern of migrations is well understood and has gained widespread acceptance. Frameworks such as DBDeploy, DBMaintain, MyBatis migrations, Flyway, Liquibase, Active Record Migrations and many others. These tools allow to migrate the database and maintain the version history of the database in the database.\nWith the rise of NoSQL Databases and their adoption in development teams we are faced with the problem of migrations in NoSQL databases. What are the patterns of data migrations that work in NoSQL databases? as NoSQL databases are schema free and the database does not enforce any schema validation, the schema of the data is in the application and thus allows for different techniques of data migration.\nMigrate all the data in one go # In this pattern, we have to write a script that access all the objects in the database and migrates them to the latest version of the schema in the code, this pattern assumes that\nIt would be possible to access all the objects in the database, modify them and persist them back. In key-value stores its an expensive operation to retrieve all the keys During update of all the objects, the application may not modify the objects and create collusions. All the existing objects are at the same version of the schema Given the above assumptions, are valid we can use the same pattern used in relational databases of maintaining an list of versions applied to the database changelog table and then deciding which versions need to be applied for this deployment of the application mongodb migrations, mutagen cassandra, cdeploy Mongoid rails migration are an example of this approach.\nMigrate data during read # In this pattern, the data is read as is when required by the application, the data is then migrated to the latest version needed by the application and used by the application and written back when the user is done with the operation and now consists of upgraded data, this pattern assumes that\nThe application can read the oldest version of the data and upgrade it to the latest, sometimes keeping all this code in the code repository may dirty the code and reduce programmer productivity. Each object needs to know which version of the migration was applied to it, which means an additional attribute on the object. There maybe some objects that never get accessed and thus will never get migrated. Given the above assumptions, we need to write code that can deal with multiple versions of the data and upgrade all of the versions to the required version and persist it back with the latest version. Curator is an example for rails which allows the objects to be migrated to the latest version on read.\n##Hybrid approach In this pattern we can migrate the objects one at a time during read operations, at the same time we can have a background job that is running constantly and migrating the objects one at a time. This approach allows for all the objects to be migrated in a known period of time without having the need to keep all the code around to migrate the oldest versions of the object, thus allowing us to clean out code that is not needed.\n","date":"14 October 2014","externalUrl":null,"permalink":"/post/migrations-in-nosql-databases/","section":"Posts","summary":"In relational database usage the pattern of migrations is well understood and has gained widespread acceptance. Frameworks such as DBDeploy, DBMaintain, MyBatis migrations, Flyway, Liquibase, Active Record Migrations and many others. These tools allow to migrate the database and maintain the version history of the database in the database.\n","title":"Migrations in NoSQL databases","type":"post"},{"content":"","date":"14 October 2014","externalUrl":null,"permalink":"/tags/nosql/","section":"Tags","summary":"","title":"Nosql","type":"tags"},{"content":"While doing evalauation of NoSQL databases, we had a 10 node riak cluster and wanted check how a similar setup would work with mongodb. So started to setup a 10 node mongodb cluster. Since this was for initial spikes, we decided to set this up on a single machine as with the other test setup using Riak.\nBefore I explain how we setup 10 node mongodb ReplicaSet, let me talk about replica sets. MongoDB implements replication, providing high availability using replica sets. In a replica set, there are two or more nodes participating in an asynchronous master-slave replication. The replica-set nodes elect the master node, or primary node, among themselves and when the primary node goes down, the rest of the node elect the new primary node.\nWhen setting up ReplicaSets we need to know the ports and node names/address on which all the mongod instances are running. In this example we are using mongodb 2.6.1, the shell script below downloads mongodb and extracts into mongodb-2-6-1 and creates 10 folders to represent the 10 nodes where data will be stored. The replicaset needs a name and this example its named as snow\n#!/bin/bash -ex workingdir=`pwd` nodedirectory=\u0026#39;mongo-node\u0026#39; #!/bin/bash echo \u0026#34;Checking if mongo is downloaded\u0026#34; if [ ! -f \u0026#34;mongodb-osx-x86_64-2.6.1.tgz\u0026#34; ]; then rm -rf mongodb-2-6-1 wget http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-2.6.1.tgz fi echo \u0026#34;Unzip and Setup MongoDB\u0026#34; if [ ! -d \u0026#34;mongodb-2-6-1\u0026#34; ]; then tar -xvf mongodb-osx-x86_64-2.6.1.tgz mv mongodb-osx-x86_64-2.6.1 mongodb-2-6-1 echo \u0026#34;Setting up nodes\u0026#34; for i in {1..10} do cd $workingdir mongo_port=`expr 10000 + $i` nodeid=$nodedirectory$i mkdir $nodeid cd $nodeid mkdir data mkdir log $workingdir/mongodb-2-6-1/bin/mongod --dbpath=$workingdir/$nodeid/data --logpath=$workingdir/$nodeid/log/mongo.log --fork --port=$mongo_port --replSet snow done fi echo \u0026#34;Waiting for all nodes to initialize..\u0026#34; sleep 5 $workingdir/mongodb-2-6-1/bin/mongo localhost:10001 $workingdir/initialize-replicaset.js This script works on my mac and has been tested with OSX-10.9\nOnce all the nodes are started, we need to initialize the replicaset, in this javascript you will notice that three of the nodes have no votes this is because mongodb does not allow more than 7 nodes to particiapte in voting to choose the master. These 3 nodes can still be used as secondaries (slaves) and can still be used to read from.\n\u0026#34;use admin;\u0026#34; cfg = {\u0026#34;_id\u0026#34; : \u0026#34;snow\u0026#34;, \u0026#34;members\u0026#34; : [ { \u0026#34;_id\u0026#34; : 0, \u0026#34;host\u0026#34; : \u0026#34;localhost:10001\u0026#34;}, { \u0026#34;_id\u0026#34; : 1, \u0026#34;host\u0026#34; : \u0026#34;localhost:10002\u0026#34;}, { \u0026#34;_id\u0026#34; : 2, \u0026#34;host\u0026#34; : \u0026#34;localhost:10003\u0026#34;}, { \u0026#34;_id\u0026#34; : 3, \u0026#34;host\u0026#34; : \u0026#34;localhost:10004\u0026#34;}, { \u0026#34;_id\u0026#34; : 4, \u0026#34;host\u0026#34; : \u0026#34;localhost:10005\u0026#34;}, { \u0026#34;_id\u0026#34; : 5, \u0026#34;host\u0026#34; : \u0026#34;localhost:10006\u0026#34;}, { \u0026#34;_id\u0026#34; : 6, \u0026#34;host\u0026#34; : \u0026#34;localhost:10007\u0026#34;}, { \u0026#34;_id\u0026#34; : 7, \u0026#34;host\u0026#34; : \u0026#34;localhost:10008\u0026#34;, \u0026#34;votes\u0026#34;:0}, { \u0026#34;_id\u0026#34; : 8, \u0026#34;host\u0026#34; : \u0026#34;localhost:10009\u0026#34;, \u0026#34;votes\u0026#34;:0}, { \u0026#34;_id\u0026#34; : 9, \u0026#34;host\u0026#34; : \u0026#34;localhost:10010\u0026#34;, \u0026#34;votes\u0026#34;:0} ] }; rs.initiate(cfg); print(\u0026#34;Waiting for Replica-Set to initialize..\u0026#34;); var rsState = rs.status().myState; do { rsState = rs.status().myState; } while (rsState == undefined) printjson(rs.status()); print(\u0026#34;Replica-Set initialized..\u0026#34;) This script works on my mac and has been tested with OSX-10.9\nUpon initilization of the replica set, we can connect to the replicaset by connecting to a single node using mongo localhost:10001 and query the replicaset status using rs.status() here is a screen shot using the admin interface of the 10 nodes and their status along with the votes each of the nodes have. After the testing of the replicaset is done, we can shutdown down the replicaset and clean up all the data from all the nodes using the shell script.\n#!/bin/bash echo \u0026#34;Stopping all mongo processes\u0026#34; ps -ef |grep [m]ongo | awk \u0026#39;{print $2}\u0026#39; | xargs kill -2 echo \u0026#34;Cleaning up folders\u0026#34; read -p \u0026#34;Press any key to continue...Deletes all mongo-node folders... \u0026#34; -n1 -s rm -rf mongo-node* rm -rf mongodb-* echo \u0026#34;Done..\u0026#34; This script works on my mac and has been tested with OSX-10.9\nSetting up and testing a multi-node replicaset is easy and can be done on a single machine. Offcourse we are ignoring network issues, firewalls when crossing multiple data centers and other mayrid challenges, but we can still test how the replicaset behaves and how our code would interact with the replicaset, especially when nodes go down.\n","date":"9 June 2014","externalUrl":null,"permalink":"/post/node-mongodb-replicaset-on-a-single-machine/","section":"Posts","summary":"While doing evalauation of NoSQL databases, we had a 10 node riak cluster and wanted check how a similar setup would work with mongodb. So started to setup a 10 node mongodb cluster. Since this was for initial spikes, we decided to set this up on a single machine as with the other test setup using Riak.\n","title":"10 node mongodb ReplicaSet on a Single Machine\"","type":"post"},{"content":"Its been about a month since my blog moved to octopress, wanted to write about my experience. I had been running my blog for a some time now using Movable Type upgrading as and when new versions where released. Over time I realized that upgrading was fraught with errors as lot of steps had to be done manually. Customizing the layout was risky as there was no way to preview your changes and commit only when I was comfortable. With the release of Movable Type 6 there is no longer a free version to download.\nThere had been a a surge is usage of static html to write blogs. I was thinking about moving to a static blogging platform like Octopress. One of the major problems of this move was to extract existing blog entries along with their published dates, categories and blog txt from the Movable Text blog. The I read Neal Sheeran\u0026rsquo;s blog about using templates to export the data.\nUsing the techniques described by Neal I created a template to export my Movable Type blog into files which I curated using some scripts, manual editing, some de-duplication and normalization of categories. After words I had to move the assets I had into Octopress and change the links accordingly in the blog posts. Once this was done I put a custom 404 page in place and was ready to go.\nUsing octopress has been great, especially since I can preview all my changes before delopyment and deploy using rake deploy.\n","date":"31 January 2014","externalUrl":null,"permalink":"/post/moved-my-blog-to-octopress/","section":"Posts","summary":"Its been about a month since my blog moved to octopress, wanted to write about my experience. I had been running my blog for a some time now using Movable Type upgrading as and when new versions where released. Over time I realized that upgrading was fraught with errors as lot of steps had to be done manually. Customizing the layout was risky as there was no way to preview your changes and commit only when I was comfortable. With the release of Movable Type 6 there is no longer a free version to download.\n","title":"Moved my blog to octopress","type":"post"},{"content":"","date":"31 January 2014","externalUrl":null,"permalink":"/tags/news/","section":"Tags","summary":"","title":"News","type":"tags"},{"content":"Some versions back, Oracle would not allow to create database object names with mixed cases, even if we tried to create them, we could not. In newer versions of Oracle we can create tables, columns, indexes etc using mixed case or lower case, when the names are put inside double quotes. For example\nCREATE TABLE \u0026#34;Customer\u0026#34; ( \u0026#34;CustomerID\u0026#34; number(10) ... ); CREATE INDEX \u0026#34;IDX_Customer_CustomerID\u0026#34; on \u0026#34;Customer\u0026#34;(\u0026#34;CustomerID\u0026#34;); We created table named \u0026ldquo;Customer\u0026rdquo; with a column \u0026ldquo;CustomerID\u0026rdquo; and the index is named \u0026ldquo;IDX_Customer_CustomerID\u0026rdquo;. In the above example we can see that mixed case or lower case is supported and the table, column and index are created in the database. When these names are used, we have to reference them everywhere using the lower case letters. The following statement would be invalid.\nSELECT * FROM Customer While the below statement is valid.\nSELECT * FROM \u0026#34;Customer\u0026#34; We can even create a table such as \u0026ldquo;CusTomer\u0026rdquo; and it would be valid.\nCREATE TABLE \u0026#34;CusTomer\u0026#34; ( \u0026#34;CustomerID\u0026#34; number(10) ... ); With the large amount of work involved in matching the case for the name of the database object, every time a DML/DDL statement is used against a database object, the confusion it creates when duplicate database objects are allowed to be created (\u0026ldquo;Customer\u0026rdquo; and \u0026ldquo;CusTomer\u0026rdquo; are valid names).\nWe can see that its better to avoid using database objects with mixed case names.\n","date":"6 August 2013","externalUrl":null,"permalink":"/post/mixed_case_database_objects/","section":"Posts","summary":"Some versions back, Oracle would not allow to create database object names with mixed cases, even if we tried to create them, we could not. In newer versions of Oracle we can create tables, columns, indexes etc using mixed case or lower case, when the names are put inside double quotes. For example\n","title":"Usage of mixed case database object names is dangerous","type":"post"},{"content":"When trying to evaluate NoSQL databases, its usually better to try them out. While trying them out, its better to use them with multiple node configurations instead of running single node. Such as clusters in Riak or Replica-set in mongodb maybe even a sharded setup. On our project we evaluated a 10 node Riak cluster so that we could experiment with N, R and W values and decide which values where optimal for us. In Riak here is what N, R and W mean.\nN = Number of Riak nodes to which data will be replicated R = Number of Riak nodes which have to return results for the read to be considered successful W = Number of Riak nodes which have to return a write success before the write is considered successful\nThese N,R and W settings provide us the ability to tune our CAP requirements, thus they need to be carefully considered when architecting the system. What better way is there to test the assumptions we make than trying the assumptions out with some code and running Riak nodes. To experiment with our assumptions, we built a script that will download Riak 1.3.1 and create 10 nodes with different ports for pb_port, http port, handoff_port by changing them in app.config and -name in vm.args file for each node.\nThe first node is used as a master node to which all other nodes join after they are started using the riak-admin cluster join master_node_name@127.0.0.1 command to join the cluster. When all the nodes have been started, we can look at the cluster plan or configuration using riak-admin cluster plan and then commit the cluster plan using riak-admin cluster commit. This commits the cluster changes and makes all the nodes part of a cluster, we can view the status of the cluster using riak-admin status or see the nodes of the cluster using\nriak-admin status | grep \u0026#39;ring_members\u0026#39; #!/bin/bash -e working_dir=`pwd` master_node_id=\u0026#39;1\u0026#39; node_directory=\u0026#39;riak-node\u0026#39; node_name_prefix=\u0026#39;node\u0026#39; master_node_name=$node_name_prefix$master_node_id echo \u0026#34;Checking if Riak is downloaded\u0026#34; if [ ! -f \u0026#34;riak-1.3.*.tar\u0026#34; ]; then rm -rf riak-1.3.*.tar wget http://s3.amazonaws.com/downloads.basho.com/riak/1.3/1.3.1/osx/10.6/riak-1.3.1-osx-x86_64.tar.gz fi echo \u0026#34;Unzip and Set up Riak\u0026#34; gunzip riak-1.3.1-osx-x86_64.tar.gz tar -xf riak-1.3.1-osx-x86_64.tar echo \u0026#34;Setting up 10 nodes\u0026#34; for i in {1..10} do protocolbuffer_port=`expr 8000 + $i` http_port=`expr 8100 + $i` handoff_port=`expr 8200 + $i` cd $working_dir nodeid=$node_directory$i cp -r riak-1.3.1 $nodeid sed -e s/8087/$protocolbuffer_port/g -i \u0026#39;\u0026#39; $nodeid/etc/app.config sed -e s/8098/$http_port/g -i \u0026#39;\u0026#39; $nodeid/etc/app.config sed -e s/8099/$handoff_port/g -i \u0026#39;\u0026#39; $nodeid/etc/app.config sed -e s/riak@/$node_name_prefix$i@/g -i \u0026#39;\u0026#39; $nodeid/etc/vm.args cd $nodeid/bin ./riak start if [ $i -ne $master_node_id ]; then ./riak-admin cluster join $master_node_name@127.0.0.1 fi done cd $working_dir/$node_directory$master_node_id/bin/ ./riak-admin cluster plan ./riak-admin cluster commit ./riak-admin status | grep \u0026#39;node.@\u0026#39; echo \u0026#34;10 node Riak cluster setup\u0026#34; These shell scripts works on my mac and has not been tested on anything other than OSX-10.7+\nOnce experimenting, prototyping with the Riak cluster is done, we can gracefully shutdown the riak cluster and clean up all the folders using this shell script\n#!/bin/bash -e working_dir=`pwd` master_node_id=\u0026#39;1\u0026#39; node_directory=\u0026#39;riak-node\u0026#39; echo \u0026#34;Cleaning up riak cluster\u0026#34; for i in {1..10} do echo \u0026#34;Shutdown node:\u0026#34;$i nodeid=$node_directory$i cd $working_dir cd $nodeid/bin ./riak stop done cd $working_dir ./riak-node1/erts-5.9.1/bin/epmd -kill rm -rf riak-node rm .tar rm -rf riak-1.3.1 These scripts and approach can be modified to work across platforms\n","date":"25 April 2013","externalUrl":null,"permalink":"/post/node_riak_cluster/","section":"Posts","summary":"When trying to evaluate NoSQL databases, its usually better to try them out. While trying them out, its better to use them with multiple node configurations instead of running single node. Such as clusters in Riak or Replica-set in mongodb maybe even a sharded setup. On our project we evaluated a 10 node Riak cluster so that we could experiment with N, R and W values and decide which values where optimal for us. In Riak here is what N, R and W mean.\n","title":"10 node Riak cluster on a single machine","type":"post"},{"content":"When using Groovy with Spring framework, interacting with the database can be done using the Groovy.SQL class which provides a easy to use interface. When using Groovy.SQL, if we have a need to do transactions, we have the .withTransaction method that accepts a closure, to which we can pass in code to execute within the transaction. In our project since we were using spring already, using annotations to define transactions would be a great. Standard @Transactional annotations with Groovy.SQL will not work, since every place where the Groovy.SQL is used a new connection is acquired from the connection pool causing the database work to span multiple connections, which can result in dead-locks on the database. What we really want is that the database connection be the same across all invocations of Groovy.SQL with-in the same transaction started by the annotated method.\nThis series of spring configuration helped us get the right behavior. Setup annotations for spring using.\n\u0026lt;mvc:annotation-driven/\u0026gt; Setup connection pooling using the the BoneCP connection pool library, note that we have set autoCommit to false, since autoCommit is true by default.\n\u0026lt;bean id=\u0026#34;datasource\u0026#34; class=\u0026#34;com.jolbox.bonecp.BoneCPDataSource\u0026#34; destroy-method=\u0026#34;close\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;driverClass\u0026#34; value=\u0026#34;${db.driver}\u0026#34;/\u0026gt; \u0026lt;property name=\u0026#34;jdbcUrl\u0026#34; value=\u0026#34;${db.url}\u0026#34;/\u0026gt; \u0026lt;property name=\u0026#34;username\u0026#34; value=\u0026#34;${db.user}\u0026#34;/\u0026gt; \u0026lt;property name=\u0026#34;password\u0026#34; value=\u0026#34;${db.passwd}\u0026#34;/\u0026gt; \u0026lt;property name=\u0026#34;defaultAutoCommit\u0026#34; value=\u0026#34;false\u0026#34;/\u0026gt; \u0026lt;/bean\u0026gt; Setup annotation driven transaction manager and ensure that proxy-target-class is true.\n\u0026lt;tx:annotation-driven transaction-manager=\u0026#34;txManager\u0026#34; proxy-target-class=\u0026#34;true\u0026#34;/\u0026gt; Next setup TransactionAwareDataSourceProxy using the datasource, previously defined. The transactionAwareDataSourceProxy is named datasourceProxy here.\n\u0026lt;bean id=\u0026#34;datasourceProxy\u0026#34; class=\u0026#34;org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy\u0026#34;\u0026gt; \u0026lt;constructor-arg ref=\u0026#34;datasource\u0026#34;/\u0026gt; \u0026lt;/bean\u0026gt; Next pass the datasourceProxy to the DataSourceTransactionManager\n\u0026lt;bean id=\u0026#34;txManager\u0026#34; class=\u0026#34;org.springframework.jdbc.datasource.DataSourceTransactionManager\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;dataSource\u0026#34; ref=\u0026#34;datasourceProxy\u0026#34;/\u0026gt; \u0026lt;/bean\u0026gt; This datasourceProxy can now be used to setup Groovy.SQL.\n\u0026lt;bean id=\u0026#34;sql\u0026#34; class=\u0026#34;groovy.sql.Sql\u0026#34;\u0026gt; \u0026lt;constructor-arg ref=\u0026#34;datasourceProxy\u0026#34;/\u0026gt; \u0026lt;/bean\u0026gt; This sql bean can now be autowired inside any class to be used by read or write methods and with annotations so that we can have transactions.\nclass ProductGateway { @Autowired Sql sql List findBy(productId) { def productRows = sql.rows(....) // retrieve and return list of products // handle exceptions } @Transactional int create(product) { // Create rows, update data under transaction sql.executeInsert(...) sql.executeInsert(...) // handle exceptions } } Edit: Changed SQLUtil to Sql to clarify example, based on feedback from @robpatrick\n","date":"14 January 2013","externalUrl":null,"permalink":"/post/transactions_using_groovysql/","section":"Posts","summary":"When using Groovy with Spring framework, interacting with the database can be done using the Groovy.SQL class which provides a easy to use interface. When using Groovy.SQL, if we have a need to do transactions, we have the .withTransaction method that accepts a closure, to which we can pass in code to execute within the transaction. In our project since we were using spring already, using annotations to define transactions would be a great. Standard @Transactional annotations with Groovy.SQL will not work, since every place where the Groovy.SQL is used a new connection is acquired from the connection pool causing the database work to span multiple connections, which can result in dead-locks on the database. What we really want is that the database connection be the same across all invocations of Groovy.SQL with-in the same transaction started by the annotated method.\n","title":"Transactions using Groovy.SQL with Spring annotations and connection pools","type":"post"},{"content":"There are multiple ways to take backups of mongodb is different configuraitions, one of the configuration that I have been involved recently is replica-sets. When mongodb is running in replica-set configuration, there is a single primary node and multiple secondary nodes. To take backup of the replica-set we can either do a mongodump of one of the nodes or shutdown one of the secondary nodes and take file copies, since in a replica-set all nodes have the same data (except arbiter). Lets see how we could deal with mongodump method of taking backup.\nSync the primary node so that all writes are flused to disk and lock the database for writes, doing a fsync allows for all writes to be persisted to the disk.\nuse admin db.fsyncLock() now we can issue the mongodump command, there are many more options for mongodump that can be changed, using defaults here\nmongodump -h node_name --out /data/backups/backup_file_name once the mongodump command is done we can unlock the database so that writes can be issued.\nuse admin db.fsyncUnlock() The downside of this approach is that the primary is not available for writes, but reads are fine. If by any chance there is write issued, all reads are also blocked after that, which is pretty drastic. The other option is to operate on one of the secondaries, this allows us to keep the primary available for write and read, the secondary can be used for backup purposes. Which node is PRIMARY or SECONDARY can be dynamically determined by running some javascript on the command line\n#!/usr/bin/env ruby require \u0026#39;rubygems\u0026#39; require \u0026#39;json\u0026#39; mongo_nodes = JSON.parse `mongo node_name --quiet --eval \u0026#34;printjson(rs.status().members.map( function(m) { return {\u0026#39;name\u0026#39;:m.name, \u0026#39;stateStr\u0026#39;:m.stateStr} }))\u0026#34;` primary_node = mongo_nodes.detect { |member| member[\u0026#39;stateStr\u0026#39;] == \u0026#39;PRIMARY\u0026#39; } This dynamic script allows us to find the node that we want to take backup from, either the primary or secondary.\n","date":"27 December 2012","externalUrl":null,"permalink":"/post/backup_in_mongodb_replica-sets/","section":"Posts","summary":"There are multiple ways to take backups of mongodb is different configuraitions, one of the configuration that I have been involved recently is replica-sets. When mongodb is running in replica-set configuration, there is a single primary node and multiple secondary nodes. To take backup of the replica-set we can either do a mongodump of one of the nodes or shutdown one of the secondary nodes and take file copies, since in a replica-set all nodes have the same data (except arbiter). Lets see how we could deal with mongodump method of taking backup.\n","title":"Backup in mongodb replica-set configurations","type":"post"},{"content":" There has been a long pause in my blogging activity. I was trying to finish of my latest writing engagement in regards to NoSQL. Working with Martin Fowler on NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence was really fun. This book will provide a concise text and easy way to understand for everyone the rise of the NoSQL movement and help with what kinds of trade-offs need to be made while working with NoSQL.\nThe book should soon be in print and e-book formats. Martin has written more about it here\n","date":"22 April 2012","externalUrl":null,"permalink":"/post/back_to_blogging/","section":"Posts","summary":" There has been a long pause in my blogging activity. I was trying to finish of my latest writing engagement in regards to NoSQL. Working with Martin Fowler on NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence was really fun. This book will provide a concise text and easy way to understand for everyone the rise of the NoSQL movement and help with what kinds of trade-offs need to be made while working with NoSQL.\n","title":"Back to blogging","type":"post"},{"content":"My latest project involves talking to MS-SQL Server using the JDBC driver and Java. While doing this we setup the database connection and had a simple SQL to get the first_name and last_name for a unique user_id from the application_user table in the database.\nSELECT first_name,last_name FROM application_user WHERE user_id = ? Given the above SQL, we did not think too much about performance as the user_id was indexed. The java code as below was used to run the SQL.\ntry { PreparedStatement stmt = prepare(conn, \u0026#34;SELECT first_name, last_name FROM application_user \u0026#34; + \u0026#34;WHERE user_id = ?\u0026#34;); stmt.setString(1, username); return execute(stmt); } catch (SQLException e) { e.printStackTrace(); } When writing integration tests we started noticing that the SQL was taking about 6 seconds to execute. The same SQL would execute inside 100 milliseconds on the MSSQL query analyzer. The friendly DBA\u0026rsquo;s on our team pointed out that the SQL was doing some data type conversion as the user_id field was of the type VARCHAR but the SQL sent by the the JDBC driver set the data type to NVARCHAR because of this the index was not being used and the SQL took more than 6 seconds to execute. Researching this topic further we decided to cast the variable to VARCHAR as shown below.\ntry { PreparedStatement stmt = prepare(conn, \u0026#34;SELECT first_name, last_name FROM application_user WHERE user_id = cast(? AS VARCHAR)\u0026#34;); stmt.setString(1, username); return execute(stmt); } catch (SQLException e) { e.printStackTrace(); } The above code executed under 100 milliseconds and showed us that the data types being used did not match the datatype in the database. We later found out that the MS-SQL JDBC driver does this to properly deal with unicode characters. This behavior can be turned off using the sendStringParametersAsUnicode flag on the database connection. Once this flag is set to false on the connection, then all the SQL we issue do not need the cast\nConnection conn = db.conn(DATABASE_URL + \u0026#34;;sendStringParametersAsUnicode=false\u0026#34;); try { PreparedStatement stmt = prepare(conn, \u0026#34;SELECT first_name, last_name FROM application_user WHERE user_id = ? \u0026#34;); stmt.setString(1, username); return execute(stmt) } catch (SQLException e) { e.printStackTrace(); } Off course this only works if there is no unicode data in your database if there is any unicode data in the database, we will have to revert to casting individual SQL statements.\n","date":"22 April 2012","externalUrl":null,"permalink":"/post/mssql_jdbc_driver_behavior/","section":"Posts","summary":"My latest project involves talking to MS-SQL Server using the JDBC driver and Java. While doing this we setup the database connection and had a simple SQL to get the first_name and last_name for a unique user_id from the application_user table in the database.\n","title":"MSSQL JDBC Driver behavior\"","type":"post"},{"content":"","date":"22 April 2012","externalUrl":null,"permalink":"/tags/writing/","section":"Tags","summary":"","title":"Writing","type":"tags"},{"content":"","date":"19 January 2011","externalUrl":null,"permalink":"/tags/code/","section":"Tags","summary":"","title":"Code","type":"tags"},{"content":"I keep encountering situations where all the business logic for the applications is in stored procedures and the application layer is just calling the stored procedures to get the work done and return the data. There are many problems with this approach some of them are.\nWriting stored procedure code is fraught with danger as there are no modern IDE\u0026rsquo;s that support refactoring, provide code smells like \u0026ldquo;variable not used\u0026rdquo;, \u0026ldquo;variable out of scope\u0026rdquo;.\nFinding usages of a given stored procedure or function usually means doing a text search of the whole code base for the name of the function or stored procedure, so refactoring to change name is painful, which means names that do not make any sense are propagated, causing pain and loss of developer productivity\nWhen coding of stored procedures is done, you need a database to compile the code, this usually means a large database install on your desktop or laptop the other option being to connect to the central database server, again this leads to developers having to carry a lot of dependent systems just to compile their code, this can to solved by database vendors providing a way to compile the code outside of the database.\nCode complexity tools, PMD metrics, Checkstyle etc type of tools are very rare to find for stored procedures, thus making the visualization of metrics around the stored procedure code almost impossible or very hard\nUnit testing stored procedures using *Unit testing frameworks out there like pl/sql unit, ounit, tsql unit is hard, since these frameworks need to be run inside the database and integrating them with Continuous Integration further exasperates the problems\nOrder or creation of stored procedures becomes important as you start creating lots of stored procedures and they become interdependent. While creating them in a brand new database, there are false notifications thrown around about missing stored procedures, usually to get around this problem, I have seen a master list of ordered stored procedures for creation maintained by the team or just recompile all stored procedures once they are created ALTER STORED-PROCEDURE-NAME RECOMPILE was built for this. Both of these solutions have their own overhead.\nWhile running CPU intensive stored procedures, the database engine is the only machine (like JVM) available for the code to run, so if you want to start more processes so that we can handle more requests, its not possible without a database engine. So the only solution left is to get a bigger box (Vertical Scaling)\nThere certainly are lots of other problems associated with using stored procedures, which I will not get into.\n","date":"19 January 2011","externalUrl":null,"permalink":"/post/why_use_stored_procedures/","section":"Posts","summary":"I keep encountering situations where all the business logic for the applications is in stored procedures and the application layer is just calling the stored procedures to get the work done and return the data. There are many problems with this approach some of them are.\n","title":"With so much pain, why are stored procedures used so much","type":"post"},{"content":"Replica sets is a feature of MongoDB for Automatic Failover, in this setup there is a primary server and the rest are secondary servers. If the primary server goes down, the rest of the secondary servers choose a new primary via an election process, each server can also be assigned number of votes, so that you can decide the next primary based on data-center location, machine properties etc, you can also start mongo database processes that act only as election tie-breakers these are known as arbiters, these arbiters will never have data, but just act as agents that break the tie.\nAll operations are directed at the primary server, the primary server writes the operations to its operation log (also known as opslog), the secondary servers get updates from the primary server. The data is written to the primary server and later replicated to the other secondary servers, so when the write happens at the primary and before the write is replicated to the secondary servers, if the primary server goes down you will loose the data that was written to the primary but never replicated to the secondary servers, you can get around this by specifying how many servers should have the data, before the write is considered good\ndb.runCommand( { getlasterror : 1 , w : 3 } ) in the above command, you are saying that the write to the database is considered good, only if the write has been propagated to at least 3 servers, off course doing this for every write is going to be very expensive, so you should batch all your writes for a user action and then issue getlasterror\nThis is how you start the the mongod servers, in a replica set, the can run on any machine any port, as long as they can talk to each other over the network and all of them have the same \u0026ldquo;\u0026ndash;replSet\u0026rdquo; parameter, in the example below its \u0026ldquo;prod\u0026rdquo;\nmongod --replSet prod --port 27017 --dbpath /data/node1 mongod --replSet prod --port 27027 --dbpath /data/node2 mongod --replSet prod --port 27037 --dbpath /data/node3 Once the three servers are up, you have to create a replica configuration as shown below, if you use localhost as a server name, then all the members of the replica set have to be on localhost, if the mongo servers are on different servers, you should use distinct machine names and not localhost for anyone of them, once the replica config is defined, you then initiate the replica using the configuration as shown below\nreplica_config = {_id: \u0026#39;prod\u0026#39;, members: [ {_id: 0, host: \u0026#39;localhost:27017\u0026#39;}, {_id: 1, host: \u0026#39;localhost:27027\u0026#39;}, {_id: 2, host: \u0026#39;localhost:27037\u0026#39;}]} #Now initiate the replica_config rs.initiate(replica_config); When you are connecting to a replica set, you have to connect to atleast one server which is alive, using the ruby driver you can connect to more than one server using the \u0026ldquo;multi\u0026rdquo; method, one part you should be careful about is, lets say you define all the servers in the replica set as your connection string, but one of the members of the replica set is down, you will get connection failures, so the best thing to do is give members of the replica set that are up and the drivers will discover the other servers when they come online or go offline. Here is a sample ruby program to find a doc in a loop.\n#!/usr/bin/env ruby require \u0026#39;mongo\u0026#39; begin @connection = Mongo::Connection.multi([ [\u0026#39;localhost\u0026#39;,27017], [\u0026#39;localhost\u0026#39;,27027], [\u0026#39;localhost\u0026#39;,27037]]) @collection = @connection.db(\u0026#34;sales\u0026#34;).collection(\u0026#34;products\u0026#34;) product = { \u0026#34;name\u0026#34; =\u0026gt; \u0026#34;Refactoring\u0026#34;, \u0026#34;code\u0026#34; =\u0026gt; \u0026#34;023XX3\u0026#34;, \u0026#34;type\u0026#34; =\u0026gt; \u0026#34;book\u0026#34;, \u0026#34;in_stock\u0026#34; =\u0026gt; 100} @collection.insert(product) 100.times do sleep 0.5 begin product = @collection.find_one \u0026#34;code\u0026#34; =\u0026gt; \u0026#34;023XX3\u0026#34; puts \u0026#34;Found Book: \u0026#34;+product[\u0026#34;name\u0026#34;] rescue Exception =\u0026gt; e puts e.message next end end end {% video http://sadalage.com/videos/replicacast.mp4 640 320 %}\nWhile the ruby program is running, you can kill the current primary and you will see that the program gets connection exceptions, while the replica set is figuring out the next master, once the next master is picked, the program starts going about its way finding the same data from the newly elected primary, here is a screen cast of the replica sets in action.\n","date":"31 October 2010","externalUrl":null,"permalink":"/post/replica_sets_in_mongodb/","section":"Posts","summary":"Replica sets is a feature of MongoDB for Automatic Failover, in this setup there is a primary server and the rest are secondary servers. If the primary server goes down, the rest of the secondary servers choose a new primary via an election process, each server can also be assigned number of votes, so that you can decide the next primary based on data-center location, machine properties etc, you can also start mongo database processes that act only as election tie-breakers these are known as arbiters, these arbiters will never have data, but just act as agents that break the tie.\n","title":"Replica sets in MongoDB","type":"post"},{"content":"In the No-SQL land schema-less is a power full feature that is advertised a lot, schema-less basically means you don\u0026rsquo;t have to worry about column names and table names in a traditional sense, if you want to change the column name you just start saving the data using the new column name Lets say you have a document database like mongoDB and you have JSON document as shown below.\n{ \u0026#34;_id\u0026#34;:\u0026#34;4bc9157e201f254d204226bf\u0026#34;, \u0026#34;FIRST_NAME\u0026#34;:\u0026#34;JOHN\u0026#34;, \u0026#34;MIDDLE_NAME\u0026#34;:\u0026#34;D\u0026#34;, \u0026#34;LAST_NAME\u0026#34;:\u0026#34;DOE\u0026#34;, \u0026#34;CREATED\u0026#34;:\u0026#34;2010-10-12\u0026#34; } You have some corresponding code to read the documents from the database and lets say you lots of data in the database in the order of millions of documents. If you want to change the name of some attributes or columns at this point and the new JSON would look like\n{ \u0026#34;_id\u0026#34;:\u0026#34;4bc9157e201f254d204226bf\u0026#34;, \u0026#34;first_name\u0026#34;:\u0026#34;JOHN\u0026#34;, \u0026#34;middle_name\u0026#34;:\u0026#34;D\u0026#34;, \u0026#34;last_name\u0026#34;:\u0026#34;DOE\u0026#34;, \u0026#34;created\u0026#34;:\u0026#34;2010-10-12\u0026#34; } You will have to either change every document in the database to match the new attribute names or you have to make sure you code can handle both types of attribute names like\nfirst_name = doc[\u0026#34;first_name\u0026#34;] first_name = doc[\u0026#34;FIRST_NAME\u0026#34;] unless !first_name.nil? middle_name = doc[\u0026#34;middle_name\u0026#34;] middle_name = doc[\u0026#34;MIDDLE_NAME\u0026#34;] unless !middle_name.nil? last_name = doc[\u0026#34;last_name\u0026#34;] last_name = doc[\u0026#34;LAST_NAME\u0026#34;] unless !last_name.nil? This attribute name change also affects the indexes created on mongoDB, since the attribute name change is not across all the documents, an Index created on\ndb.people.ensureIndex({first_name:1}) will not index documents where the attribute name is FIRST_NAME, so you have to create another index for this new attribute name\ndb.people.ensureIndex({FIRST_NAME:1}) As you can see this gets really complicated if you do multiple refactorings, over a period of time. So when you hear schema less make sure you understand the ramifications of refactoring the attribute names at will and its effect on the code base and the database.\n","date":"12 October 2010","externalUrl":null,"permalink":"/post/schema_less_databases/","section":"Posts","summary":"In the No-SQL land schema-less is a power full feature that is advertised a lot, schema-less basically means you don’t have to worry about column names and table names in a traditional sense, if you want to change the column name you just start saving the data using the new column name Lets say you have a document database like mongoDB and you have JSON document as shown below.\n","title":"Schema less databases and its ramifications.","type":"post"},{"content":"","date":"31 August 2010","externalUrl":null,"permalink":"/tags/analytics/","section":"Tags","summary":"","title":"Analytics","type":"tags"},{"content":"For more than seven years I have been getting offers for credit cards from Airlines and Banks. One particular bank has been sending me these solicitations for more than seven years. That is 12 mailings per year, more than 72 mailings so far, remember these are physical paper mailings not the electronic kind. I don\u0026rsquo;t like the junk, it hurts the environment and worst of all I think its not good use of the data they have. How hard is it to design a system around the data they have.\nLets say they have a table of all the targeted customers they want to send a credit card applications to, why not have a attribute on the table for counting how many times the solicitation was sent, or they can even have the date the first solicitation was sent.\nCustomer Name Address City FirstSolicitationSent or\nCustomer Name Address City Solicitations So they could say if the days between today and the firstSolicitationSent is more than 90 days, then not send another solicitation, or if this number of solicitations is more than three do not send another solicitation.\nThis allows them to not send solicitations for years and ultimately loose the customer, I understand the argument of the customer needing time to react to the solicitation, but seven years of trying to convert a prospect is pure waste of time and effort. The data available can be used in better ways to reduce waste.\n","date":"31 August 2010","externalUrl":null,"permalink":"/post/effective_use_of_data/","section":"Posts","summary":"For more than seven years I have been getting offers for credit cards from Airlines and Banks. One particular bank has been sending me these solicitations for more than seven years. That is 12 mailings per year, more than 72 mailings so far, remember these are physical paper mailings not the electronic kind. I don’t like the junk, it hurts the environment and worst of all I think its not good use of the data they have. How hard is it to design a system around the data they have.\n","title":"Effective use of data for better customer experience.","type":"post"},{"content":"We are using MongoDB on our project, since mongo is document store, schema design is somewhat different, when you are using traditional RDBMS data stores, one thinks about tables and rows, while using a document database you have to think about the schema in a some what different way. Lets say, we want to save a customer object, when using a RDBMS we would come up with Customer, Address, Phone, Email. They are related to each other as shown below. When doing a document database, the schema design actually does not change much, the Customer document contains an array of Addresses, a one to many relationship. You will not need the FK columns or the Primary Key columns on the child tables, since the child rows are embedded in the parent object. The JSON object below shows how the data would look.\n{ \u0026#34;_id\u0026#34; : ObjectId(\u0026#34;4bd8ae97c47016442af4a580\u0026#34;), \u0026#34;customerid\u0026#34; : 99999, \u0026#34;name\u0026#34; : \u0026#34;Foo Sushi Inc\u0026#34;, \u0026#34;type\u0026#34; : \u0026#34;Good\u0026#34;, \u0026#34;since\u0026#34; : \u0026#34;12/12/2001\u0026#34;, \u0026#34;addresses\u0026#34; : [{ \u0026#34;address\u0026#34; : \u0026#34;4821 Big Street\u0026#34;, \u0026#34;city\u0026#34; : \u0026#34;Stone\u0026#34;,\t\u0026#34;state\u0026#34; : \u0026#34;IL\u0026#34;, \u0026#34;country\u0026#34; : \u0026#34;USA\u0026#34; }, {\t\u0026#34;address\u0026#34; : \u0026#34;1248 Barlow Ln\u0026#34;, \u0026#34;city\u0026#34; : \u0026#34;Hedgestone\u0026#34;,\t\u0026#34;country\u0026#34; : \u0026#34;UK\u0026#34; }\t], \u0026#34;emails\u0026#34; : [ {\u0026#34;email\u0026#34; : \u0026#34;foousa@sushi.com\u0026#34;}, {\u0026#34;email\u0026#34; : \u0026#34;foouk@sushi.com\u0026#34;} ], \u0026#34;phones\u0026#34; : [ {\u0026#34;phone\u0026#34; : \u0026#34;773-7777-7777\u0026#34;}, {\u0026#34;phone\u0026#34; : \u0026#34;020-6666-6666\u0026#34;} ] } So Instead of 1 Row for customer, 2 rows for address, phone and email each, you get one Customer document. If you want to query for customers in USA. Using RDBMS you would do\nSELECT customer.name FROM customer, address WHERE customer.customerid = address.customerid AND address.country=\u0026#34;USA\u0026#34; The same query in mongo would look like\ndb.customers.find({\u0026#34;addresses.country\u0026#34;:\u0026#34;USA\u0026#34;},{\u0026#34;name\u0026#34;:true}) where customers is the collection in which we store the customers information.\n","date":"28 April 2010","externalUrl":null,"permalink":"/post/schema_design_in_a_document_databases/","section":"Posts","summary":"We are using MongoDB on our project, since mongo is document store, schema design is somewhat different, when you are using traditional RDBMS data stores, one thinks about tables and rows, while using a document database you have to think about the schema in a some what different way. Lets say, we want to save a customer object, when using a RDBMS we would come up with Customer, Address, Phone, Email. They are related to each other as shown below. ","title":"Schema design in a document database","type":"post"},{"content":"","date":"18 April 2010","externalUrl":null,"permalink":"/tags/lessons/","section":"Tags","summary":"","title":"Lessons","type":"tags"},{"content":"The current project I\u0026rsquo;m on is using MongoDB. MongoDB is a document based database, it stores JSON objects as BSON (Binary JSON objects). MongoDB provides a middle ground between the traditional RDBMS and the NOSql databases out there, it provides for indexes, dynamic queries, replication, map reduce and auto sharding, its open source and can be downloaded Mongodb, starting up mongodb is pretty easy.\n./mongod --dbpath=/user/data/db is all you need, where /user/data/db is the path where you want mongo to create its data files. There are many other options that you can use to customize the mongo instance.\nEach mongo instance has databases and each database has many collections, mapping back to oracle, mongo database is a oracle schema and mongo collection is a oracle table, The difference is, each collection can hold any type of object, basically every row can be different.\nAn example connection to the database using java looks like this\nMongo mongo = new Mongo(\u0026#34;localhost\u0026#34;); db = mongo.getDB(\u0026#34;mydatabase\u0026#34;); If the \u0026ldquo;mydatabase\u0026rdquo; does not exist, it will be created. When you want to put objects in the database, you need to have a collection which holds the objects.\nusers = db.getCollection(\u0026#34;applicationusers\u0026#34;); if the applicationusers collection does not exist, it will be created, at this point you are ready to put objects into the collection.\nBasicDBObject userDocument = new BasicDBObject(); userDocument.put(\u0026#34;name\u0026#34;, \u0026#34;jack\u0026#34;); userDocument.put(\u0026#34;type\u0026#34;, \u0026#34;super\u0026#34;); users.insert(userDocument); You create a document by using the BasicDBObject and put attribute names and their values, in the above example name is the attribute and jack is the value, the users.insert takes the document and inserts it into the collection users. At this point you have a JSON object put into the database.\nYou can query for the object using the mongo query tool or the restfull api mongo provides using the flag \u0026ndash;rest, when you start mongodb, visiting http://127.0.0.1:28017/mydatabase/users/ should give you\n{ \u0026#34;offset\u0026#34; : 0, \u0026#34;rows\u0026#34;: [ { \u0026#34;_id\u0026#34; : { \u0026#34;$oid\u0026#34; : \u0026#34;4bc9157e201f254d204226bf\u0026#34; }, \u0026#34;name\u0026#34; : \u0026#34;jack\u0026#34;, \u0026#34;type\u0026#34; : \u0026#34;super\u0026#34; } ], \u0026#34;total_rows\u0026#34; : 1 , \u0026#34;query\u0026#34; : {} , \u0026#34;millis\u0026#34; : 0 } Every object you insert, gets a auto generated id, more about update, delete and complex objects in next blog post.\n","date":"18 April 2010","externalUrl":null,"permalink":"/post/my_experience_with_mongodb/","section":"Posts","summary":"The current project I’m on is using MongoDB. MongoDB is a document based database, it stores JSON objects as BSON (Binary JSON objects). MongoDB provides a middle ground between the traditional RDBMS and the NOSql databases out there, it provides for indexes, dynamic queries, replication, map reduce and auto sharding, its open source and can be downloaded Mongodb, starting up mongodb is pretty easy.\n","title":"My experience with MongoDB","type":"post"},{"content":"Doing a workshop on Agile Database Development at Enterprise Data World 2010 at SF. See you there.\n","date":"29 January 2010","externalUrl":null,"permalink":"/post/workshop_at_enterprise_data_warehouse/","section":"Posts","summary":"Doing a workshop on Agile Database Development at Enterprise Data World 2010 at SF. See you there.\n","title":"Workshop at Enterprise Data World 2010","type":"post"},{"content":"","date":"18 November 2009","externalUrl":null,"permalink":"/tags/conversion/","section":"Tags","summary":"","title":"Conversion","type":"tags"},{"content":"When working on projects involving Conversion of data or Migration/Moving of data from a legacy database. The testing effort is enormous and testing takes a lot of time, some test automation can help this effort.\nSince data is moved/changed from a source database to destination database, we can write sql which should provide results for the types of tests you want to perform, for example: write a sql to give us number of customers, write a sql to give us account balance for a specific account.\nThese sqls can be run on your source database as well as your destination database and the results can be compared programmatically, providing us an easy way to compare the state of the database before and after conversion/migration. This testing can be run through a CI engine to make it a regression test suite.\nHere is an example implementation using ruby,\nWe have two databases SOURCE and DESTINATION and two sql files names source.sql and destination.sql. The ruby program picks up sql from these two files and runs them against their database i.e. sql from source.sql is run against the SOURCE database and sql from destination.sql is run against DESTINATION database. The results of both of those sqls is compared and an failure is raised when the results do not match.\nresults statement = get_sql_statement_to_execute begin source_statement = statement[0] destination_statement = statement[1] source_rows = exec_sql_in_source_return_rows(source_statement) destination_rows = exec_sql_in_destination_return_rows(destination_statement) result = compare_rows(source_rows, destination_rows, destination_statement, source_statement) results \u0026lt;\u0026lt; result rescue Log.log(\u0026#34;Could not process: \u0026#34;+statement) end if (results.size \u0026gt; 0) Log.log(\u0026#34;Results do not match in source and destination\u0026#34;) end The sample ruby code above shows how the solution can be implemented, thus enabling automation of database conversion/migration testing.\n","date":"18 November 2009","externalUrl":null,"permalink":"/post/automated_data_compare_in_migrations/","section":"Posts","summary":"When working on projects involving Conversion of data or Migration/Moving of data from a legacy database. The testing effort is enormous and testing takes a lot of time, some test automation can help this effort.\n","title":"Testing in data conversion projects","type":"post"},{"content":"We have been doing some data moving lately using Ruby and Ruby-OCI. We started with Ruby OCI 1.0 and did use prepared statements with bind variables (since we are using oracle database and pulling data from an oracle database and pushing data to an oracle database). Later we found this really cool feature in Ruby-OCI8 2.0 where you can bind a whole array and just make one database trip for many database operations.\nLets say you want to insert 10 rows, using the insert one row at a time would be 10 trips to the database.\ndef save_accounts(accounts) stmt = $connection.parse \u0026#34;INSERT INTO account (accountid,name) values (:account_id,:name)\u0026#34; accounts.each do |account| stmt.bind_param(:account_id, account[0], Float) stmt.bind_param(:name, account[1], String) stmt.exec end $connection.commit end Using the array bind feature, its actually just one trip to the database (off course depends on the array size you are going to bind, but you get the picture, it reduces database trips)\ndef save_accounts(account_ids, account_names) stmt = $connection.parse \u0026#34;INSERT INTO account (accountid,name) values (:account_id,:name)\u0026#34; stmt.max_array_size= account_ids.size stmt.bind_param_array(:account_id, account_ids) stmt.bind_param_array(:name, account_names) stmt.exec_array $connection.commit end We saw a 100% improvement in performance by changing the way we bind the variables in just one place. Looks like a feature to look out for.\n","date":"8 October 2009","externalUrl":null,"permalink":"/post/ruby_oci_20_array_binding/","section":"Posts","summary":"We have been doing some data moving lately using Ruby and Ruby-OCI. We started with Ruby OCI 1.0 and did use prepared statements with bind variables (since we are using oracle database and pulling data from an oracle database and pushing data to an oracle database). Later we found this really cool feature in Ruby-OCI8 2.0 where you can bind a whole array and just make one database trip for many database operations.\n","title":"Ruby OCI 2.0 Array binding","type":"post"},{"content":"Most of the time I have seen database foreign key constraints on tables without indexes on those columns. Lets say the application is trying to delete a row from the CUSTOMER table\nDELETE FROM CUSTOMER WHERE CUSTOMERID = 1000; When the database goes about deleting the customerId of 1000, if there are foreign key constraints defined on customerId, then the database is going to try to find if the customerId of 1000 is used in any of those tables. Lets say ORDER table has the customerId column, the database is going to issue\nSELECT ... FROM ORDER WHERE CUSTOMERID = 1000; now if there is no index on ORDER.CUSTOMERID, the database will have to do a full Table scan which is very expensive in terms of IO and resources, imagine customerId being used in lots of tables, the problem just multiplies significantly. In an multiuser scenario, this will lead to deadlocks, since the same tables are being read and locks being applied to find dependend children. Introducing an index on all the columns that are foreign key referenced helps a lot in this case.\n","date":"3 September 2009","externalUrl":null,"permalink":"/post/create_an_index_for_all_fk_columns/","section":"Posts","summary":"Most of the time I have seen database foreign key constraints on tables without indexes on those columns. Lets say the application is trying to delete a row from the CUSTOMER table\n","title":"Create an Index for all FK Columns in the database","type":"post"},{"content":"","date":"3 September 2009","externalUrl":null,"permalink":"/tags/index/","section":"Tags","summary":"","title":"Index","type":"tags"},{"content":"Recently one of my colleague Jeff Norris had a weird error. He was trying to build a materialized view over some tables in his local database and some tables in his remote database using database links the sql to create the view ran fine and provided the results as expected, but when put inside a materialized view statement complained with ORA-00942 errors.\nLets say the two databases in question are local and remote, so the sql to create the materialized view to load immediately and refresh everyday is\nCREATE MATERIALIZED VIEW MV_CUSTOMERBALANCE BUILD IMMEDIATE REFRESH FORCE START WITH ROUND(SYSDATE) + 23/24 NEXT SYSDATE + 1 AS SELECT customer.name , account.balance, accounttype.name FROM customer , account@remotedb account, accounttype@remotedb accounttype WHERE customer.id = account.customerid AND account.accounttyppeid = accounttype.id / Oracle started to complain when creating the above materialized view issuing an error ORA-00942: table or view does not exist, but the SQL without the create materialized view command ran fine giving the expected results.\nSELECT customer.name , account.balance, accounttype.name FROM customer , account@remotedb account, accounttype@remotedb accounttype WHERE customer.id = account.customerid AND account.accounttyppeid = accounttype.id / After some searching around and experimenting I found, in the create materialized view statement the database link name can be used only once, which meant we can only use the \u0026ldquo;remotedb\u0026rdquo; name once, we got around this restriction by creating two database links to the remote database as REMOTEACCOUNT and REMOTEACCOUNTTYPE and using them in the creation of the materialized view as shown below.\nCREATE MATERIALIZED VIEW MV_CUSTOMERBALANCE BUILD IMMEDIATE REFRESH FORCE START WITH ROUND(SYSDATE) + 23/24 NEXT SYSDATE + 1 AS SELECT customer.name , account.balance, accounttype.name FROM customer , account@remoteaccount account, accounttype@remoteaccounttype accounttype WHERE customer.id = account.customerid AND account.accounttyppeid = accounttype.id / ","date":"10 August 2009","externalUrl":null,"permalink":"/post/materialized_views_and_databases/","section":"Posts","summary":"Recently one of my colleague Jeff Norris had a weird error. He was trying to build a materialized view over some tables in his local database and some tables in his remote database using database links the sql to create the view ran fine and provided the results as expected, but when put inside a materialized view statement complained with ORA-00942 errors.\n","title":"Materialized views and database links in oracle.","type":"post"},{"content":"Okay this is kind of a rant, maybe I\u0026rsquo;m too picky or just that I hate to see perfectly good data not being used. This is how it goes..\nI go regularly to this store to get Horizon organic milk for my family, about 60% of the time I see milk I need NOT in stock, okay I can live with that, may be lots of folks are buying organic milk, but not when it happens frequently, especially when the store knows how much milk was ordered (or supplied from the warehouse) and how much milk was sold, the store should be able to figure out that organic milk gets sold out pretty fast, putting my Business Intelligence (BI) hat on, I think the store should be able to predict when they are going to run out of organic milk ( for that matter any product), its especially frustrating when they have all the data they need to get it done.\nOne more non usage of data that really makes me red is, when the organic milk in the store is already expired (past the sell by date). I mean how hard is it for someone to generate a list of all the products that expire today and ask the store associates to remove them from the shelves by the end of the day, especially when its edible items.\n","date":"5 August 2009","externalUrl":null,"permalink":"/post/use_the_data_you_have_already/","section":"Posts","summary":"Okay this is kind of a rant, maybe I’m too picky or just that I hate to see perfectly good data not being used. This is how it goes..\nI go regularly to this store to get Horizon organic milk for my family, about 60% of the time I see milk I need NOT in stock, okay I can live with that, may be lots of folks are buying organic milk, but not when it happens frequently, especially when the store knows how much milk was ordered (or supplied from the warehouse) and how much milk was sold, the store should be able to figure out that organic milk gets sold out pretty fast, putting my Business Intelligence (BI) hat on, I think the store should be able to predict when they are going to run out of organic milk ( for that matter any product), its especially frustrating when they have all the data they need to get it done.\n","title":"Perfectly good data.. wasted","type":"post"},{"content":"Dead lock is caused in the database when you have resources (connections) waiting for other connections to release locks on the rows that are needed by the session, resulting in all session being blocked. Oracle automatically detects deadlocks are resolves the deadlock by rolling back the statement in the transaction that detected the deadlock. Thing to remember is that last statement is rolled back and not the whole transaction, which means that if you had other modifications, those rows are still locked and the application should make sure that it does a explicit rollback on the connection.\nFor example. Lets assume there are two tables Parent(ParentID) and Child(ChildID)\nSESSION_A \u0026gt;create table parent (parentId number(10)); Table created. SESSION_A \u0026gt;create table child (childId number(10)); Table created. SESSION_A \u0026gt;insert into parent values (100); 1 row created. SESSION_A \u0026gt;insert into child values (200); 1 row created. SESSION_A \u0026gt;commit; Commit complete. SESSION_A \u0026gt;select * from parent; PARENTID 100 SESSION_A \u0026gt;select * from child; CHILDID 200 SESSION_A \u0026gt; Now lets create a situation where a deadlock happens. There are two sessions connected to the same database and same user, SESSION_A and SESSION_B are the two sessions in question.\nSESSION_A \u0026gt;update parent set parentid = 1000 where parentid=100; 1 row updated. SESSION_B \u0026gt;update child set childid = 2000 where childid = 200; 1 row updated. SESSION_B \u0026gt;update parent set parentid = 2001 where parentid=100; --Waiting For Lock on Row in Parent Table, held by SESSION_A SESSION_A \u0026gt;update child set childid = 1001 where childid = 200; update child set childid = 1001 where childid = 200 * ERROR at line 1: ORA-00060: deadlock detected while waiting for resource --SESSION_A requesting lock on row, held by SESSION_B causing deadlock. SESSION_A \u0026gt; After you get the ORA-00060 error the statement update child set childid = 1001 where childid = 200; is rolled back.. but SESSION_B is still waiting for the lock on the Parent table to be released.\nSo when your application get the ORA-00060 or any deadlock exception in any other database, explicitly rollback your transaction (not just the current statement) so that all the changes made in the transaction and all the locks held by the transaction are released.\n","date":"26 May 2009","externalUrl":null,"permalink":"/post/explicitly_rollback_when_you_get_deadlock/","section":"Posts","summary":"Dead lock is caused in the database when you have resources (connections) waiting for other connections to release locks on the rows that are needed by the session, resulting in all session being blocked. Oracle automatically detects deadlocks are resolves the deadlock by rolling back the statement in the transaction that detected the deadlock. Thing to remember is that last statement is rolled back and not the whole transaction, which means that if you had other modifications, those rows are still locked and the application should make sure that it does a explicit rollback on the connection.\n","title":"Explicitly rollback when you encounter a deadlock.","type":"post"},{"content":"Ever since I moved to the Mac, I had to run some other OS inside a VM so that I could run Oracle and use it, since Oracle was not available for the the Mac. Now that is no longer the case. Oracle 10gR2 (10.2.0.4) is now available for Mac\nThis is especially nice since the Oracle for Mac was the most voted requirement on mix.oracle.com\n","date":"14 May 2009","externalUrl":null,"permalink":"/post/oracle_for_the_mac/","section":"Posts","summary":"Ever since I moved to the Mac, I had to run some other OS inside a VM so that I could run Oracle and use it, since Oracle was not available for the the Mac. Now that is no longer the case. Oracle 10gR2 (10.2.0.4) is now available for Mac\n","title":"Oracle for the Mac","type":"post"},{"content":"In Oracle 10g and before we all know that passwords are not case sensitive, so PASSWORD, Password, password would let you in and everything would be okay.\nIf you upgrade to Oracle 11g (I know lot of you are waiting for 11gR2), you will find that passwords are case sensitive. Here is an example of case sensitive passwords.\nc:\\Software\u0026gt;sqlplus bddd/bddd@dosa SQL*Plus: Release 11.1.0.6.0 - Production on Wed May 6 15:17:43 2009 Copyright (c) 1982, 2007, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production With the Partitioning and OLAP options BDDD@dosa \u0026gt; Lets try to connect with a upper case password\nc:\\Software\u0026gt;sqlplus bddd/BDDD@dosa SQL*Plus: Release 11.1.0.6.0 - Production on Wed May 6 15:19:25 2009 Copyright (c) 1982, 2007, Oracle. All rights reserved. ERROR: ORA-01017: invalid username/password; logon denied Enter user-name: So what does this mean to apps running with 10g, that get ported to run with 11g. Make sure that the password set in the properties files is of the correct case.\nYou can also revert to 10g behavior by changing sec_case_sensitive_logon parameter to FALSE, since its TRUE by default.\nalter system set sec_case_sensitive_logon=FALSE; ","date":"6 May 2009","externalUrl":null,"permalink":"/post/oracle_11g_password_is_case_sensitive/","section":"Posts","summary":"In Oracle 10g and before we all know that passwords are not case sensitive, so PASSWORD, Password, password would let you in and everything would be okay.\nIf you upgrade to Oracle 11g (I know lot of you are waiting for 11gR2), you will find that passwords are case sensitive. Here is an example of case sensitive passwords.\n","title":"In Oracle 11g password is case sensitive","type":"post"},{"content":"Oracle has metadata about all its objects in various tables/views. One such view is the USER_OBJECTS or ALL_OBJECTS, this view has a column named as STATUS which shows you if the given object is VALID or INVALID. The status applies to DB Code (Stored Procedures, Functions, Triggers etc).\nTo find all the INVALID objects in the schema, issue\nSELECT * FROM USER_OBJECTS WHERE STATUS=\u0026#39;INVALID\u0026#39;; One problem with the way oracle maintains this metadata is, changing the underlying table on which the DB Code depends, oracle marks the objects are INVALID even though the underlying table may have changed in such a way, that it does not affect the DB Code at all (like adding a new column, or making a colum nullable). Here is some code which shows you what I mean. Run it through SQLPlus.\nCOLUMN OBJECT_NAME FORMAT A30 COLUMN STATUS FORMAT A15 spool objects.log CREATE TABLE FOO (ID NUMBER(10), NAME VARCHAR2(30)); CREATE OR REPLACE TRIGGER TRIG_FOO BEFORE INSERT OR UPDATE ON FOO REFERENCING OLD AS OLD NEW AS NEW FOR EACH ROW BEGIN IF :NEW.name IS NULL THEN :NEW.name := \u0026#39;NOT AVAILABLE\u0026#39;; END IF; END; / CREATE OR REPLACE FUNCTION FUNCTION_GET_NAME_FOR_FOOID(inFooId number) RETURN VARCHAR2 IS fooName VARCHAR2(30); BEGIN BEGIN SELECT name INTO fooName FROM foo WHERE id = inFooId ; EXCEPTION WHEN NO_DATA_FOUND THEN RETURN \u0026#39;NOT FOUND\u0026#39;; END; RETURN fooName; END; / SELECT OBJECT_NAME,STATUS FROM USER_OBJECTS WHERE STATUS=\u0026#39;INVALID\u0026#39;; ALTER TABLE FOO ADD ( DESCRIPTION VARCHAR2(100)); SELECT OBJECT_NAME,STATUS FROM USER_OBJECTS WHERE STATUS=\u0026#39;INVALID\u0026#39;; spool off To get the objects back to VALID status, all that needs to be done is\nALTER TRIGGER TRIG_FOO COMPILE; ALTER FUNCTION FUNCTION_GET_NAME_FOR_FOOID COMPILE; SELECT OBJECT_NAME,STATUS FROM USER_OBJECTS WHERE STATUS=\u0026#39;INVALID\u0026#39;; ","date":"31 March 2009","externalUrl":null,"permalink":"/post/oracle_metadata_can_be_misleading/","section":"Posts","summary":"Oracle has metadata about all its objects in various tables/views. One such view is the USER_OBJECTS or ALL_OBJECTS, this view has a column named as STATUS which shows you if the given object is VALID or INVALID. The status applies to DB Code (Stored Procedures, Functions, Triggers etc).\n","title":"Oracle Metadata can be mis-leading","type":"post"},{"content":"My Presentation on Database Refactoring at QCon was recorded and is live now on infoQ here\n","date":"26 March 2009","externalUrl":null,"permalink":"/post/presentation_on_database_refactoing/","section":"Posts","summary":"My Presentation on Database Refactoring at QCon was recorded and is live now on infoQ here\n","title":"Presentation on Database Refactoring","type":"post"},{"content":"We came across a need to save just the Time in the database, the requirement is to store time of the day, like say the user likes to have Breakfast at 8.15AM and Lunch at 12.32PM etc. Off course oracle does not have a time only data type. So we ended up using DATE as the data type and just setting the time. for example:\nCREATE TABLE FOO (PREFERRED_TIME DATE NULL); INSERT INTO FOO (TO_DATE(\u0026#39;11:34\u0026#39;,\u0026#39;HH24:MI\u0026#39;)); oracle automatically sets the date to the first day of the current month. so when you do a select from the FOO table the data would be\nSELECT TO_CHAR(PREFERRED_TIME, \u0026#39;dd-mon-yyyy hh24:mi:ss\u0026#39;) from FOO; 01-feb-2009 11:24:00 on the client side you will have to know to ignore the date component, this is possible area for confusion in future, since I could have many preferences over different months and my preferred time would have different date components, ideally I would just want the time, otherwise I would think having a time component and the date being a constant known value like 01/01/0001. We could achieve this using a BEFORE INSERT/UPDATE trigger which keeps the time component but updates the date component to a known constant value or you can also use your OR mapping layer like hibernate to set the value as such. If you are using hibernate, you can map the filed using the java.sql.Time object and the date is automatically set to 01/01/1970. SqlServer 2008 seems has a Time data type.\n","date":"12 February 2009","externalUrl":null,"permalink":"/post/storing_time_in_oracle/","section":"Posts","summary":"We came across a need to save just the Time in the database, the requirement is to store time of the day, like say the user likes to have Breakfast at 8.15AM and Lunch at 12.32PM etc. Off course oracle does not have a time only data type. So we ended up using DATE as the data type and just setting the time. for example:\n","title":"Storing just the time in Oracle","type":"post"},{"content":"Some environments like to have access to the database tables routed via stored procedures. Instead of using Create/Read/Update/Delete aka CRUD with DML, stored procedures are invoked with the parameters to perform the required operation. I\u0026rsquo;m not arguing about the benefits/pitfalls of this approach, if you have to do stored procedures, here are some things to look at.\nMake the stored procedure handle one object/table only and not multiple objects or tables. Do not commit open transactions inside the stored procedures. Do not do business logic in stored procedures. If they are straight CRUD stored procedures, see if you can generate the stored procedure code using some metadata? Make sure creation and execution of the stored procedures is part of your Continuous Integration build and developer build. Make sure stored procedures (or the metadata used to generate them) is under Version Control, have seen many problems when the stored procedure version does not match application code version Develop against the production stack database. Make sure exceptions thrown by the database are passed back to the application. ","date":"3 February 2009","externalUrl":null,"permalink":"/post/considerations_when_using_procedures/","section":"Posts","summary":"Some environments like to have access to the database tables routed via stored procedures. Instead of using Create/Read/Update/Delete aka CRUD with DML, stored procedures are invoked with the parameters to perform the required operation. I’m not arguing about the benefits/pitfalls of this approach, if you have to do stored procedures, here are some things to look at.\n","title":"Considerations when using stored procedures for doing CRUD","type":"post"},{"content":"Recently I have been thinking a lot about making specification and behavior expected about the database and the code that interfaces with the databases executable. The Behavior Driven Design has a lot of parallels in the database world.\nJust finished writing a article about Behavior Driven Design applied to Databases or Behavior Driven Database Design for Methods and Tools.\n","date":"22 December 2008","externalUrl":null,"permalink":"/post/article_about_behavior_driven/","section":"Posts","summary":"Recently I have been thinking a lot about making specification and behavior expected about the database and the code that interfaces with the databases executable. The Behavior Driven Design has a lot of parallels in the database world.\n","title":"Article about Behavior Driven Database Design is out","type":"post"},{"content":"Consider this Hibernate mapping\n@Column(name = \u0026#34;qReferenceId\u0026#34;) public Long getQReferenceId() { return qReferenceId; } Where qReferenceId is data provided to our application via a external reference, we do not have a QReference Object or Table for FK references. When trying to query this object using DetachedQuery, this Simple expression was used.\npublic List\u0026lt;Movie\u0026gt; findByQReferenceId(Long id) { final SimpleExpression matchesId = Property.forName(\u0026#34;qReferenceId\u0026#34;).eq(id); DetachedCriteria criteria = DetachedCriteria.forClass(Movie.class); criteria = criteria.add(matchesId); List\u0026lt;Movie\u0026gt; movies = (List\u0026lt;Movie\u0026gt;) getHibernateTemplate().findByCriteria(criteria); return movies; } When running this method, I kept getting errors shown below.\ncould not resolve property: qReferenceId of: com.example.Movie; nested exception is org.hibernate.QueryException: could not resolve property: qReferenceId of: com.example.Movie .... .... I thought I had a spelling mistake on the property name and also tried many other combinations. Finally I said why not use \u0026ldquo;QReferenceId\u0026rdquo; and suddenly everything was hunky dory, this kind of weirdness does not happen when working with property names that do not have consecutive double uppercase characters in the getter/setter for the property. very interesting. So this method worked\npublic List\u0026lt;Movie\u0026gt; findByQReferenceId(Long id) { final SimpleExpression matchesId = Property.forName(\u0026#34;QReferenceId\u0026#34;).eq(id); DetachedCriteria criteria = DetachedCriteria.forClass(Movie.class); criteria = criteria.add(matchesId); List\u0026lt;Movie\u0026gt; movies = (List\u0026lt;Movie\u0026gt;) getHibernateTemplate().findByCriteria(criteria); return movies; } ","date":"24 October 2008","externalUrl":null,"permalink":"/post/hibernate_weirdness_with_properties/","section":"Posts","summary":"Consider this Hibernate mapping\n@Column(name = \"qReferenceId\") public Long getQReferenceId() { return qReferenceId; } Where qReferenceId is data provided to our application via a external reference, we do not have a QReference Object or Table for FK references. When trying to query this object using DetachedQuery, this Simple expression was used.\n","title":"Hibernate weirdness with property names","type":"post"},{"content":"","date":"30 September 2008","externalUrl":null,"permalink":"/tags/standards/","section":"Tags","summary":"","title":"Standards","type":"tags"},{"content":"Recently when our test databases where upgraded new version of Oracle, we started noticing that the order in which some drop down lists were being displayed was not correct. It turns out that the SELECT statement we had, did not have a ORDER BY clause and the data was being returned in the ORDER of the creation of rows (in the order of ROWID) when the database was upgraded these ROWID\u0026rsquo;s got changed and hence the ORDER of the data being shown in the drop down lists.\nLesson learnt: have a EXPLICIT ORDER BY in every SQL that provides data to be shown on the screen. In other words do not rely on the order of the result set provided by the current version of the database server. if you prefer a certain order make that choice explicit in the SQL SELECT statement.\n","date":"30 September 2008","externalUrl":null,"permalink":"/post/using_explicit_order_by_in_sql/","section":"Posts","summary":"Recently when our test databases where upgraded new version of Oracle, we started noticing that the order in which some drop down lists were being displayed was not correct. It turns out that the SELECT statement we had, did not have a ORDER BY clause and the data was being returned in the ORDER of the creation of rows (in the order of ROWID) when the database was upgraded these ROWID’s got changed and hence the ORDER of the data being shown in the drop down lists.\n","title":"Using Explicit Order By in your SQL Statements","type":"post"},{"content":"Couple of weeks back I was given a choice to upgrade my work Laptop to a Mac Book Pro or a Windows Laptop. I choose Mac ( I know everyone is into macs nowadays). The transition was pretty good, with the exception of moving my oracle database from windows to mac, since there is no native installation of oracle on mac I had to use VMWare fusion to install oracle.\nAfter having oracle run inside the VM, I started setting up the dev environment, I ran into issues with ANT version and JAVA version but they where resolved by pointing to the right location.\nOverall this has been a positive experience so far.\n","date":"1 September 2008","externalUrl":null,"permalink":"/post/moved_to_a_mac/","section":"Posts","summary":"Couple of weeks back I was given a choice to upgrade my work Laptop to a Mac Book Pro or a Windows Laptop. I choose Mac ( I know everyone is into macs nowadays). The transition was pretty good, with the exception of moving my oracle database from windows to mac, since there is no native installation of oracle on mac I had to use VMWare fusion to install oracle.\n","title":"Moved to a Mac","type":"post"},{"content":"When creating a Foreign Key constraint on the database as shown below\nALTER TABLE BOOK ADD (CONSTRAINT FK_BOOK_AUTHOR FOREIGN KEY (AUTHORID) REFERENCES AUTHOR) / In the above example we are telling the database to check if the BOOK.AUTHORID is a valid value in the Author.AuthorID. When the Author table is being changed, the database does data verification on the BOOK table using SELECT against the BOOK table for the AUTHORID some thing like this\nSELECT count(*) FROM BOOK WHERE AUTHORID = nnnn Basically the database server is trying to check if it has children rows for the row that just changed (inserted or deleted). While doing this if there is not index on BOOK.AUTHORID, the database will have to scan the whole table which is slow. Hence when creating a Foreign Key constraint, remember to create a corresponding INDEX on the table, so that the performance does not degrade, or when observing slow performance on a database after you put in Foreign Key constraints. Make sure to look for Indexes on the columns that are constrained.\n","date":"15 July 2008","externalUrl":null,"permalink":"/post/create_a_index_for_every_foreign_key/","section":"Posts","summary":"When creating a Foreign Key constraint on the database as shown below\nALTER TABLE BOOK ADD (CONSTRAINT FK_BOOK_AUTHOR FOREIGN KEY (AUTHORID) REFERENCES AUTHOR) / In the above example we are telling the database to check if the BOOK.AUTHORID is a valid value in the Author.AuthorID. When the Author table is being changed, the database does data verification on the BOOK table using SELECT against the BOOK table for the AUTHORID some thing like this\n","title":"Create a Index for every Foreign Key constraint created","type":"post"},{"content":"So we version control/source control everything on our project.. code/data/artifacts/diagrams etc. yesterday I said why not extend it to my writings to everything I have. So I started this long journey of refactoring my folder layout and making a nice folder structure to hold all the things I have written about have other artifacts in the process of writing and moved them all to subversion, now all my example code and writings are all under version control that gets backed up everyday\u0026hellip;. feels liberating\n","date":"6 June 2008","externalUrl":null,"permalink":"/post/version_control_your_work/","section":"Posts","summary":"So we version control/source control everything on our project.. code/data/artifacts/diagrams etc. yesterday I said why not extend it to my writings to everything I have. So I started this long journey of refactoring my folder layout and making a nice folder structure to hold all the things I have written about have other artifacts in the process of writing and moved them all to subversion, now all my example code and writings are all under version control that gets backed up everyday…. feels liberating\n","title":"Version Control your work..","type":"post"},{"content":"Japanese translation of Refactoring Databases: Evolutionary Database Designhas been released, thanks to Yasuo Honda for the information. The Japanese version can be found here\n","date":"4 June 2008","externalUrl":null,"permalink":"/post/japanese_version_released/","section":"Posts","summary":"Japanese translation of Refactoring Databases: Evolutionary Database Designhas been released, thanks to Yasuo Honda for the information. The Japanese version can be found here\n","title":"Japanese Version released\"","type":"post"},{"content":"I\u0026rsquo;m always on the lookout for better tool support to do database refactoring. Just noticed that liquibase has come out with a IntelliJ plugin to support database refactoring.\nThis is really cool and hopefully one of long list of tools that will support database refactoring in the future. so enjoy\n","date":"20 May 2008","externalUrl":null,"permalink":"/post/tool_support_for_database_refactoring/","section":"Posts","summary":"I’m always on the lookout for better tool support to do database refactoring. Just noticed that liquibase has come out with a IntelliJ plugin to support database refactoring.\nThis is really cool and hopefully one of long list of tools that will support database refactoring in the future. so enjoy\n","title":"Tool support for Database Refactoring","type":"post"},{"content":"We had a weird requirement on our project recently..\nFind all the Rows in All the tables that do not comply with the Constraints that we have in development but not in QA environments\nBest way to do this we thought was to write a SQL statement against the table for each column that was going to have a Foreign Key constrained column and find out what data was not right or did not match the constraint. For example: If we have a INVOICE table that has a ITEMID on it. I want to find all the rows in the INVOICE table that have a ITEMID that does not exist in the ITEM table. Writing this SQL for our 400+ tables database was going to be huge task.\nOracles (or for that matter any databases metadata) metadata to the rescue and we ended up writing a SQL that would generate our above SQL.\nhere is the SQL that generated the above SQL\nSELECT \u0026#39;SELECT \u0026#39;\u0026#39;\u0026#39;||table_name||\u0026#39;-\u0026#39;||column_name||\u0026#39;\u0026#39;\u0026#39;, count(*) FROM \u0026#39;|| table_name|| \u0026#39; WHERE not exists (select 1 from \u0026#39;|| remote_table ||\u0026#39; where \u0026#39;|| remote_table||\u0026#39;.\u0026#39;||remote_column||\u0026#39; = \u0026#39;||table_name||\u0026#39;.\u0026#39;||column_name||\u0026#39;) AND \u0026#39; ||table_name||\u0026#39;.\u0026#39;||column_name||\u0026#39; IS NOT NULL UNION ALL\u0026#39; FROM ( SELECT a.table_name, column_name, ( SELECT table_name FROM user_constraints WHERE constraint_name = a.R_CONSTRAINT_NAME) remote_table, ( SELECT column_name FROM user_cons_columns WHERE constraint_name = a.R_CONSTRAINT_NAME) remote_column FROM user_constraints a, user_cons_columns b WHERE a.constraint_name = b.constraint_name AND a.constraint_type = \u0026#39;R\u0026#39; ) This SQL generates SQL that when run will give us data about tables that do not match our constraints requirements. If you have a CUSTOMER table which has CUSTOMERTYPEID and STATUSID on it, then the SQL generated would be.\nSELECT \u0026#39;CUSTOMER-CUSTOMERTYPEID\u0026#39;, COUNT(*) FROM CUSTOMER WHERE NOT EXISTS (SELECT 1 FROM CUSTOMERTYPE WHERE CUSTOMERTYPE.CUSTOMERTYPEID = CUSTOMER.CUSTOMERTYPEID) AND CUSTOMER.CUSTOMERTYPEID IS NOT NULL UNION ALL SELECT \u0026#39;CUSTOMER-STATUSID\u0026#39;, COUNT(*) FROM CUSTOMER WHERE NOT EXISTS (SELECT 1 FROM STATUS WHERE STATUS.STATUSID = CUSTOMER.STATUSID) AND CUSTOMER.STATUSID IS NOT NULL Once the above SQL is run, the results will show us data that does not match the constraints we want to introduce into the QA environments.\n","date":"6 May 2008","externalUrl":null,"permalink":"/post/writing_a_sql_to_generate_a_sql/","section":"Posts","summary":"We had a weird requirement on our project recently..\nFind all the Rows in All the tables that do not comply with the Constraints that we have in development but not in QA environments\n","title":"Writing a SQL to generate a SQL","type":"post"},{"content":"When doing Performance Testing or running Unit/Functional tests on a database, there is a need to periodically get the database to a known state, so that the tests behave in a predictable way and to get rid of all the data created by the tests. Some of the ways to get a clean database are.\nUsing Scripts: Recreate the database using scripts, the same scripts that are used in development environment.\nUsing DB Backup: Especially when the database (and the data set) is large (using the scripts approach above will be slow) is to make a backup of the database in its pristine state and then run the tests, once the tests are done running, restore the database with the backup that was done before the tests corrupted the data.\nUsing Virtual Machine: The DB backup approach can be improved by using Virtual Machine (VM). Setup a VM and run the database server inside the VM, get the database and data so that the tests can run. Now make a image of the VM and run the tests, when the tests are done all that needs to be done is to restore the image of the VM.\n","date":"17 April 2008","externalUrl":null,"permalink":"/post/setup_and_teardown_of_database/","section":"Posts","summary":"When doing Performance Testing or running Unit/Functional tests on a database, there is a need to periodically get the database to a known state, so that the tests behave in a predictable way and to get rid of all the data created by the tests. Some of the ways to get a clean database are.\n","title":"Setup and Teardown of database during testing","type":"post"},{"content":"Me, Andy, Jeff and Marjorie discuss how to keep a long running project fit in this Podcast, also on iTunes. We discuss the management of technology, people, processes and tools on longer and more mature applications. Specific topics such as refactoring, knowledge management, innovation, staffing, production support and others are covered.\n","date":"18 January 2008","externalUrl":null,"permalink":"/post/podcast_on_keeping_gray_code_fit/","section":"Posts","summary":"Me, Andy, Jeff and Marjorie discuss how to keep a long running project fit in this Podcast, also on iTunes. We discuss the management of technology, people, processes and tools on longer and more mature applications. Specific topics such as refactoring, knowledge management, innovation, staffing, production support and others are covered.\n","title":"Podcast on Keeping Gray code Fit","type":"post"},{"content":"We have been using DBDeploy on my project for more than 6 months now and wanted to show how things are going. First lets talk about set up, we are using dbdeploy in our Java development environment with ANT as our build scripting tool, against a Oracle 10g database.\nDefine the ANT task first\n\u0026lt;taskdef name=\u0026#34;dbdeploy\u0026#34; classname=\u0026#34;net.sf.dbdeploy.AntTarget\u0026#34; classpath=\u0026#34;lib/dbdeploy.jar\u0026#34;/\u0026gt; Now we create the main dbinitialize task a ANT task to create you database schema, using the upgrade generated by the dbdeploy file shown below. The thing to note is that dbdeploy generates the upgrade file but does not run it against your database, so we have to make sure we call the generated upgrade file via a sql ANT task.\n\u0026lt;target name=\u0026#34;dbinit\u0026#34; depends=\u0026#34;init,dbclean\u0026#34;\u0026gt; \u0026lt;echo message=\u0026#34;Working UserName: ${db.user}\u0026#34;/\u0026gt; \u0026lt;mkdir dir=\u0026#34;${migrationfolder}\u0026#34;/\u0026gt; \u0026lt;dbdeploy driver=\u0026#34;${driver.name}\u0026#34; url=\u0026#34;${db.url}\u0026#34; userid=\u0026#34;${db.user}\u0026#34; password=\u0026#34;${db.password}\u0026#34; deltaset=\u0026#34;couger\u0026#34; dir=\u0026#34;db/migration\u0026#34; outputfile=\u0026#34;${migrationfolder}/upgrade.sql\u0026#34; dbms=\u0026#34;ora\u0026#34; undoOutputfile=\u0026#34;${migrationfolder}/undo.sql\u0026#34;/\u0026gt; \u0026lt;!--Now run the generate upgrade file --\u0026gt; \u0026lt;sql password=\u0026#34;${db.password}\u0026#34; userid=\u0026#34;${db.user}\u0026#34; url=\u0026#34;${db.url}\u0026#34; driver=\u0026#34;${driver.name}\u0026#34; classpath=\u0026#34;${driver.classpath}\u0026#34; onerror=\u0026#34;abort\u0026#34;\u0026gt; \u0026lt;fileset includes=\u0026#34;upgrade.sql\u0026#34; dir=\u0026#34;${migrationfolder}\u0026#34;/\u0026gt; \u0026lt;/sql\u0026gt; \u0026lt;/target\u0026gt; So lets say we want to write the first migration (migration or delta is the same in this context) we will create a new file in the db/migration folder named as 001_CreateCustomerTable.sql the 001 is just a number to sequence the migrations, dbdeploy only cares that the numbers increment and CreateCustomerTable is used to give it a meaningful name, you can name the migration as 1.sql but thats not meaningful is it? neither does the name really say what its doing. When we are done writing the migration and confirm that it works locally, we check in the file (we used subversion). When CruiseControl build was done we also published all the migrations from db/migration folder on CruiseControl artifacts page using the onsuccess event\n\u0026lt;publishers\u0026gt; \u0026lt;onsuccess\u0026gt; ... \u0026lt;artifactspublisher dest=\u0026#34;artifacts/${project.name}\u0026#34; dir=\u0026#34;projects/${project.name}/db/migration\u0026#34;/\u0026gt; \u0026lt;/onsuccess\u0026gt; \u0026lt;/publishers\u0026gt; Publishing the migrations allowed us to know what all migrations are needed for this particular build. we have deployed to production twice already and have found this process to be smooth, doing migrations has allowed us to test our migrations hundreds of times and also test them against a copy of the production database before hand so that we can tune the migrations for performance if needed. Delivering the migrations to the client is also easy since its pure SQL that the client DBA\u0026rsquo;s can look at and be comfortable about the migration/upgrade of their database.\n","date":"14 January 2008","externalUrl":null,"permalink":"/post/exprerience_using_dbdeploy_on_project/","section":"Posts","summary":"We have been using DBDeploy on my project for more than 6 months now and wanted to show how things are going. First lets talk about set up, we are using dbdeploy in our Java development environment with ANT as our build scripting tool, against a Oracle 10g database.\n","title":"Experience using DBDeploy on my project","type":"post"},{"content":"I have been working on a project that I had worked in 2005, trying to get a handle on what I had done about 3 years back. Exploring code and the database has been fun, also discovering the data layout and building new set of data for production has been extremely entertaining. What I learnt from this whole experience was this if your code(application or other wise) is not expecting data, this data should not be provided or even considered valid by the database, the database should be designed such that it does not even allow invalid combinations of the data.\nHere is an example. We have a base table named as SETTINGS, that can be changed by the user by overriding the value in the extension table known as SETTINGSEXTENSION, table structure is show below\nSETTINGS ID (PK) KEY (BusinessKey) VALUE (Value for the Key) SETTINGSEXTENTION ID (PK) SETTINGSID (FK to SETTINGS table) OVERRIDDENVALUE (Value overriding the value in the SETTINGS table) BYUSERID (Value over ridden for user) in this scheme the same user can create extensions for the same base key causing the application to throw exceptions. If we introduce unique index on SETTINGSEXTENSION TABLE(SETTINGSID, BYUSERID) we ensure that the database does not allow this data and makes sure that the application will not barf.\nThere are many more other things I learnt.. those will follow..\n","date":"27 December 2007","externalUrl":null,"permalink":"/post/relearning/","section":"Posts","summary":"I have been working on a project that I had worked in 2005, trying to get a handle on what I had done about 3 years back. Exploring code and the database has been fun, also discovering the data layout and building new set of data for production has been extremely entertaining. What I learnt from this whole experience was this if your code(application or other wise) is not expecting data, this data should not be provided or even considered valid by the database, the database should be designed such that it does not even allow invalid combinations of the data.\n","title":"Lessons from Re-Learning","type":"post"},{"content":"After a lot of frustration about my schedule, I have had to come to this conclusion, that I cannot physically make it to London, XP day.\nI\u0026rsquo;m going to miss it. Nick Ashley is going to take up my spot and I know he will do a great job.\n","date":"13 November 2007","externalUrl":null,"permalink":"/post/cannot_make_it_out_to_xp_day_london/","section":"Posts","summary":"After a lot of frustration about my schedule, I have had to come to this conclusion, that I cannot physically make it to London, XP day.\nI’m going to miss it. Nick Ashley is going to take up my spot and I know he will do a great job.\n","title":"Cannot make it out to XP Day London","type":"post"},{"content":"Why do Evolutionary Design or Iterative Design or Incremental Design? Everyone who has not worked in an evolutionary manner asks this? My answer, if you think the system you designed is NOT GOING TO CHANGE EVER then sure you can do design once and deploy once and you are done, move on to next project. But tell me one project you have been on, that does not have any changes in requirements, changes in technology, changes in look and feel etc after it was deployed.\nSo if every project changes after it was deployed, why live in the fallacy that nothing is going to change ever.\nInstead, since you know requirements change, why not get better at managing change, which mean get better at Evolutionary Design or Iterative Design or Incremental Design, so that you are ready for the next requirement change that comes along.\n","date":"5 November 2007","externalUrl":null,"permalink":"/post/why_do_evolutionary_design/","section":"Posts","summary":"Why do Evolutionary Design or Iterative Design or Incremental Design? Everyone who has not worked in an evolutionary manner asks this? My answer, if you think the system you designed is NOT GOING TO CHANGE EVER then sure you can do design once and deploy once and you are done, move on to next project. But tell me one project you have been on, that does not have any changes in requirements, changes in technology, changes in look and feel etc after it was deployed.\n","title":"Why do Evolutionary Design","type":"post"},{"content":"Thoughtworks is going to be at Oracle Open World. I\u0026rsquo;m excited about this especially since it will give ThoughtWorks and Me to talk about software practices and how to apply these software practices to the database development world, off course I will talk about my books Refactoring Databases and Continuous Database Integration. ThoughtWorks will have a booth at 343 Moscone South and I will be there on Nov 14.\n","date":"30 October 2007","externalUrl":null,"permalink":"/post/thoughtworks_at_oracle_open_world/","section":"Posts","summary":"Thoughtworks is going to be at Oracle Open World. I’m excited about this especially since it will give ThoughtWorks and Me to talk about software practices and how to apply these software practices to the database development world, off course I will talk about my books Refactoring Databases and Continuous Database Integration. ThoughtWorks will have a booth at 343 Moscone South and I will be there on Nov 14.\n","title":"ThoughtWorks at Oracle Open World","type":"post"},{"content":"Recently talking to someone about a persistence framework that they are using, this framework creates a rows in the database table with just Primary Key value and then later on updates the table row with values for other columns. Because of this framework, the tables cannot have any NOT-NULL constraints defined.\nFor example, the framework is doing\nINSERT INTO customer (customerId, name, birthdate) VALUES (1,null,null); UPDATE customer set name = \u0026#39;FOO Name\u0026#39;, birthdate = \u0026#39;12/12/1978\u0026#39; WHERE customerId = 1; You cannot have NON-NULL constraint defined on name or birthdate column, since the INSERT statement would blow up, forcing you to change table design\nCREATE TABLE Customer ( CustomerId NUMBER NOT NULL, Name VARCHAR2(40) NULL, birthdate DATE NULL, CONSTRAINT PK_Customer PRIMARY KEY (CustomerId) ) / What can you do, so that table design does not suffer? and have better data quality constraints? one argument we see is that, application will constrain the data and not allow customers with null name or birthdate to be persisted. I have seen way too many projects where this argument fails after some time when someone starts to import customers from other system or starts to implement a service to push customers and now the constraint in the application layer is pretty useless allowing bad data to get in, since there is nothing stopping these bad rows from getting in the Customer table.\nDiffered constraints come to the rescue here in oracle (bet there is equivalent types of differed constraint checking for other database vendors) Using the Customer table example, create a differed constraint on Name and BirthDate columns, as shown below\nALTER TABLE Customer ADD (CONSTRAINT CHK_Customer_Name_NotNull CHECK (Name IS NOT NULL) DEFERRABLE INITIALLY DEFERRED) / ALTER TABLE Customer ADD (CONSTRAINT CHK_Customer_BirthDate_NotNull CHECK (BirthDate IS NOT NULL) DEFERRABLE INITIALLY DEFERRED) / now run the same Insert and Update shown above, they would run fine, but you would get a constraint violation error when you try to commit with a null Name or BirthDate.\nINSERT INTO customer (customerId, name, birthdate) VALUES (1,null,null); UPDATE customer set name = \u0026#39;FOO Name\u0026#39;, birthdate = \u0026#39;12/12/1978\u0026#39; WHERE customerId = 1; The radical Idea, I\u0026rsquo;m trying to highlight is to be creative in you database design and use the features of the database to be able to enforce data quality constraints or other constraints and still not have to suffer on application development velocity.\n","date":"3 October 2007","externalUrl":null,"permalink":"/post/frameworks_should_not_constrain_design/","section":"Posts","summary":"Recently talking to someone about a persistence framework that they are using, this framework creates a rows in the database table with just Primary Key value and then later on updates the table row with values for other columns. Because of this framework, the tables cannot have any NOT-NULL constraints defined.\n","title":"Frameworks should not constrain your table design","type":"post"},{"content":"On any machine running network related services, like in my case Oracle Listener and Dispatch Services. Don\u0026rsquo;t rebuild/stop and restart the firewall stuff like iptables.\nHad to spend a lot of time, figuring out what was going on. So hard lesson learnt don\u0026rsquo;t mess with iptables when you are running oracle listener/dispatcher\n","date":"13 September 2007","externalUrl":null,"permalink":"/post/lesson_learnt_firewall/","section":"Posts","summary":"On any machine running network related services, like in my case Oracle Listener and Dispatch Services. Don’t rebuild/stop and restart the firewall stuff like iptables.\nHad to spend a lot of time, figuring out what was going on. So hard lesson learnt don’t mess with iptables when you are running oracle listener/dispatcher\n","title":"Lesson learnt changing firewall settings","type":"post"},{"content":"A question I get, mostly related to the evolutionary database design and development. When the pair (team) gets a new feature (story) to work on, the team looks at the existing table/database design and sees if the current design is enough to implement the feature they are working on. If the currency database design does support the feature they are trying to implement, then they do not have to change the database at all, they will move on to implement the feature and change the application code as necessary.\nIf the current database/table design does not allow them to implement the feature that they are trying to complete, then they start looking at how to change the database/table design so that they can implement the feature.\nWithout getting into the discussion of what is the correct design, I wanted to highlight how the design decision is made.\nLets take a example: The feature Joe and Andy (pair) are working on is \u0026ldquo;As a Customer I should be able to have billing and shipping address\u0026rdquo;. Now to implement this feature Joe and Andy see if the Customer has the ability to store any addresses, if we can store addresses, if we have address table etc.\nJoe and Andy find out that they have a Customer table and an Address table, also that Customer table has a AddressID on it, which shows the CustomerAddress. Joe and Andy decide to create two more attributes/columns on the Customer table named BillingAddressID and ShippingAddressID and remove the previous AddressID, copying the data from AddressID to BillingAddressID as well as ShippingAddressID, changing the AddressID makes for better column name.\nSo Joe and Andy implemented the following refactorings.\nIntroduce New Column (ShippingAddressID) Introduce New Column (BillingAddressID) Move Data (AddressID to ShippingAddressID and BillingAddressID) Add Foreign Key Constraint (ShippingAddressID depends on Address table) Add Foreign Key Constraint (BillingAddressID depends on Address table) Drop Column (AddressID)\nNow they go on to implement the rest of the feature in the Application, I find that one pair (Joe and Andy in our example) working from the UI layer down to the database layer touching all the layers of the application makes for much more sense than having one pair work on the database side of things, another on the domain layer and then another on the UI layer. This allows for the team to learn all aspects of the application and also makes everyone productive and not wait on team to finish their part before starting to work on our feature. When you are lacking in certain skills, you can always pair with the Expert in the given area. If Joe and Andy thought they need help writing the data migration script for the above refactorings, then they would pull the data expert on the team and then pair with him. This is how collaborative and evolutionary design happens.\n","date":"28 August 2007","externalUrl":null,"permalink":"/post/when_does_evolutionary_design/","section":"Posts","summary":"A question I get, mostly related to the evolutionary database design and development. When the pair (team) gets a new feature (story) to work on, the team looks at the existing table/database design and sees if the current design is enough to implement the feature they are working on. If the currency database design does support the feature they are trying to implement, then they do not have to change the database at all, they will move on to implement the feature and change the application code as necessary.\n","title":"When does evolutionary design happen?","type":"post"},{"content":"Currently working on a legacy application, thats been in production for a long time now. I wanted to find out what are the Tables and Columns being used by the application. Since we could see that some table columns where not being used. We are using a Object Relational mapping framework on the project, so we decided to write some code that would parse all the mapping files and gives us a list of table names and columns. We used this list to create rows in a table with two columns tablename and columnname. Once the table had this data, we just ran one more SELECT against the metadata of the database and our table which pretty much gave us a list of Table and Columns that we are not using\nThe SQL we used to get the Tables and Columns not used, from Oracles metadata was\nSELECT table_name,column_name FROM user_tab_columns MINUS SELECT usedTableName, usedColumnName FROM usedTableColumns I thought that was a pretty easy way to find out all the tables, columns used by the application and not to have to do manual analysis.\n","date":"19 July 2007","externalUrl":null,"permalink":"/post/parsing_mapping_files_for_usage_information/","section":"Posts","summary":"Currently working on a legacy application, thats been in production for a long time now. I wanted to find out what are the Tables and Columns being used by the application. Since we could see that some table columns where not being used. We are using a Object Relational mapping framework on the project, so we decided to write some code that would parse all the mapping files and gives us a list of table names and columns. We used this list to create rows in a table with two columns tablename and columnname. Once the table had this data, we just ran one more SELECT against the metadata of the database and our table which pretty much gave us a list of Table and Columns that we are not using\n","title":"Parsing mapping files for usage information","type":"post"},{"content":"When you are refactoring large databases, you will have certain tables that have millions of rows, so lets say we are doing the Move Column refactoring, moving the TaxAmount column from Charge table which has millions of rows to TaxCharge table. Create the TaxAmount column in the TaxCharge table. Then have to move the data from the TaxAmount column in the Charge table to the TaxAmount column you created in the TaxCharge table.\nOne way to move the data would be to write a single update statement as shown below.\nUPDATE taxCharge set taxAmount= (SELECT taxAmount FROM charge WHERE charge.chargeid = taxCharge.chargeID); Now this update will run for some time and will be run as one transaction, the Update operation will need a lot of UNDO space (UNDO space in oracle other databases may need other types of space, but the general idea is some space is needed by the database server ), so there is a probability of the update failing if the database server runs out of UNDO space, another side effect of moving large amounts of data using a single update (transaction) is that all the updated rows in the TaxCharge table will be locked till the whole update transaction is done and a commit is issued.\nGetting around this problem is to use a programmatic update, like the PL/SQL code shown below. This code can be implemented in the database procedural code like TSQL or any other language appropriate for the task.\nDECLARE CURSOR allCharges IS SELECT chargeId, taxAmount FROM charge; numberOfRowsToCommit NUMBER :=1000; BEGIN FOR thisCharge IN allCharges LOOP UPDATE taxCharge SET taxamount=thisCharge.taxAmount WHERE chargeid = thisCharge.chargeid; IF allCharges%rowcount MOD numberOfRowsToCommit = 0 THEN COMMIT; END IF; END LOOP; END; / The above PL/SQL code basically loops through all the rows in the Charge table and updates the TaxCharge table with the taxAmount and commits after every 1000 rows (as defined by numberOfRowsToCommit). This approach allows us to work with a smaller UNDO size, releases (not lock) rows once they are committed after every 1000 rows. Off course there are many variations of this technique and many ways to implement this particular update. The point I\u0026rsquo;m trying to get to is that, depending on the situation, I will choose different ways to update the data and maybe not always use UPDATE statements.\n","date":"29 June 2007","externalUrl":null,"permalink":"/post/long_running_data_migrations/","section":"Posts","summary":"When you are refactoring large databases, you will have certain tables that have millions of rows, so lets say we are doing the Move Column refactoring, moving the TaxAmount column from Charge table which has millions of rows to TaxCharge table. Create the TaxAmount column in the TaxCharge table. Then have to move the data from the TaxAmount column in the Charge table to the TaxAmount column you created in the TaxCharge table.\n","title":"Long Running Data Migrations during Database Refactorings","type":"post"},{"content":"","date":"29 June 2007","externalUrl":null,"permalink":"/tags/refactoring/","section":"Tags","summary":"","title":"Refactoring","type":"tags"},{"content":"","date":"8 June 2007","externalUrl":null,"permalink":"/tags/bddd/","section":"Tags","summary":"","title":"BDDD","type":"tags"},{"content":"When you are writing xUnit tests you are in certain ways trying to make sure that the test breaks when the code that is being tested changes the assumptions you made when writing the Test and Production code.\nSimilarly if you are relying on the database to throw a error when you put invalid data, then you should write a test around this assumption, so that when someone changes the database to not behave the way you assumed it to behave, the test you wrote will break and it will force the team to think about the change to the database that is being undertaken. If the change is really required, then the team would fix the test else rollback the change being made.\nHere is a example.\npublic void testShouldNotCreateEmployeeWhenHireDateGreaterThanTerminationDate() { employee.setEmployeeID(employeeId); employee.setHireDate(hireDate); employee.setTerminatedDate(terminatedDate); boolean hadException = false; try { employeeGateway.insert(employee); } catch (SQLException e) { hadException = true; } assertTrue(\u0026#34;Termination date before hiredate should not be allowed by the database\u0026#34;, hadException); } On the Employee table, we have a Check constraint that checks if the TerminationDate is before HireDate. When a employee with hiredata greater than termination date is entered the database will throw a exception. This functionality provided by the database can be assumed by the application. If this database design changes the team will need to change all the affected code, hence its better to put a test around this assumption and fail the test suite when the database design changes.\nThe failed test will force the team to look at why the database design was the way it was designed and what other places in the code need to change if the database design changes. Obviously you can do this validation in the application before you persist the Employee object to the database. In situations where the database is also being accessed by other applications, its better to delegate the data quality constraints to the database, instead of relying on other apps to do the right thing.\nAnother way to do the above test would be as suggested by Sudhindra.\npublic void testShouldNotCreateEmployeeWhenHireDateGreaterThanTerminationDate() { employee.setEmployeeID(employeeId); employee.setHireDate(hireDate); employee.setTerminatedDate(terminatedDate); try { employeeGateway.insert(employee); fail(\u0026#34;Termination date before hiredate should not be allowed by the database\u0026#34;); } catch (SQLException e) { // expected SQLException } } ","date":"8 June 2007","externalUrl":null,"permalink":"/post/enforcing_your_assumptions/","section":"Posts","summary":"When you are writing xUnit tests you are in certain ways trying to make sure that the test breaks when the code that is being tested changes the assumptions you made when writing the Test and Production code.\n","title":"Enforcing your assumptions about database functionality","type":"post"},{"content":"","date":"8 June 2007","externalUrl":null,"permalink":"/tags/talks/","section":"Tags","summary":"","title":"Talks","type":"tags"},{"content":"I will presenting about Database Refactoring: Evolutionary Database Design at XP 2007 here is the Tutorial schedule\n","date":"8 June 2007","externalUrl":null,"permalink":"/post/xp2007_tutorial/","section":"Posts","summary":"I will presenting about Database Refactoring: Evolutionary Database Design at XP 2007 here is the Tutorial schedule\n","title":"XP2007 Tutorial","type":"post"},{"content":"I will be presenting about Evolutionary Database Design and Database Refactoring at ThoughtWorks Master Class Series 2007 at Bangalore on May 19th and Pune on May 26. This will be the first time I will be presenting in India.\nThe Master Class Series is an annual seminar organized by ThoughtWorks India. It focuses on topics which are cutting edge, but the content of the seminars is drawn from real-life experiences on live projects. The presenters are all people with extensive hands-on experience and have delivered successful projects using the concepts they are talking about.\n","date":"8 May 2007","externalUrl":null,"permalink":"/post/thoughtworks_master_class_series_india/","section":"Posts","summary":"I will be presenting about Evolutionary Database Design and Database Refactoring at ThoughtWorks Master Class Series 2007 at Bangalore on May 19th and Pune on May 26. This will be the first time I will be presenting in India.\n","title":"ThoughtWorks Master class series in India","type":"post"},{"content":"After finishing the first Refactoring Databases book. I started on a short ebook project, this book was going to tackle on a very specific technical topic mentioned in the first book. I wanted to write about all the specific scenarios and all the techniques I follow on the various projects.\nThe result of this effort was Recipes for Continuous Database Integration: Evolutionary Database Development, thanks to Martin Fowler for the title. This ebook laid out in a recipes kind of format provides various ways you can integrate you database into you development cycle and make evolutionary database development a fun thing to work with\nSpecial thanks to my current ThoughtWorks project team, Andy Slocum in particular and Scott Ambler for all the help.\n","date":"23 April 2007","externalUrl":null,"permalink":"/post/recipes_for_continuous_database_integration/","section":"Posts","summary":"After finishing the first Refactoring Databases book. I started on a short ebook project, this book was going to tackle on a very specific technical topic mentioned in the first book. I wanted to write about all the specific scenarios and all the techniques I follow on the various projects.\n","title":"My Latest eBook is Out\"","type":"post"},{"content":"Last week I was at SD Best Practices in Moscow, doing a presentation on \u0026ldquo;Refactoring Databases: Evolutionary Database Design\u0026rdquo;. Moscow seems like a interesting place, loads of huge buildings, squares, fountains and roads. Things some how feel rundown, feels like a player trying to regain his former ability or glory.\nOpening Keynote by Jim McCarthy about how teams should operate was interesting, he proposed 11 principals or protocols as he calls them, to be followed by members in a team so that the team becomes more productive, many of these protocols are about avoiding waste and promoting clear communication channels.\nWhile doing my presentation I had live translators which I experienced for the very first time, I also had a hard time trying to understand questions from the audience, since there was no reverse translator who would translate from Russian to English. The audience would try and ask the questions in English which would invariably get confusing.Anyway I think I had a good time meeting all the folks out in Moscow and was pleasantly surprised to find out that ThoughtWorks has a following in Russia.\n","date":"11 April 2007","externalUrl":null,"permalink":"/post/moscow/","section":"Posts","summary":"Last week I was at SD Best Practices in Moscow, doing a presentation on “Refactoring Databases: Evolutionary Database Design”. Moscow seems like a interesting place, loads of huge buildings, squares, fountains and roads. Things some how feel rundown, feels like a player trying to regain his former ability or glory.\n","title":"Moscow","type":"post"},{"content":"I have been working at Thoughtworks for 8 years now, its a fun place to work. Reason I\u0026rsquo;m blogging about this, ThoughtWorks is hiring in the US, UK, Australia, India, China and Canada. So go ahead send your resume to work@thoughtworks.com\n","date":"3 April 2007","externalUrl":null,"permalink":"/post/promoting_thoughtworks/","section":"Posts","summary":"I have been working at Thoughtworks for 8 years now, its a fun place to work. Reason I’m blogging about this, ThoughtWorks is hiring in the US, UK, Australia, India, China and Canada. So go ahead send your resume to work@thoughtworks.com\n","title":"Promoting Thoughtworks","type":"post"},{"content":"Last week I received the good news. The book I co-authored with Scott Ambler won the 2007 Jolt Productivity Award in the Technical Books category. I was dumb enough not to attend the awards ceremony and receive the award, anyway when I started on the book project couple of years back I was afraid if the book would do justice to the Martin Fowler signature series, under which this book appears. The Jolt award award and all the comments I have received from many people in the last year, put me at easy, give me the feeling that finally I can relax and not worry about letting down Martin\u0026rsquo;s signature series.\nI think the challenges of working with the database in an evolutionary manner, are finally getting the attention they deserve. ThoughtWorks won two awards at Jolt night, my book and dbdeploy and both of them related to the database area helping projects deliver in an Evolutionary manner. More about Jolt Awards can be found at Jolt Awards\nOverall it has been a good week, getting the news about the Jolt Productivity Award the appreciation and well wishes from all ThoughtWorkers, friends and family.\n","date":"3 April 2007","externalUrl":null,"permalink":"/post/winning_a_award/","section":"Posts","summary":"Last week I received the good news. The book I co-authored with Scott Ambler won the 2007 Jolt Productivity Award in the Technical Books category. I was dumb enough not to attend the awards ceremony and receive the award, anyway when I started on the book project couple of years back I was afraid if the book would do justice to the Martin Fowler signature series, under which this book appears. The Jolt award award and all the comments I have received from many people in the last year, put me at easy, give me the feeling that finally I can relax and not worry about letting down Martin’s signature series.\n","title":"Winning a award","type":"post"},{"content":"Recently we had peculiar problem. Some of the data in the database was not being created in a proper fashion. Once we found that out we fixed the problem in the application. The customer still had the perception that the code is still broken, because the fixed code was now interacting with the data that was broken (since it was created much earlier by code that was broken). Data has a life of its own (more on this later)\nSo I started thinking (this happens very rarely) about data quality. Does improved data quality give a perception of improved code quality? Does poor data quality drag down the perception of code quality? Anyway the effort invested in code quality can be negated if Data Quality is not thought about!.\nWhen unit tests could be written to flush out design/code quality problems in the application. what could be done to improve data quality, what kind of tests could be written? Would these tests apply to the design of the database or to the quality of the data.\n","date":"15 February 2007","externalUrl":null,"permalink":"/post/data_quality_and_code_quality/","section":"Posts","summary":"Recently we had peculiar problem. Some of the data in the database was not being created in a proper fashion. Once we found that out we fixed the problem in the application. The customer still had the perception that the code is still broken, because the fixed code was now interacting with the data that was broken (since it was created much earlier by code that was broken). Data has a life of its own (more on this later)\n","title":"Data Quality and Code Quality","type":"post"},{"content":"The following SELECT statement in code\nstmt = DB.prepare(\u0026#34;select id,name,state,zip \u0026#34; + \u0026#34;from customer \u0026#34; + \u0026#34;where \u0026#34; + \u0026#34;phone = ? \u0026#34; + \u0026#34;and active = ?\u0026#34;); stmt.setString(1, customerPhone); stmt.setBoolean(2, isActive); stmt.execute(); where customerPhone and isActive are values you would pass in to the SELECT before its executed. Everything is fine when one day the value passed for customerPhone is NULL. For a database (Oracle is what I know most) a NULL will never be equal to NULL , the SELECT will not return rows where the customer.phone is NULL, leading to wrong results. The SELECT will have to be changed to\nstmt = DB.prepare(\u0026#34;select id,name,state,zip \u0026#34; + \u0026#34;from customer \u0026#34; + \u0026#34;where \u0026#34; + \u0026#34;(phone IS NULL or phone = ?) \u0026#34; + \u0026#34;and \u0026#34; + \u0026#34;active = ?\u0026#34;); stmt.setString(1, customerPhone); stmt.setBoolean(2, isActive); stmt.execute(); We could dynamically write the SELECT so that we don\u0026rsquo;t have to do the OR in the where clause, which could be expensive.\nboolean hasPhone = false; StringBuffer sqlQuery = new StringBuffer(); sqlQuery.append(\u0026#34;select id,name,state,zip \u0026#34; + \u0026#34;from customer \u0026#34; + \u0026#34;where 1=1 \u0026#34;); if (phoneNumber != null) { sqlQuery.append(\u0026#34;and phone=? \u0026#34;); hasPhone = true; } else { sqlQuery.append(\u0026#34;and phone IS NULL \u0026#34;); } sqlQuery.append(\u0026#34;and isActive=?\u0026#34;); stmt = DB.prepare(sqlQuery.toString()); if (hasPhone) { stmt.setString(1, phoneNumber); stmt.setBoolean(2, isActive); } else { stmt.setBoolean(1, isActive); } stmt.execute(); ","date":"15 February 2007","externalUrl":null,"permalink":"/post/nulls_need_special_love/","section":"Posts","summary":"The following SELECT statement in code\nstmt = DB.prepare(\"select id,name,state,zip \" + \"from customer \" + \"where \" + \"phone = ? \" + \"and active = ?\"); stmt.setString(1, customerPhone); stmt.setBoolean(2, isActive); stmt.execute(); where customerPhone and isActive are values you would pass in to the SELECT before its executed. Everything is fine when one day the value passed for customerPhone is NULL. For a database (Oracle is what I know most) a NULL will never be equal to NULL , the SELECT will not return rows where the customer.phone is NULL, leading to wrong results. The SELECT will have to be changed to\n","title":"Nulls need special love","type":"post"},{"content":"While working on a Legacy Application with Legacy Database design as part of fixing a bug, I thought this bug would not have ever happened if a particular column was defined as Non Nullable since this particular column was the identifier to the parent table.\nWe had a Customer table and the all the names a customer could have like LegalName, LongName, ShortName etc are stored in the CustomerName table. CustomerName cannot exist without Customer hence its logical that the CustomerName.CustomerID column cannot be nullable and should also have a Foreign Key constraint enforcing the relationship. Implementing just the Foreign Key constraint is not enough since the application could potentially be inserting null values in the CustomerName.CustomerID creating orphan records.\nSo we had to implement Add Foreign Key Constraint and later on implement Make column non nullable\nThe changes we had to make to the legacy application and legacy database involved\nFind and fix the place where the application creates or updates CustomerName so that it does not create a CustomerName with Null or Invalid CustomerID values. I did it by using the Applications Customer and CustomerName domain objects and the business logic that created these objects and wrote a unit test that interacts with the database. So that the object was persisted and I could see the resulting rows in the database. So once I do Step 2 and 3 below, this unit test will fail. The failing test will point me to the problem in the application code.\nWrite a SQL Script to fix the data in CustomerName, such that we delete all CustomerName rows where the CustomerName.CustomerID is null and also where CustomerName.CustomerID does not exist in Customer.CustomerID. The reason I did this was all the CustomerName rows where CustomerID was null where orphan rows that could not be reached too by the application anyway and hence could be removed.\nCreate a Foreignkey Constraint between Customer and CustomerName on the CustomerID column.\nNow that we have no nulls in the CustomerName.CustomerID column and all the refrentially improper data has been removed. We run our unit test from Step 1. THe failure will point us to the places in the application code where we need to fix the code.\nNow our application code has been changed and we improved it, we have improved the the database design, in the process we also have a unit test that will break when anyone else makes any application code changes that break the Referentially Integrity and/or the Not Null rule.\nOverall a satisfying day at work I think.\n","date":"15 December 2006","externalUrl":null,"permalink":"/post/implementing_make_column_non_nullable/","section":"Posts","summary":"While working on a Legacy Application with Legacy Database design as part of fixing a bug, I thought this bug would not have ever happened if a particular column was defined as Non Nullable since this particular column was the identifier to the parent table.\n","title":"Implementing Make Column Non Nullable","type":"post"},{"content":"Some time ago I wrote about what it means to do database testing.. more I think about this and having had some strange situations recently I want to add more to the list of things we should be testing.\nPersistence Layer We should persist the objects to the database using the applications persistence layer and retrieve the objects using the same mechanism and test that we get the same object back. If we have a lot of business logic in out persistence layer we may also want to retrieve the object using Direct SQL and test that the correct values got persisted.\nDatabase Structure The application assumes a certain database structure and objects to be around for it to function properly and we want to make sure that all the database objects we depend on are there and are defined as we expect them to be, for example if we need a Auto Incrementing column on a table, then we should test that the database has such a column using the database metadata. In other situations you are expecting the Customer.Name column to be 128 characters wide and the database does not have the Customer.Name column at 128 characters, these kinds of errors can be found out using the database metadata.\nDatabase Code Stored procedures need to be tested, using unit testing frameworks like utPlsql or whatever is equivalent to your situation\nDatabase Objects Views are database objects that the application will usually depend on and the views have business logic in them. We need to test these views and you can use your application code to test them or you can use the database unit testing frame work. One I have found is easy to do is to test them using JUnit (or whatever that works for). Setup the correct data in your tables and use the view to retrieve the data and test of the view returns the correct rows for the data in the database. These kinds of test could get really complicated if you have no control over what data is present in the database. DBUnit can help here by providing re-loadable data.\n","date":"9 November 2006","externalUrl":null,"permalink":"/post/database_testing_revisited/","section":"Posts","summary":"Some time ago I wrote about what it means to do database testing.. more I think about this and having had some strange situations recently I want to add more to the list of things we should be testing.\n","title":"Database Testing revisited","type":"post"},{"content":"You learn a lot from kids, and this lesson I will never forget.\nSo I\u0026rsquo;m at Lyon (in France) airport and there are a couple of kids (4-5 year olds) playing in the airport play area which had slides, call these two kids TA and TB my daughter call her AA joins these kids and they all start playing with the slide.\nAll is going well when a family with two kids, mom, dad and grandma join in, mom and dad are staying away and grandma takes the kids lets call them FA and FB to play on the slides. Grandma makes the previous kids (AA, AB, AA) stop playing and clears the slide for her grand kids (EA and EB) to play the slide. Grandkids start playing for a while and TA and TB are just waiting around not knowing what to do. AA by now has moved on to a different toy in the airport.\nSometime later TB opens his bottle of water to get a drink and pours it all on FA. FA is now totally wet and grandma is furious at TA and TB. Now she has to take FA away to get a new shirt and get FA dry. TA and TB by now start playing with the slide, happy that they have control of the slide.\nThe incident may sound run of the mill, if I don\u0026rsquo;t revel the nationalities of the kids involved TA and TB are from a Arabic country. FA and FB are from a European country. This little incident has taught me that when you push people around they are going to react and the reaction may sometimes be violent. It has also taught me to stay out and let the kids play and resolve their issues themselves.\nThese lessons are so relevant in these times. Wish our leaders learnt something from kids.\n","date":"26 September 2006","externalUrl":null,"permalink":"/post/kids_teaching_international_relations/","section":"Posts","summary":"You learn a lot from kids, and this lesson I will never forget.\nSo I’m at Lyon (in France) airport and there are a couple of kids (4-5 year olds) playing in the airport play area which had slides, call these two kids TA and TB my daughter call her AA joins these kids and they all start playing with the slide.\n","title":"Kids teaching International Relations\"","type":"post"},{"content":"","date":"28 August 2006","externalUrl":null,"permalink":"/tags/cd/","section":"Tags","summary":"","title":"CD","type":"tags"},{"content":"","date":"28 August 2006","externalUrl":null,"permalink":"/tags/ci/","section":"Tags","summary":"","title":"CI","type":"tags"},{"content":"I have taken up the hobby of searching the opensource landscape for tools that help me do Agile database development. I\u0026rsquo;m going to write about all the Tools that I come across that help me, my preference is opensource software but not limited to it. I will try to provide some sound examples and share my experiences with all that tools that I come across and share the example code I used.\nMigrate DB\nThis is a simple xml based solution that applies all the changes defined by you in the XML file, you provide a pre-condition or test condition for the execution of the sql and the SQL is executed when the condition is met. The tool provides command line support and also gives ANT integration.\nlets take this example build file, I\u0026rsquo;m using this build.properties file to define my connection properties\nI created a ANT task named as dbrelease, used the ANT task to create a ANT target named as migratedb. The ANT target uses a XML file to write a pre condition SQL and the actual SQL to migrate the DB. example XML can be found here.\nFull example can be found here ( is a zip file and contains the migratedb.jar all the xml files and is setup to work with oracle)\n","date":"28 August 2006","externalUrl":null,"permalink":"/post/database_migration_utility/","section":"Posts","summary":"I have taken up the hobby of searching the opensource landscape for tools that help me do Agile database development. I’m going to write about all the Tools that I come across that help me, my preference is opensource software but not limited to it. I will try to provide some sound examples and share my experiences with all that tools that I come across and share the example code I used.\n","title":"Database Migration Utility","type":"post"},{"content":"I had been on vacation for sometime. Started back work and getting connected with work and life again.\n","date":"26 July 2006","externalUrl":null,"permalink":"/post/inactivity/","section":"Posts","summary":"I had been on vacation for sometime. Started back work and getting connected with work and life again.\n","title":"Inactivity\"","type":"post"},{"content":"What does it mean to test your Database? usually when someone mentions database testing, what is that they want to test. The application code that interacts with the database, or the sql code the resides in the database like stored procedures and triggers etc. I see all these aspects to database testing as important.\nTesting the applications persistence mechanism We should test that the application persists what its supposed to save and retrieve the data using SQL and see if the database contains the same information that is being saved, this kind of testing makes sense when the application has complex persistence layer. This type of testing can be achieved using unit tests, functional tests etc.\nTesting the Database Code The database code like Views, Stored procedures etc, contains business logic that you want tested, since a lot of your application code will depend on it since database code is the API that the database provides for the application to use. You can use your application unit tests(or other test frameworks that your are using) to test the database API. You can achieve this using unit tests that actually hit the database, these tests will be slow, but having some tests is better than nothing. You can also use unit testing frameworks for the database code like Ounit, PL/SQL Unit etc (more about how to use these in later posts).\n","date":"23 May 2006","externalUrl":null,"permalink":"/post/database_testing/","section":"Posts","summary":"What does it mean to test your Database? usually when someone mentions database testing, what is that they want to test. The application code that interacts with the database, or the sql code the resides in the database like stored procedures and triggers etc. I see all these aspects to database testing as important.\n","title":"Database Testing","type":"post"},{"content":"Many IT organizations I have seen have groups of specialists, typical are UNIX Group, DBA group etc.\nWhen a project starts the developers on the project have to meet with all the Groups (I have more experience with the DBA group, so I write with the DBA group in mind) that they need to interact with and explain to the groups their design, the projects operational needs and other requirements, later when development starts they have to email these groups about all the setup that needs to be done and also the day to day changes that are needed. This way of working slows down the productivity of the team and the organization.\nWhat I have found works best is when one of the member (this member could/should be rotated) of the DBA group is placed on the development team. When that happens, the team has a expert on hand they can ask him all kinds of questions, run different scenarios, design options by him, the member of the DBA group will also understand in depth what is happening on the team, what are their needs and how to tackle them, it also makes use of his link back to the group (since he is part of the DBA group, the DBA group is more likely to not ask any questions of the requirements/issues raised by him), since he knows all the members of his group, he can communicate better with the DBA group and make the tasks easier for the development team and also make it easier on the DBA group to execute the tasks expected of them. DBA\u0026rsquo;s will also learn a lot from this, since now they are not sitting behind the email wall and guessing about the needs of the teams.\n","date":"23 May 2006","externalUrl":null,"permalink":"/post/move_dba_to_project_team/","section":"Posts","summary":"Many IT organizations I have seen have groups of specialists, typical are UNIX Group, DBA group etc.\nWhen a project starts the developers on the project have to meet with all the Groups (I have more experience with the DBA group, so I write with the DBA group in mind) that they need to interact with and explain to the groups their design, the projects operational needs and other requirements, later when development starts they have to email these groups about all the setup that needs to be done and also the day to day changes that are needed. This way of working slows down the productivity of the team and the organization.\n","title":"Move your DBAs to the Project team locations\"","type":"post"},{"content":"","date":"23 May 2006","externalUrl":null,"permalink":"/tags/persistence/","section":"Tags","summary":"","title":"Persistence","type":"tags"},{"content":"In development mode you don\u0026rsquo;t want to worry about which table goes into what Tablespace in production as it complicates development environments. The production DBA\u0026rsquo;s want to have their input and control over deciding what table goes into what Tablespace. To allow for this I used a mapping scheme as shown below.\nLets assume we have 3 tables in our system Customer, CustomerOrder, OrderStatus. Where we are expecting Customer table to have large numbers of rows and CustomerOrder to have significanly large number of rows while OrderStatus would have few rows and not change as much. In development environments all these tables and their indexes will be put under the same tablespace. In production like environments we want to put them into seperate tablespaces.\nSo we had 3 types of tablespaces Small, Medium and Large to hold tables of Small, Medium and Large Sizes. We also had 3 tablespaces SmallIndex, MediumIndex and LargeIndex to hold the tables Indexes. The production DBA provided a MAP of tables and tablespace names that the table needed to reside in, when we created the tables and their indexes in production like environements we created all the objects in the same tablespace and later on a pl/sql script moved all the tables into their respective tablespaces using the MAP provided by the production DBA\u0026rsquo;s.\nNow lets look at the code. The tablespace MAP looks as below, where we map a table to the tablespace where the table needs to reside. By naming convention we can also move all the indexes of the table. For example the indexes of OrderStatus table will be created in tablespace named as SmallIndex\nTable_Name Tablespace_name ORDERSTATUS Small CUSTOMER Medium CUSTOMERORDER Large The pl/sql code to check if all the tables are mapped, if all tables are not mapped then we get a exception.\nDECLARE non_mapped_tables number; BEGIN SELECT count(*) INTO non_mapped_tables FROM user_tables WHERE table_name NOT IN (SELECT table_name FROM tablespace_map) AND table_name != \u0026#39;TABLESPACE_MAP\u0026#39;; IF non_mapped_tables != 0 THEN THROW_ERROR(non_mapped_tables,\u0026#39; All table/s have not been mapped to TABLESPACES\u0026#39;); END IF; END; / Now we are ready to move the tables and their indexes to their respective tablespaces.\nDECLARE CURSOR tabs IS SELECT \u0026#39;ALTER TABLE \u0026#39; || segment_name || \u0026#39; MOVE tablespace \u0026#39; stmt,m.tablespace_name FROM user_segments u,tablespace_map m WHERE u.segment_type = \u0026#39;TABLE\u0026#39; AND u.segment_name = m.table_name ORDER BY u.segment_name; CURSOR idx IS SELECT \u0026#39;ALTER INDEX \u0026#39;|| segment_name || \u0026#39; REBUILD tablespace \u0026#39; stmt,m.tablespace_name FROM user_segments u,user_indexes i,tablespace_map m WHERE u.segment_type = \u0026#39;INDEX\u0026#39; AND u.segment_name = i.index_name AND i.table_name=m.table_name ORDER BY u.segment_name; BEGIN --Move all the tables FOR tabsrec IN tabs LOOP EXECUTE IMMEDIATE tabsrec.stmt||tabsrec.tablespace_name; END LOOP; --Move all the indexes FOR idxrec IN idx LOOP EXECUTE IMMEDIATE idxrec.stmt||idxrec.tablespace_name; END LOOP; END; / Deployments done this way will sheild the development team worrying about production database specifics and allows the production DBA\u0026rsquo;s to fine tune the production deployment.\n","date":"1 May 2006","externalUrl":null,"permalink":"/post/automated_tablespace_deployment/","section":"Posts","summary":"In development mode you don’t want to worry about which table goes into what Tablespace in production as it complicates development environments. The production DBA’s want to have their input and control over deciding what table goes into what Tablespace. To allow for this I used a mapping scheme as shown below.\n","title":"Automated Tablespace deployment","type":"post"},{"content":"","date":"1 May 2006","externalUrl":null,"permalink":"/tags/automation/","section":"Tags","summary":"","title":"Automation","type":"tags"},{"content":"","date":"26 March 2006","externalUrl":null,"permalink":"/tags/book/","section":"Tags","summary":"","title":"Book","type":"tags"},{"content":"This last week I received my copies of the book, for the first time I could touch what I had worked on for almost a year and half now, I could see it flip the pages. When I opened the UPS box and saw the book for the first time it felt great but in a minute or two I was not feeling anything, after some tea and some walking around I took the book in my hands and started flipping the pages this is when the real feeling started to sink in, but still I was not sure of the situation, was not really sure if it was real, after about 2-3 hours when I saw the book and held the book and started reading, I was started feeling the joy the happiness and all the goodness. I think when I see it being read by others then I will feel a lot different, the other day I saw Nick take out the book out of his backpack I was thrilled to see it being read.\n","date":"26 March 2006","externalUrl":null,"permalink":"/post/database_refactoring_book/","section":"Posts","summary":"This last week I received my copies of the book, for the first time I could touch what I had worked on for almost a year and half now, I could see it flip the pages. When I opened the UPS box and saw the book for the first time it felt great but in a minute or two I was not feeling anything, after some tea and some walking around I took the book in my hands and started flipping the pages this is when the real feeling started to sink in, but still I was not sure of the situation, was not really sure if it was real, after about 2-3 hours when I saw the book and held the book and started reading, I was started feeling the joy the happiness and all the goodness. I think when I see it being read by others then I will feel a lot different, the other day I saw Nick take out the book out of his backpack I was thrilled to see it being read.\n","title":"The day I received my Boo","type":"post"},{"content":"Database designs I have seen tend to not constrain the data in the database. For example make the Item.ManufacturerID non-nullable and make it a foreign key to the Manufacturer table. Similarly Manufacturer.Name and Item.Rate as non-nullable columns. In any greenfield application (existing production application is a topic for another post). When you design table(s) lets say you have Item and Manufacturer table as shown below\nCREATE TABLE Item (ItemID NUMBER NOT NULL, ManufacturerID NUMBER, Name VARCHAR2(128), Rate NUMBER, CONSTRAINT PK_ITEM PRIMARY KEY (ItemID) ); CREATE TABLE Manufacturer (ManufacturerID NUMBER NOT NULL, Name VARCAHR2(128), CONSTRAINT PK_MANUFACTURER PRIMARY KEY (ManufacturerID) ); For now let\u0026rsquo;s talk about NOT NULL constraint. Many say they don\u0026rsquo;t make columns non-nullable, because they don\u0026rsquo;t know the requirements at design time, or they don\u0026rsquo;t want their tests to create elaborate sets of data, or that the application enforces the constraint then why enforce the constraint on the database?\nWhen this design goes into production, bad data gets into the database over time. Since databases are used as integration points, users or other applications put data in to the database without using the application, you don\u0026rsquo;t have data entry screens for many setup kind of tables so data has to be put into the database without using the application. When this happens your data in no longer protected by the constraints on the application side and you are forced to counter the effects of bad data with more application changes like making joins with child tables as outer joins.\nSELECT item.Name,Manufacturer.Name FROM Item, Manufacturer WHERE Item.ManufacturerID = Manufacturer.ManufacturerID(+) Having null checks in the application code like\nif (item.getManufacturer() != null) { return item.getManufacturer().getName(); } You also have the very difficult task of fixing data later on. Like what Item should have which ManufacturerID\nSo why not make the columns non-nullable to begin with. If you don\u0026rsquo;t have the requirements when designing the tables, then you should make them non-nullable when the requirement becomes clear later on or when you start enforcing constraints on the application side. Unit tests can be changed or made to work with not-null constraints on the database.\nMaking good decisions about Data Quality in your applications database design will better serve you in the long term maintainability of the application.\n","date":"8 February 2006","externalUrl":null,"permalink":"/post/to_null_or_not/","section":"Posts","summary":"Database designs I have seen tend to not constrain the data in the database. For example make the Item.ManufacturerID non-nullable and make it a foreign key to the Manufacturer table. Similarly Manufacturer.Name and Item.Rate as non-nullable columns. In any greenfield application (existing production application is a topic for another post). When you design table(s) lets say you have Item and Manufacturer table as shown below\n","title":"To allow NULLs or NOT","type":"post"},{"content":"Why is that many a time I say something and the person hears something else.\nI have been wondering is it me or is it just the way people listen or interpret my words. How can I communicate better, making it easier for myself and cause less of an hassle for myself and all the ones around me.\nWhen I get to the bottom of this, it will be one heck of an achievement.\n","date":"3 February 2006","externalUrl":null,"permalink":"/post/communication/","section":"Posts","summary":"Why is that many a time I say something and the person hears something else.\nI have been wondering is it me or is it just the way people listen or interpret my words. How can I communicate better, making it easier for myself and cause less of an hassle for myself and all the ones around me.\n","title":"Communication","type":"post"},{"content":"","date":"3 February 2006","externalUrl":null,"permalink":"/tags/learning/","section":"Tags","summary":"","title":"Learning","type":"tags"},{"content":"Many a times Refactoring is talked about in the context of code, recently I finished working with Scott Ambler on Database Refactoring\nLately I have been working on changing data in an production database, and have been wondering how do I define it, Data Refactoring? what are the patterns of Data Refactoring. First let me talk about what I mean by Data Refactoring.\nWhen a given application goes into production, and starts life as a live application we find bugs with the application, these bugs create a weird data in the database, also with the way people change data through the app and some times through the database (yikes) and these data changes do lead to bad data. How do you go about fixing these data problems, are there patterns to these fixes.\nWell as I work through this I think I will have some thing to report on or maybe its such a situational problem that maybe there are no patterns to this.\n","date":"19 January 2006","externalUrl":null,"permalink":"/post/refactoring_databases/","section":"Posts","summary":"Many a times Refactoring is talked about in the context of code, recently I finished working with Scott Ambler on Database Refactoring\nLately I have been working on changing data in an production database, and have been wondering how do I define it, Data Refactoring? what are the patterns of Data Refactoring. First let me talk about what I mean by Data Refactoring.\n","title":"Refactoring Data","type":"post"},{"content":"Pramod Sadalage is Director at ThoughtWorks leading the Modern Data Architecture service for North America, he enjoys the rare role of bridging the divide between database professionals and application developers. He is usually sent in to clients with particularly challenging data needs, which require new technologies and techniques. In the early 00\u0026rsquo;s he developed techniques to allow relational databases to be designed in an evolutionary manner based on version-controlled schema migrations.\nHe is co-author of Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures, co-author author for Building Evolutionary Architectures: Automated Software Governance, co-author of Refactoring Databases: Evolutionary Database Design, co-author of NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence, author of Recipes for Continuous Database Integration and continues to speak and write about the insights he and his clients learn.\n","externalUrl":null,"permalink":"/about/","section":"Architecture and Data Blog","summary":"Pramod Sadalage is Director at ThoughtWorks leading the Modern Data Architecture service for North America, he enjoys the rare role of bridging the divide between database professionals and application developers. He is usually sent in to clients with particularly challenging data needs, which require new technologies and techniques. In the early 00’s he developed techniques to allow relational databases to be designed in an evolutionary manner based on version-controlled schema migrations.\n","title":"","type":"page"},{"content":" Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures # by Neal Ford, Mark Richards, Pramod Sadalage \u0026amp; Zamak Dehghani\nAll software architecture involves trade offs. But traditional analysis tools don’t work well for today’s distributed systems.\nThis book provides techniques to help you discover and weigh the trade-offs as you confront the issues you face as an architect. It investigates why architecture is so difficult and provides proven mechanisms to address these complex problems and make them understandable. Co-authors Neal Ford, Mark Richards, Pramod Sadalage, and Zhamak Dehghani examine everything from how to determine service granularity, manage workflows and orchestration, manage and decouple contracts, and manage distributed transactions to how to optimize operational characteristics, such as scalability, elasticity, and performance.\nThis book is not just for software architects — data architects, DBAs, product managers and others will glean valuable insights into some of the complex issues architects face every day.\nArchitecture is full of hard parts; by tracing the common reasons and applying lessons more universally, we can make it softer.\nBuy on Amazon\nSoftware Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures\nBuilding Evolutionary Architectures: Automated Software Governance (2nd Edition) # by Neal Ford, Rebecca Parsons, Patrick Kua, Pramod Sadalage\nFor a variety of reasons, parts of software systems defy change, becoming more brittle and intractable over time. However, the world we inhabit has exactly the opposite characteristic. Business constantly changes, but so does the software development ecosystem. New tools, techniques, approaches, and frameworks constantly impact that equilibrium in unanticipatable ways. While this creates a headache for brittle systems, it also provides the ultimate solution. Over the last few years, incremental developments in core engineering practices for software development created the foundations for us to rethink how architecture changes over time, along with ways to protect important architectural characteristics as it evolves. This book ties those parts together with a new way to think about architecture and time.\nAlong the way, we also answer the questions How is long term planning possible when everything changes all the time? and Once I\u0026rsquo;ve built an architecture, how can I prevent it from gradually degrading over time?.\nThis book is about evolutionary architectures, building systems that allow architects and developers to make sweeping changes to the most important parts of their systems with confidence. It covers practices that allow developers to build continual architectures, which evolve cleanly without the need for a crystal ball.\nI contributed to this book, discussing about database architectures and how they can be evolved.\nBuy on Amazon\nBuilding Evolutionary Architectures: Support Constant Change\nNoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence # by Pramod J. Sadalage and Martin Fowler\nThe need to handle increasingly larger data volumes is one factor driving the adoption of a new class of nonrelational “NoSQL” databases. Advocates of NoSQL databases claim they can be used to build systems that are more performant, scale better, and are easier to program.\nNoSQL Distilled is a concise but thorough introduction to this rapidly emerging technology. Pramod J. Sadalage and Martin Fowler explain how NoSQL databases work and the ways that they may be a superior alternative to a traditional RDBMS. The authors provide a fast-paced guide to the concepts you need to know in order to evaluate whether NoSQL databases are right for your needs and, if so, which technologies you should explore further.\nThe first part of the book concentrates on core concepts, including schemaless data models, aggregates, new distribution models, the CAP theorem, and map-reduce. In the second part, the authors explore architectural and design issues associated with implementing NoSQL. They also present realistic use cases that demonstrate NoSQL databases at work and feature representative examples using Riak, MongoDB, Cassandra, and Neo4j.\nIn addition, by drawing on Pramod Sadalage’s pioneering work, NoSQL Distilled shows how to implement evolutionary design with schema migration: an essential technique for applying NoSQL databases. The book concludes by describing how NoSQL is ushering in a new age of Polyglot Persistence, where multiple data-storage worlds coexist, and architects can choose the technology best optimized for each type of data access.\nBuy on Amazon\nNoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence\nRefactoring Databases: Evolutionary Database Design # by Scott Ambler and Pramod J. Sadalage\nA comprehensive guide published by Addison-Wesley as part of the Martin Fowler signature series. This reference book helps you overcome the practical obstacles to refactoring real-world databases by covering every fundamental concept underlying database refactoring. Using start-to-finish examples, the authors walk you through refactoring simple standalone database applications as well as sophisticated multi-application scenarios. You’ll master every task involved in refactoring database schemas, and discover best practices for deploying refactorings in even the most complex production environments.\nThe second half of this book systematically covers five major categories of database refactorings. You’ll learn how to use refactoring to enhance database structure, data quality, and referential integrity; and how to refactor both architectures and methods. This book provides an extensive set of examples built with Oracle and Java and easily adaptable for other languages, such as C#, C++, or VB.NET, and other databases, such as DB2, SQL Server, MySQL, and Sybase. Using this book’s techniques and examples, you can reduce waste, rework, risk, and cost—and build database systems capable of evolving smoothly, far into the future.\nBuy on Amazon\nRefactoring Databases: Evolutionary Database Design\nRecipes for Continuous Database Integration # by Pramod J. Sadalage\nThe past few years have seen the rise of agile or evolutionary methods in software development. These methods embrace change in requirements even late in the project. The ability to change software is because of certain practices that are followed within teams, such as Test Driven Development, Pair Programming, and Continuous Integration. Continuous Integration provides a way for software teams to integrate their work more than once a day, and promotes confidence in the software that is being developed by the team. It is thought that this practice is difficult to apply when continuously integrating the database with application code; hence, Evolutionary Database Development is considered a mismatch with agile methods. This is not necessarily true.\nContinuous Integration changed the way software is written. Why not extend and make the database part of the same Continuous Integration cycle so that you can see integrated results of your application as well as your database? Delivered in PDF format for quick and easy access, Recipes for Continuous Database Integration shows how the database can be brought under the preview of Continuous Integration, allowing all teams to integrate not only their application code, but also their database.\nBuy on Amazon\nRecipes for Continuous Database Integration\n","externalUrl":null,"permalink":"/books/","section":"Architecture and Data Blog","summary":"Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures # by Neal Ford, Mark Richards, Pramod Sadalage \u0026 Zamak Dehghani\n","title":"","type":"page"},{"content":" Books I have read recently and enjoyed. # Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith Building Evolutionary Architectures: Support Constant Change Range: Why Generalists Triumph in a Specialized World Lean Enterprise: How High Performance Organizations Innovate at Scale Building Microservices: Designing Fine-Grained Systems Lean Enterprise: How High Performance Organizations Innovate at Scale Thinking, Fast and Slow The Organized Mind: Thinking Straight in the Age of Information Overload Graph Databases Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing (Agile Software Development Series) Jugaad Innovation: Think Frugal, Be Flexible, Generate Breakthrough Growth The Lean Startup: How Today\u0026rsquo;s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses Switch: How to Change Things When Change Is Hard The Power of Now A New Earth SQL Anti patterns The 4-Hour Workweek Loving Our Kids On Purpose Tuesdays with Morrie The Secret Presentation Zen Programming Ruby 1.9 Release It! The Toyota Way Agile Data Warehousing Outliers: The Story of Success India After Gandhi Why Does Software Cost So Much? It Happened in India Oracle Tuning: The Definitive Reference Ship it! Service-Oriented Architecture My Job Went to India Let My People Go Surfing Practices of an Agile Developer Rails Recipes Programming Ruby The Art of SQL Execution: The Discipline of Getting Things Done The Monk Who Sold His Ferrari The Greatness Guide Who Will Cry When You Die? Zen and the Art of Motorcycle Maintenance Freakonomics Head First Design Patterns (Head First) Agile Database Techniques Hibernate in Action (In Action series) The World Is Flat Guns, Germs, and Steel ","externalUrl":null,"permalink":"/reading/","section":"Architecture and Data Blog","summary":"Books I have read recently and enjoyed. # Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith Building Evolutionary Architectures: Support Constant Change Range: Why Generalists Triumph in a Specialized World Lean Enterprise: How High Performance Organizations Innovate at Scale Building Microservices: Designing Fine-Grained Systems Lean Enterprise: How High Performance Organizations Innovate at Scale Thinking, Fast and Slow The Organized Mind: Thinking Straight in the Age of Information Overload Graph Databases Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing (Agile Software Development Series) Jugaad Innovation: Think Frugal, Be Flexible, Generate Breakthrough Growth The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses Switch: How to Change Things When Change Is Hard The Power of Now A New Earth SQL Anti patterns The 4-Hour Workweek Loving Our Kids On Purpose Tuesdays with Morrie The Secret Presentation Zen Programming Ruby 1.9 Release It! The Toyota Way Agile Data Warehousing Outliers: The Story of Success India After Gandhi Why Does Software Cost So Much? It Happened in India Oracle Tuning: The Definitive Reference Ship it! Service-Oriented Architecture My Job Went to India Let My People Go Surfing Practices of an Agile Developer Rails Recipes Programming Ruby The Art of SQL Execution: The Discipline of Getting Things Done The Monk Who Sold His Ferrari The Greatness Guide Who Will Cry When You Die? Zen and the Art of Motorcycle Maintenance Freakonomics Head First Design Patterns (Head First) Agile Database Techniques Hibernate in Action (In Action series) The World Is Flat Guns, Germs, and Steel ","title":"","type":"page"},{"content":"","externalUrl":null,"permalink":"/search/placeholder/","section":"Searches","summary":"","title":"","type":"search"},{"content":"Over the many years I have given many talks at conferences, here are some selected talks available freely.\nDevOps Practices for the Database Team # NoSQL Distilled - Polyglot Persistence Øredev 2014 # Enabling Continous Delivery Practices in Databases - Øredev 2014 # NoSQL Distilled - GOTO GeekNight - Hamburg # Practices for Agile Database Development - Agile India 2012 # Links to video # InfoQ: Evolving Database Design and Architecture: Patterns and Practices # InfoQ: Refactoring Databases: Evolutionary Database Design # ","externalUrl":null,"permalink":"/talks/","section":"Architecture and Data Blog","summary":"Over the many years I have given many talks at conferences, here are some selected talks available freely.\nDevOps Practices for the Database Team # NoSQL Distilled - Polyglot Persistence Øredev 2014 # Enabling Continous Delivery Practices in Databases - Øredev 2014 # NoSQL Distilled - GOTO GeekNight - Hamburg # Practices for Agile Database Development - Agile India 2012 # Links to video # InfoQ: Evolving Database Design and Architecture: Patterns and Practices # InfoQ: Refactoring Databases: Evolutionary Database Design # ","title":"","type":"page"},{"content":"","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","externalUrl":null,"permalink":"/archive/","section":"Architecture and Data Blog","summary":"","title":"Posts Archive","type":"archive"},{"content":"","externalUrl":null,"permalink":"/search/","section":"Searches","summary":"","title":"Searches","type":"search"}]