Acquiring JAR files to access Hive server from SQurirrel SQL Client, using Maven's dependency plugin

Hive support JDBC interface, so we can utilize JDBC based SQL client programs such as SQurirrel SQL Client, Eclipse DTD(Data Tools Platform), DBViewer(Eclipse plugin), or even Oracle SQL Developer, to access and execute statements on Hive.

The following wiki page explains that step by step in case of SQurirrel SQL Client

Hive Wiki/HiveServer2 Clients/Integration with SQuirrel SQL Client

Unlike Oracle, MySQL or other most of databases, Hive JDBC driver is not distributed as a self-contained archive, which means lots of other JAR files are necessary to use it.
Above page doesn't explain exactly what files are necessary. And more, explained way to get those JAR files seems not convenient.

The following article kindly explains exact file set in case of Hive 0.10 but doesn't show how to get those files.

Connect to HiveServer2 with a kerberized JDBC client (Squirrel)

Using Maven's dependency plugin, we can download those necessary JAR files in convenient way.

In case of Hive 0.13,x, following JAR files are necessary to use JDBC connectivity.

• commons-configuration-1.10.jar (Apache Commons Configuration)
• commons-logging-1.2.jar (Apache Commons Logging)
• guava-17.0.jar (Google Guava)
• hadoop-auth-2.4.1.jar
• hadoop-common-2.4.1.jar
• hadoop-hdfs-2.4.1.jar
• hadoop-mapreduce-client-core-2.4.1.jar
• hive-common-0.13.1.jar
• hive-exec-0.13.1.jar
• hive-jdbc-0.13.1.jar
• hive-metastore-0.13.1.jar
• hive-service-0.13.1.jar
• hive-shims-0.13.1.jar
• libfb303-0.9.1.jar
• libthrift-0.9.1.jar
• httpcore-4.3.3.jar (Apache HttpComponents Core)
• httpclient-4.3.6.jar (Apache HttpComponents Client)
• log4j-1.2.17.jar (Apache Log4j)
• slf4j-api-1.7.6.jar (SLF4J API)
• slf4j-log4j12-1.7.6.jar (SLF4J Log4j Adapter)

The following shell script can be used anywhere just with Maven to get above files.
get goal of dependency plugin can be executed without pom file, so you just need the following script.
The local location to save the downloaded files can be specified by the value of -Ddest option of the mvn command-line in the script.

For Windows, you can use the following batch script.

References

• Connect to HiveServer2 with a kerberized JDBC client (Squirrel)
• SQuirreL SQL Client
• DBViewer Plugin for Eclipse
• Eclipse Data Tools Platform
• Apache Maven Dependency Plugin

Tutorials for Beginners on HTML, CSS, JavaScript and XML

HTML

• HTML5 Tutorial at W3Schools : http://www.w3schools.com/html/html5_intro.asp
• HTML Tutorial at W3Schools : http://www.w3schools.com/html/
• HTML 4 Reference at WDG : http://www.htmlhelp.com/reference/html40/

• For more, refer http://3rdstage.wikia.com/wiki/Readings_on_Information_Technology#HTML

CSS

• CSS Guide at WDG : http://www.htmlhelp.com/reference/css/
• CSS3 Tutorial at W3Schools : http://www.w3schools.com/css3/
• CSS Tutorial at W3Schools : http://www.w3schools.com/css/
• Selectors Level 2 : http://www.w3.org/TR/CSS2/selector.html
• Selectors Level 3 : http://www.w3.org/TR/2011/REC-css3-selectors-20110929/

• For more, refer http://3rdstage.wikia.com/wiki/Readings_on_Information_Technology#CSS

JavaScript

• JavaScript Tutorial at W3Schools : http://www.w3schools.com/js/
• JavaScript Guide at Mozilla Developer Network : https://developer.mozilla.org/en-US/docs/JavaScript/Guide
• JavaScript Reference at Mozilla Developer Network : https://developer.mozilla.org/en/JavaScript/Reference

• For more, refer http://3rdstage.wikia.com/wiki/Readings_on_Information_Technology#JavaScript

XML

• XML Tutorial at W3Schools : http://www.w3schools.com/xml/
• XML Tutorials : http://edutechwiki.unige.ch/en/Web_technology_tutorials#XML_tutorials

• For more, refer http://3rdstage.wikia.com/wiki/Readings_on_Information_Technology#XML_2

ToDo

empty

Documentation Management

Documentation Management

Examples of well-known products

• Fedora documentation
  
  • site : http://docs.fedoraproject.org/
  • tool : MediaWiki
• JBoss AS 7 documentation
  
  • site : https://docs.jboss.org/author/display/AS71/Documentation
  • tool : Confluence
• Red Hat documentation
  
  • site : http://docs.redhat.com/
  • tool : Publican, DocBook XML

Implicit type conversion guauantee proper execution plan when using character type value with numeric type column

Implicit type conversion guauantee proper execution plan when using character type value with numeric type column in Oracle

Explanation

Oracle database has lots of implicit type conversion rules and one of them goes like that when comparing a character value with a numeric value, Oracle converts the character data to a numeric value.
This is very important considering execution plan and the performance of query.

For example, with the following query where price column is varchar type.

When executing the above query, Oracle database converts the query like the following and so, there's no problem for the query to use the index on price column. In other words, the above query using character type value with numeric type column doesn't hurt the execution plan of the query and this can be generalized.

Sample

The following is sample table and script showing the above explanation.

Table

Script

DML script to execute query and identify the execution plan used.

Result

The execution of the above script goes like this

For more

• http://download.oracle.com/docs/cd/B14117_01/server.101/b10759/sql_elements002.htm#sthref363

Exception Handling of EJB

EJB 2.1

from Chapter 18. Exception Handling of Enterprise JavaBeans Specification Version 2.1

Exceptions from a Session or Entity Bean’s Business Methods

EJB 3.0

from Chapter 14. Exception Handling of Enterprise JavaBeans Specification Version 3.0

Exceptions from a Session Bean’s Business Interface Methods

Using SyntaxHighlighter in Redmine

Redmine provides built-in syntax highlighting based on CodeRay (for this, refer here). But it is not so much powerful as leading libraries such as SyntaxHighlighter.

You can use SyntaxHighlighter with Redmine by simply modifying one template file in the Redmine. The file to modify, base.rhtml is a template file specifying the layout of most pages and located at the following position.

The default contents of base.rhtml is like followings.

You can add a few header elements for styles and scripts to install SyntaxHighlighter like the following.

If your Redmine is located at a restricted network and so can reach 'http://alexgorbatchev.com/'. You should download SyntaxHighlighter and deploy it under the public directory of your Redmine installation.

When SyntaxHighlighter is deployed at ${redmine.home}/htdocs/pulbic/vendor/syntaxhighlighter_3.0.83 directory, the custom header elements would be like the following.

You should restart Redmine to apply the change of template. When loading Redmine using Apache HTTP server, just restart Apache HTTP server.

For more about SyntaxHighlighter refer its homepage at http://alexgorbatchev.com/SyntaxHighlighter/

Guidelines for Multi-moduled Maven Project []

Moved to the page http://3rdstage.wikia.com/wiki/Tips_on_Maven in my wiki as of 23th Jul. 2012.

Guidelines for Multi-moduled Maven Project

Separate parent project from container project.

Define properties in the parent POM file as possible.

Define the following properties explicitly for clearness and control.

• Encoding related properties whose default value are dependent on platform.
• project.build.sourceEncoding
• project.reporting.outputEncoding
• Properties to skip situationally unnecessary long running goals.
• skipTests
• maven.javadoc.skip
• maven.deploy.skip
• checkstyle.skip
• Properties to run maven more stable way.
• dependency.locations.enabled : refer If your maven site build is too slow
• Paths for executable programs external to Maven
• graphviz.home

Define all plugins in the <pluginManagement> element of parent POM file.

• Don't specify version to plugin definitions in child POM files.

Define repositories in the parent POM file as possible.

Define plugins to analyze or diagnose the sources or resources of the project as a reporting plugin as possible.

Start values for path with ${project.basedir}, ${project.build.directory}, ${project.build.outputDirectory} or so on

• When just specifying relative path such as 'target/classes', the result directory could be different if you build the project from the enclosing project in multi-module case.
• If you specify the path with leading ${project.build.directory}, the resulting directory always would be one that is calculated with the current POM's project.

Use resources in target/classes not in src/main/resources, considering resource filtering.

Sample of Parent POM

This is the real sample of parent POM I am using.

Frameworks, Libraries, Tools, and Utilities []

Moved to the page http://3rdstage.wikia.com/wiki/Frameworks,_Libraries,_Tools,_and_Utilities in my wiki as of 20th Jul. 2012.

Frameworks

Data Access/OR Mapping

• MyBatis : http://www.mybatis.org/
  couples objects with stored procedures or SQL statements using a XML descriptor.
  • MyBatis 3.0.3 API : http://www.jarvana.com/jarvana/view/org/mybatis/mybatis/3.0.3/mybatis-3.0.3-javadoc.jar!/index.html
• MyBatis-Spring : http://www.mybatis.org/spring/
  allows MyBatis to participate in Spring transactions, takes care of building MyBatis mappers and SqlSessions and inject them into other beans, translates MyBatis exceptions into Spring DataAccessExceptions, and finally, it lets you build your application code free of dependencies on MyBatis, Spring or MyBatis-Spring.
  • MyBatis-Spring 1.0.2 API : http://www.jarvana.com/jarvana/view/org/mybatis/mybatis-spring/1.0.2/mybatis-spring-1.0.2-javadoc.jar!/index.html

Logging

• log4jdbc : http://code.google.com/p/log4jdbc/
  a Java JDBC driver that can log SQL and/or JDBC calls (and optionally SQL timing information) for other JDBC drivers using the Simple Logging Facade For Java (SLF4J) logging system.

Templating

• FreeMarker : http://freemarker.sourceforge.net/
  a "template engine"; a generic tool to generate text output (anything from HTML to autogenerated source code) based on templates.

Testing

• JUnit : http://junit.org/
  a simple framework to write repeatable tests.
• TestNG : http://testng.org/
  a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use.

HTML/CSS

• Compass : http://compass-style.org/
  open-source CSS Authoring Framework.

Libraries

Java

• Commons-Collections with Generics : https://github.com/megamattron/collections-generic
  a new version of the popular Jakarta Commons-Collections project that introduces support for Java 1.5 Generics.
• Guava : http://code.google.com/p/guava-libraries/
  contains several of Google's core libraries that we rely on in our Java-based projects: collections, caching, primitives support, concurrency libraries, common annotations, string processing, I/O, and so forth.
• Prometheus : http://prometheus.codehaus.org/
  a concurrency library for Java 5 and higher and will be released under the MIT license.
• Jetlang : http://code.google.com/p/jetlang/
  a complement to the java.util.concurrent package introduced in 1.5 and should be used for message based concurrency similar to event based actors in Scala.
• JUNG : http://jung.sourceforge.net/
  a software library that provides a common and extendible language for the modeling, analysis, and visualization of data that can be represented as a graph or network.
• SIGAR : http://sourceforge.net/projects/sigar/
  a cross-platform, cross-language library and command-line tool for accessing operating system and hardware level information in Java, Perl and .NET.
• javaparser : http://code.google.com/p/javaparser/
  A Java 1.5 Parser with AST generation and visitor support.

Java Graphics

• JFreeChart : http://www.jfree.org/jfreechart/
  makes it easy for developers to display professional quality charts in their applications.

JavaScript

• Datejs : http://code.google.com/p/datejs/
  open source JavaScript Date library for parsing, formatting and processing.
• SyntaxHighlighter : http://alexgorbatchev.com/SyntaxHighlighter/
  a fully functional self-contained code syntax highlighter developed in JavaScript.

JavaScript Graphics

• Raphaël : http://raphaeljs.com/
  a small JavaScript library that should simplify your work with vector graphics on the web.
• Processing.js : http://processingjs.org/
  the sister project of the popular Processing visual programming language, designed for the web.
• D3 : http://d3js.org/
  a JavaScript library for manipulating documents based on data.
• Rickshaw : http://code.shutterstock.com/rickshaw/
  
  • a JavaScript toolkit for creating interactive time series graphs.
  • available under the MIT license.
• Google Chart Tools : https://developers.google.com/chart/
  provide a perfect way to visualize data on your website.
• JSXGraph : http://jsxgraph.uni-bayreuth.de/wp/
  a cross-browser library for interactive geometry, function plotting, charting, and data visualization in a web browser.



• Galleria : http://galleria.io
  a JavaScript image gallery framework.
• Nivo Slider : http://nivo.dev7studios.com/
  

JSON

• Jackson : http://jackson.codehaus.org/
  JSON processor (JSON parser + JSON generator) written in Java.
  • API for Jackson 2.0 Core Streaming
  • API for 2.0 Core Annotations
  • API for Jackson 2.0 Core Databinding

Expression Language

• Java Expression Language (JEXL) : http://commons.apache.org/jexl/
  a library intended to facilitate the implementation of dynamic and scripting features in applications and frameworks written in Java.

Tools

Java Code Analysis

• Sonar : http://www.sonarsource.org/
  Open platform to manage code quality.
• JDepend : http://www.clarkware.com/software/JDepend.html
  Traverses Java class file directories and generates design quality metrics for each Java package.
  • JDepend 2.9.1 API : http://mcs.une.edu.au/doc/javadoc/jdepend-2.9.1/
• Checkstyle : http://checkstyle.sourceforge.net/
  Development tool to help programmers write Java code that adheres to a coding standard.

Java Monitoring and Diagnostic

• VisualVM : http://visualvm.java.net/
  Visual tool integrating several commandline JDK tools and lightweight profiling capabilities.
• IBM Health Center : http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/
  enables you to assess the current status of a running Java application.

Java Dump Analysis

• IBM HeapAnalyzer : https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=4544bafe-c7a2-455f-9d43-eb866ea60091
  Analyzes Java heap dumps by parsing the Java heap dump, creating directional graphs, transforming them into directional trees, and executing the heuristic search engine.
• IBM Thread and Monitor Dump Analyzer for Java : https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
  Analyzes javacore and diagnoses monitor locks and thread activities in order to identify the root cause of hangs, deadlocks, and resource contention or monitor bottlenecks.
• Thread Dump Analyzer : http://java.net/projects/tda
  Analyzes Thread Dumps of Sun, SAP and JRockit (partial).
• Eclipse Memory Analyzer : http://www.eclipse.org/mat/
  Fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.

C/C++ Code Analysis

• Cppcheck : http://cppcheck.sourceforge.net/
  open source static code analyzer tool for C/C++ programming languages. It's a versatile tool that can check non-standard code.
• Dehydra : https://wiki.mozilla.org/Dehydra_GCC
  a lightweight, scriptable, general purpose static analysis tool capable of application-specific analyses of C++ code.

Web Development

• Firebug : http://getfirebug.com/
  Edit, debug, and monitor CSS, HTML, and JavaScript live in any web page.
• JSHint : http://www.jshint.com/
  a static code analysis tool used in software development for checking if JavaScript source code complies with coding rules.
• JSLint : http://www.jslint.com/
  a JavaScript program that looks for problems in JavaScript programs.

SQL Client/Database Management

• SQuirreL SQL Client : http://www.squirrelsql.org/
  A graphical Java program that will allow you to view the structure of a JDBC compliant database, browse the data in tables, issue SQL commands etc.
• IBM Data Studio : http://www-01.ibm.com/software/data/optim/data-studio/
  • Provides database developers and DBAs an integrated, modular environment for productive administration of DB2 for Linux, UNIX and Windows.
  • Provides collaborative database development tools for DB2, Informix, Oracle, and Sybase at no charge.
• Oracle SQL Developer : http://www.oracle.com/technetwork/developer-tools/sql-developer/

System Monitor Tool

• nmon : http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon
  free tool giving you a huge amount of important performance information in one go.
• top : http://www.unixtop.org/
  a UNIX utility that provides a rolling display of top cpu using processes.
• Nmap : http://nmap.org/
  a free and open source (license) utility for network discovery and security auditing.
• Wireshark : http://www.wireshark.org/
  lets you capture and interactively browse the traffic running on a computer network.

Installer

• IzPack : http://izpack.org/
  A widely used tool for packaging applications on the Java platform.
• Inno Setup : http://www.jrsoftware.org/isinfo.php
  A free installer for Windows programs.
• Nullsoft Scriptable Install System : http://nsis.sourceforge.net/
  A professional open source system to create Windows installers.

Documentation

• Publican : https://fedorahosted.org/publican/
  A tool for publishing material authored in DocBook XML.

Frontend Tools

• MySchedule : http://code.google.com/p/myschedule/
  a web application for managing Quartz Schedulers.

ETL

• Scriptella : http://scriptella.javaforge.com/
  An open source ETL (Extract-Transform-Load) and script execution tool written in Java.

Utilities

PC Diagnostic and Recovery

• Hiren's BootCD : http://www.hiren.info/pages/bootcd
  A bootable software CD containing a number of diagnostic programs such as partitioning agents, system performance benchmarks, disk cloning and imaging tools, data recovery tools, MBR tools, BIOS tools, and many others for fixing various computer problems.
  For download and more information, refer fan & discussion site.

File Synchronization

• rsync : http://rsync.samba.org/
  An open source utility that provides fast incremental file transfer.
  For front-end soft-wares or solutions based on rsync, refer http://en.wikipedia.org/wiki/Rsync#Solutions_using_Rsync.

misc

Graphics Asset

• Open Source Web Design : http://www.oswd.org/
  A site to download free web design templates and share yours with others.
• Tango Desktop Project : http://tango.freedesktop.org/
  Defines an icon style guideline to which artists and designers can adhere and provides a sample implementation of the style as an icon theme based upon a standardized icon naming specification.
• Silk Icons : http://www.famfamfam.com/lab/icons/silk/
  A smooth, free icon set, containing over 700 16-by-16 pixel icons in strokably-soft PNG format.
• Liquid Look And Feel : http://sourceforge.net/projects/liquidlnf/
  Java2 Swing Look and Feel of Mosfet Liquid KDE 3.x theme.

Readings on Business Applications

Data-warehouse, BI

• Introduction to OLAP