CloudFront Setup to Route Requests to Services Based on Request Path

AWS CloudFront allows to have multiple origins for the distribution and, along with lambda@edge functions, that makes it possible to use CloudFront as an entry point to route the requests to different services based on the request path.

For example:

Read more »

Git Hook to Add Issue Number to Commit Message

When using project management system (Jira, Redmine, Github issues, etc) it is useful to add the issue number into commit message that makes is easier to understand which issue the commit belongs to and often allows the project management system to display related commits.

For same reasons, it is also useful to include the issue number into branch name, such as 123-branch-description or feature/PROJECT-123-branch-description.

Read more »

Managing NPM packages on github

Sometimes it is simpler to keep the package on github, for example, if you have a fork of a published package with some private changes. So you can avoid cluttering npm registry with similar packages, creating confusing for other people.

NPM supports installing dependencies from github, but it is also good to have versioning for your package so you can use it exactly as other packages, develop it independently and upgrade the dependency for the main project in a controlled way.

Read more »

Recording Linux Terminal Session to GIF with asciinema

The asciinema is a good and simple to use tool to record a screencast from the terminal session.

And asciicast2gif allows to convert the recording to gif animation.

Read more »

Debugging Python With ipdb and pdbpp

To get a very convenient full-screen console debugger for python, install ipdb and pdbpp packages.

Then use __import__('ipdb').set_trace() to start the debugger and enter sticky to switch to the full-screen mode.

Read more »

Formatting Parameter Blocks in Python

There are two ways recommended in pep-8 to format the blocks with long parameter lists in Python:

# Arguments start on the next line
foo = long_function_name(
    var_one, var_two,
    var_three, var_four)

Another way is:

# Arguments start on the same line
foo = long_function_name(var_one, var_two,
                         var_three, var_four)

I always prefer the first option and the other one is problematic for a few reasons.

Read more »

There Is No Callback Hell In JavaScript

There is no "callback hell" in Javascript, it is just a bad programming style. The infamous JavaScript "callback hell" can easily be fixed by un-nesting all the callbacks into separate functions.

Read more »

Disqus - code formatting and highlighting in comments

It is possible to format and have syntax highlighting for code in Disqus comments. To do that, wrap the code into <pre><code> tags (see the example comment to this post).

Read more »

SSH Tunnels (How to Access AWS RDS Locally Without Exposing it to Internet)

Using SSH tunnels, it is possible to access remote resources that are not exposed to the Internet through the intermediate hosts or expose your local services to the Internet.

Read more »

AWS error - Default subnet in us-east-1f not found

I suddenly started getting the Default subnet in us-east-1f not found error during the ElasticBeanstalk environment update.

Read more »

Simple Git Workflow

The main purpose of this workflow is to have a reliable, but simple to use git workflow. It is simple enough to be used by git beginners and minimizes possibility of mistakes (comparing to advanced flows which use rebase and related git features to achieve clean history).

The main idea of this workflow is that we create a new branch for every task and one developer works on this branch until the task is finished. Once done, the work from the branch is merged back to the master branch.

Read more »

Setup Automatic Deployment, Updates and Backups of Multiple Web Applications with Docker on the Scaleway Server

The purpose of this setup is:

  • Setup multiple web apps with different dependencies on the same server
  • Link all apps to the same MySQL server
  • Manage uploaded files for web apps in the single place (so it is easy to backup them)
  • Automatically deploy and update apps on the remote server
  • Run the same setup locally, so development environment is very close to production
  • Setup backups for MySQL databases and for uploaded files

In this case I deploy to Scaleway, but same approach can be used for almost any cloud service.

Read more »

OOP SOLID Principles "L" - Liskov Substitution Principle

According to the Wikipedia the Liskov Substitution Principle (LSP) is defined as:

Subtype Requirement:
Let f(x) be a property provable about objects x of type T.
Then f(y) should be true for objects y of type S where S is a subtype of T.

The basic idea - if you have an object of type T then you can also use objects of its subclasses instead of it.

Or, in other words: the subclass should behave the same way as the base class. It can add some new features on top of the base class (that's the purpose of inheritance, right?), but it can not break expectations about the base class behavior.

Read more »

AWS PostgreSQL RDS - remaining connection slots are reserved error

Today I had a problem with PostgreSQL connection, both my application and psql tool returned an error:

FATAL:  remaining connection slots are reserved for non-replication superuser connections

The PostgreSQL server was running on the db.t1.micro RDS instance and the 'Current activity' column showed '22 connections' and a red line which should represent a connection limit was far away from the 22 value.

Read more »

How to set up Drone CI on EC2 instance via Elastic Beanstalk

Drone CI is a Continuous Integration platform. It uses Docker containers to run tests for your application hosted on github.

It not complex to set up the automatic testing for your application and run Drone CI on EC2 instance using Elastic Beanstalk. It is even not necessary to have a dedicated EC2 instance for CI system, for example, I run it on the staging server.

Read more »

CloudWatch Logs - how to log data from multiple instances to the single stream

After using CloudWatch Logs for some time I found that it is very inconvenient to have one stream per instance. The Logs UI is really complex to use - I need to remember instance names, open the log group I need and then go into each instance logs one-by-one to check them.

A more convenient alternative is to use one stream like error_log for all instances.

Read more »

Elastic Beanstalk - how to setup CloudWatch Logs

CloudWatch Logs is an AWS service to collect and monitor system and application logs. On the top level setup is this:

  • install CloudWatch agent to collect logs data and send to CloudWatch Logs service
  • define log metric filters to extract useful data, like number of all errors or information about some specific events
  • create alarms for metrics to get notifications about logs
  • make sure that the instance role has permissions to push logs to CloudWatch (see comments for details about this issue)

All the configuration can be done using the Elastic Beanstalk config. In this case when the new environment is launched or existing environment is updated the CloudWatch Logs setup is done automatically.

There is an example of the configuration in the Elastic Beanstalk docs - Using Elastic Beanstalk with Amazon CloudWatch Logs. The Setting Up CloudWatch Logs Integration with Configuration Files section briefly describes how to use config examples for different environments, but there is no detailed information about the config files. And config files are complex. It is easy to use examples as is, but it is not that easy to do the setup for own logs. I will start with the review of the Apache access log example and will show how to change it to collect data from the error log as well.

Read more »

Elastic Beanstalk - python application server structure and celery installation

Elastic beanstalk python application is deployed under /opt/python/. The application is running under Apache web server.

Read more »

Local Amazon DynamoDB - tools, dump/restore and testing


Download and extract dynamodb local to some folder.

Launch it (-sharedDb allows us to connect to the same database with other tools):

    $ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb

By default it will be running on the port 8000 and will create the db file in the same directory where it was launched.

Read more »

Amazon DynamoDB - how to add global secondary index

Note: this post is outdated, because it is already possible to add a secondary index to the existing table (it was not possible in earlier DynamoDB versions).

At the moment it is not possible to add a secondary index into the existing table. This feature is announced but not yet available.

So the only way is to create a new table and migrate the existing data to it. This can be done using Amazon EMR.

Read more »

Amazon DynamoDB, EMR and Hive notes

First you need the EMR cluster running and you should have ssh connection to the master instance like described in the getting started tutorial.

Now it is possible to run Hive commands in few following ways:

  • Connect via ssh, launch hive and run commands interactively
  • Create a script file with commands, upload it to S3 and launch as a ERM 'Hive program' step
  • Run it from Hue web-interface (see below)
Read more »

AWS - Deployment via OpsWorks from the command line

Below is a simple python script which performs application deployment using OpsWorks API library (boto). Script performs following steps

  • Execute 'update_custom_cookbooks' deployment command and wait for successful completion (or stop with an error)
  • Execute 'deploy' command and wait for completion
Read more »

AWS OpsWorks - setup mongodb ebs volume backups

I described how to setup mongodb on EC2 using OpsWroks and here is how to setup mongo data backups.

In my case all mongo data is stored on the same EBS volume so I just need to make a volume snapshot.

Read more »

Amazon OpsWorks - node.js app with MongoDB setup

Amazon OpsWorks provides a way to manage AWS resources using Chef recipes.

Here I describe a simple setup of the single-instance node.js app with single-node MongoDB server. It is similar to the php application + mysql setup described in the OpsWorks Getting Started guide.

Read more »

Git - how to revert multiple recent commits

Let's assume we have a history like this:

G1 - G2 - G3 - B1 - B2 - B3

Where G1-G3 are 'good' commits and B1-B3 are 'bad' commits and we want to revert them.

Read more »

How to keep git log and less output on the screen

When git uses less as pager the output of commands like git log disappears from the console screen when you exit from less. This is not convenient in many cases so here is how to fix this.

Read more »

Node.js - how to debug mocha test with node inspector

To debug mocha test with node inspector use the delay before test:

Read more »

Node.js - how to get core module source

It is possible to ask node to show its core module source.

Read more »

Elastic Beanstalk - cron command and RDS DB access


I have a console command in php which needs an access to DB. The command need to be launched via cron.

Read more »

Elastic Beanstalk - deploy from different machines / by different users (or how to get rid of absolute paths in configs)

By default Elastic Beanstalk console tool (eb) adds config files to .gitignore. If there are manual changes to EB configs it can be complex to manually sync these changes between different machines / different users. Of cause it is possible to add config files to git repository but there are also several parameters in the main config which are absolute paths to local files. This way it makes configs not useful for other users (except for the case when different users have exactly the same files layout). Here is how this problem can be fixed:

  1. Add eb tools under git control
  2. Add eb configs under git control
  3. Patch eb tools to accept relative file paths
  4. Change paths in configs to relative

Below are details for each step.

Read more »

JS libraties for charts (links)

Read more »

How to setup git server on ubuntu with push email notifications

Git Server

Prerequisites are git and ssh-server (apt-get install openssh-server).

The installation process is described in the Pro Git book. Below is the setup process with some comments and updates.

Read more »

Amazon NoSQL Solutions

Amazon provides following NoSQL storage options:

  • [SimpleDB] (http://aws.amazon.com/simpledb/) - Amazon SimpleDB is a highly available and flexible non-relational data store that offloads the work of database administration. Developers simply store and query data items via web services requests and Amazon SimpleDB does the rest.
  • [DynamoDB] (http://aws.amazon.com/dynamodb/) - Amazon DynamoDB is a fully-managed, high performance, NoSQL database service that is easy to set up, operate, and scale.
Read more »

Angular.js and SEO - pre-render content on the server

With angular.js you have an HTML which looks like this:

        <li ng-repeat="item in items" ng-bind="item.name"></li>

The simple way to make this content SEO-friendly is to pre-render data on the server and then allow angular to do it's job on the client.

Read more »

Angular.js POST data to PHP

By default angular.js sends all data in json. So if you do a POST request to a PHP code then the $_POST superglobal will not be populated.

This can be solved in two ways - on the client side or on the server side.

Read more »

PHP - utf-8 strings handling

Enable mbstring function overloading mode and set default encoding for string functions to utf-8 in php.ini:

mbstring.internal_encoding = UTF-8
mbstring.func_overload = 7
Read more »

Want Scalable Application Architecture? Check AngularJS.

The Scalable JavaScript Application Architecture is a presentation by Nicholas Zakas where he suggests a flexible and scalable architecture for JavaScript applications. Here are other related resources:

Read more »

Speedup unit tests by moving MySql data to memory [Ubuntu]

There are several ways to speedup slow unit tests which interact with database:

  • Refactor code and tests and do not interact with db in unit tests
  • Use sqlite db in memory instead of MySql
  • Use MySql MEMORY engine
  • Move MySql data to memory
Read more »

PHP - friend a class via extend

C++ allows to declare one class as a friend of another one.

This can be useful if you want to keep some details of class protected, but available for another particular (friend) class.

Read more »

Window managers for Google Chrome

A list of extensions to manage position of chrome tabs / windows and to create splits. Descriptions are original texts from extension pages.

Read more »

GImport yii extension

GImport yii extension implements recursive import of directories with caching.

Import is performed recursively for specified path alias. Classes found are cached, so the import process can be slow only first time.

Read more »

phpmyadmin and eaccelerator problem

Error when trying to access phpmyadmin (in Chrome):

Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data.
Read more »

git - git rm already deleted files

    git add -u
Read more »

git - find not merged branches

Read more »

git - view changes - diff and log

Diff staged changes

    git diff --cached
    # or
    git diff --staged

Diff pulled changes

    git pull origin
    git diff @{1}..
Read more »

git - Your branch is ahead of 'origin/master' by 1 commit after pull

Your branch is ahead of 'origin/master' by 1 commit (or X commits) after git pull origin master.

The sequence:

  • Have up-to-date repository
  • There is a change in the origin/master
  • Do git pull origin master
  • Change is received and merged
  • git status shows “Your branch is ahead of 'origin/master' by 1 commit.”

The reason is because during “pull origin master” reference to the remote origin/master is not changed (still points to older version).

Read more »

git - find all branches where file was changed

Solution from stackoverflow.

Find all branches which contain a change to FILENAME (even if before the (non-recorded) branch point):

    git log --all --format=%H FILENAME | while read f; do git branch --contains $f; done | sort -u
Read more »

Yii and jquery.localtime.js - display dates in user local timezone

With this method we work on the server with UTC timezone dates and convert them to a user local timezone on client.

Read more »

global ajax response handler and jquery.localtime plugin

The jquery.localtime plugin allows to convert date/time strings to a local user time on a client site. By default it works when the page is loaded initially, but if some elements are updated via ajax then they do not converted and left in an UTC format.

Possible solution is to add some special handling to $.ajax 'success' handlers, but it can require a lot of modifications. Better way is to set some global handler for all ajax requests and apply conversion to local time there. I evaluated several approaches before I could find a working solution.

Read more »

innodb notes

Below are some notes regarding the innodb transactions and locks. Most of them are just a copy-paste from innodb docs with my notes. Also there are some examples of custom-maid deadlocks.

Read more »

git - submodule helpers

Below are some git commands which can be useful to resolve problems with submodules.

Read more »

Yii url rules - optional parameters

Assume we have an action "articles/get" which accepts optional parameters and we want to setup following URLs:

articles/[article id or name]
articles/[article id or name]/draft
articles/[article id or name]/revisions/99
articles/[article id or name]/revisions/98/draft
articles/revisions/[revision id]
articles/revisions/[revision id]/draft
Read more »

selenium - problem with big file upload


Selenium 2.25.0, python 2.7, ubuntu 12.04, Firebox 4.0 (yes, old version but we need it) hangs when uploading a file larger then (about) 600KB.

Read more »

express.js and ejs - reuse template on server and client

ejs has a client-side support but documentation and examples do not describe how to reuse the same template on the server and on the client side.

For now I found two ways to do it. First way is to send a request from the client, get a template from a file and render it. And the second - put a template into the page when render it on the server and then just use the template on the client.

Read more »

selenium webdriver - get webelement by jQuery selector

This can be necessary for example for selector like #id > li:visible.

If you will try to do webdriver.find_element_by_css_selector you will get an error message "The given selector #id > li:visible is either invalid or does not result in a WebElement."

Read more »

selenium webdriver - trigger event on element via jQuery

The 'executeScript' method of the webdriver receives additional 'arguments' variable and we can pass WebElement instances to the script. So trigger an event on the elemen can be done like this (python):

event = 'click' #or 'hover' or any other
script = "$(arguments[0]).trigger('"+event+"')"
webdriver.execute_script(script, web_element)
Read more »

selenium webdriver - set php session cookie

To set the php session cookie we can use the addcookie (or python version add_cookie) method of the webdriver. But it accepts only name and value and does not allow to set additional cookie parameters like domain, path, etc.

Fortunately it is easy to do with javascript.

Read more »

Andriod - moblie network problem after BusyBox update

I have LG p500 (Optimus One), rooted with original ROM.

After update to BusyBox 1.20.1 my moblie GPRS network became broken. I got network indicator (E with two arrows), but can not access network.

I suspected that this was someting with dns settings, but did not find any solution. I uninstalled busybox, but this didn't help. I did a factory reset - also didn't help. Actually I thought factory reset wil return my device to the original state, but then noticed that I still have root and busybox.

Read more »

git - use vim with fugitive to resolve merge conflicts

To use fugitive as you mergetool you can use the following.

    git config --global mergetool.fugitive.cmd 'vim -f -c "Gdiff" "$MERGED"'
    git config --global merge.tool fugitive
Read more »

vim - replace a word with yanked text

Copy a word and paste it over other words:

    yiw     yank inner word (copy word under cursor, say "first").
    ...     Move the cursor to another word (say "second").
    viwp     select "second", then replace it with "first".
    ...     Move the cursor to another word (say "third").
    viw"0p     select "third", then replace it with "first".
Read more »

git - rename branch (local and remote)

    #rename local branch
    git branch -m old-branch-name new-branch-name

    #delete remote branch with old name
    git push origin :old-branch-name

    # create remote renamed branch
    git push origin new-branch-name
Read more »

yii - catch and log MySQL deadlock errors

This method allows to log InnoDB monitor output when deadlock error occured. This way we will have much more useful data to find and fix deadlock.

Read more »

oauth 1.0 notes

Read more »

git - colored diff, branch, etc by default

See color.* options in the git config docs:

    $ git config color.branch auto
    $ git config color.diff auto
    $ git config color.interactive auto
    $ git config color.status auto
Read more »

selenium - run tests on a virtual display

Selenium tests require browser to run, so usually we run them on the X-server enabled machine. But in some cases, like CI system running on the headless EC2 instance, we want to run it on the virtual display. This can be done using xvfb (X virtual framebuffer).

Read more »

selenium - python Firefox webdriver - unsafe setters in firefox_profile.py

Problem: after setting the boolean flags for Firefox profile the webdriver fails with "Can't load the profile" error.

Read more »

selenium - problem with loading x_ignore_nofocus.so

Problem: selenium fails to start Firefox with following error:

'The browser appears to have exited before we could connect.
The output was: Failed to dlopen /usr/lib/libX11.so.6\ndlerror
says: /usr/lib/libX11.so.6: wrong ELF class: ELFCLASS32\n'

In my case it was reproduced on the 64 bit machine with Amazon Linux AMI. The problem itself is known and there is an issue in selenium tracker.

Read more »

Git - Your branch and 'origin/xxx' have diverged

Git error (not after rebase, see below):

Your branch and 'origin/xxx' have diverged,
and have 1 and 1 different commit(s) each, respectively.

Error is caused by two independent commits - one (or more) on the local branch copy and other - on the remote branch copy (for example, commit by another person to the same branch)

Read more »

git - checkout and track remote branch

#creates and checks out "feature" branch that tracks "origin/feature"
$ git checkout -t origin/feature
Read more »

yii - class table inheritance

It seems that we have no perfect solution for class table inheritance (or multiple table inheritance) in yii (comparing to the very good one for single table inheritance).

Possible solutions are:

  1. Add support for class table inheritance to the active record class. There are some implementations of this method (see here and here for examples). But I do not like this approach because it is too complex to implement it properly and to make it work for all possible active record usages.
  2. Use MySQL VIEWs. MySQL supports a VIEW which can be updated by INSERT and UPDATE statements. So we can hide two tables (for base and child classes) behind the view and use it as a usual single table.
  3. Use single table inheritance and keep an extended data in separate tables.
Read more »

jQuery - check minimal required version

To check whether jQuery is loaded to the page and verify minimum version:

if (typeof jQuery == 'undefined' ||
) {
    throw('jQuery version 1.3.1 or above is required');
Read more »
profile for Boris Serebrov on Stack Exchange, a network of free, community-driven Q&A sites