Local to dev server running a node instance via rsync on windows

This guide has now been extended, please visit here: http://www.beingyesterday.com/linux/bitbucket-sourcetree-ide-rsync-dev-server/

 

The following setup is good for remote teams, it allows everyone from directors to developers to view the work in progress. It keeps the dev’s computer light ie no local node required just a good ide or editor.

Step 1 – Download and install cygwin

https://cygwin.com/  Download the cygwin setup.exe file. Move the .exe file to a new location, eg “C:cygwin_install_directorycygwin.exe”.

The first run through the setup will create install the all the basics and the cywin terminal.
First run through:
1 – Now run the setup-x86_64.exe, choose the first option “Install from internet”.
2 – On the next screen leave the default install location to be “C:cygwin64”.
3 – On the next screen set the “Local Package Directory” to be the folder you created at the start.
4 – Connection type, whatever suits.
5 – Choose a mirror to download from.. I have always used the cygwin.mirror.constant.com
6 – Just click next and whatever else follows to finish the install.

You should now have a cygwin terminal available, this is a linux style terminal. In “C:cygwin64” you will find a std linux directory structure. You don’t have much else yet, ie no ssh or rsync or basically anything.
Second run through, you need to download nano, ssh and rsync:
1 – Run the setup-x86_64.exe again and use the above settings, ie same local package dir and same mirror.
2 – When you land at the “select packages” dialogue box it should look like this:
select cygwin pachages
3 – Don’t just install everything as it takes about a week to get it all, also you only a fraction of what is available. You will have to run this each time you want something new, ie 3 times for nano, ssh and rsync. Start by searching for “nano”. The interface is horrid. After searching for “nano” where it reads “Install” in the image below it should read default on your screen, press the word “Default” and it should switch all the options below to install… the interface really had no thought put into the design:
cygwin intsall
4 – Just hit next and it will download and install “nano” to c:cygwin64.
5 – Repeat the last four steps for ssh then rsync.

You now have rsync ready to go on you machine, you now just need to add the path to the binaries to your $PATH environment variable.
1 – Go to “Control Panel /All Control Panel /ItemsSystem” then click “Advanced system settings on the left”
2 – On the “advanced tab” click the “Environment variables” at the bottom.
3 – Might as well add the new path to the globals, in the second box scroll down and click to edit the Path variable and add your new option which should be “C:cygwin64bin;” like this:
windows path variable
4 – You now have all the unix tools in your windows cmd, eg to list the contents of a folder both the windows “dir” and the unix “ls” will work. You also have a right click with normal options that cmd bizarely missed out, if think cmd was designed by a chimp on drugs. Here’s a list of some basic unix commands: http://mally.stanford.edu/~sr/computing/basic-unix.html

Step 2 – SSH keys between your local machine and the dev server

You should now have ssh and rsync available to your windows command prompt which means any programs on your local machine have access to rsync, however you don’t want to be entering your password every single time you want to run rsync on your local file set to push changes to your dev server. This is where ssh keys come in.
Create your key set
1 – Open your cygwin terminal.
2 – Run

to make sure you are in you users home directory.
3 – Run

4 – Just press enter and accept all the defaults.
5 – This will have created a new folder “/home/john/.ssh/” with two files inside “id_rsa” and “id_rsa.pub”. List the contents of the folder to see the hidden “.ssh” folder with the options “l” and “a”:

Now you just need to place the public key onto the dev server.
1 – The id_rsa.pub is what goes on the server, run cat on the file to display:

2 – What the cat spat out you need to copy to the server. Open another cygwin terminal and ssh to your dev server:

3 – ensure you are in you home directory,

4 – If your home directory doesn’t have a .ssh folder in it, just run ssh-keygen to create it and accept all the defaults.
5 – Create/edit a file within .ssh/ called “authorized_keys”:

6 – Right click and paste the contents of your public key, it is important that there are no line breaks in the key else shit wont work, each key must be on its own line and must occupy a single line only. Comments start with hash #.
7 – In nano there are a bunch of options listed at the bottom of the page. With the key pasted in, exit the file “ctrl + x”, it will be asking some Q’s at the bottom of the page, just press enter till you exit nano.
8 – If this is a new authorized key file set the permissions to 0600:

That’s it, you should now be able to ssh to your dev server without being prompted for a password. Test this by logging out of the server and ssh’ing back in (‘logout’ from cli in the server to logout).

Running the rsync command

So far you should have the cygwin packages for ssh nano and rsync installed and you should be able to ssh to your dev server without being prompted for your password. Now lets try running rsync.

Try running this from a new cygwin terminal, obv with your own details in 🙂

The command broken down,
-avz options are to do with compression and speed.
–delete this flag will delete any files it finds on the target that do not exist on the source.
–chmod=ug=rwx,o=rx This sets the permission of the files on the target, this is required on windows as permissions are handled differently between unix and windows. Without it you files would not have any permissions set at all
-e”ssh -p 22″ this flag saying use port 22 for the ssh connection
/cygdrive/d/work/projectX/ this is the source, note the trailing /. cygdrive is kind of like the mnt folder in linux, it is where other drives are accessed.
john@123.123.123.123:/var/node/john.projectx.com/ this is the target note the trailing /
The / at the end of the paths there says to rsync, copy everything from within the folder.

See these two posts for linking rsync commands to editors: phpstorm/webstorm komodo edit/netbeans/eclipse

Keep your node alive and auto restarts during your rsync

pm2 is your friend here. This can keep your node application running even after you close your terminal. With the –watch flag, your node instance will restart each time a file is changed

After you have started your app you can tail the logs with

https://github.com/Unitech/PM2/blob/master/ADVANCED_README.md#raw-examples

Done

Now you are all setup. Run an rsync command and watch your pm2 logs fly by in a cygwin terminal, save a file and give your graphics designer a shout to get him to look at something without having to get him to come over to you.

2 thoughts on “Local to dev server running a node instance via rsync on windows”

  1. I would like to know if there is an alternative to rsync for doing backups.

    I currently use rsync to make incremental backups based on what is changed. I use the backup feature of rsync by configuring it to move replaced or deleted files into an “arch/YYYY-mm-dd” directory. The current copy of the backed up file tree is in “sync”. Both “arch” and “sync” are in a directory designated for each backup configuration.

    The primary problem is these backups get very large, with many millions of files. What rsync does is collect all the names of all the files into memory before doing any real transfers of data. This is putting a lot of memory stress on the source (to be backed up) and target (where backups are saved) systems.

    I believe such memory hogging is really not needed. At any one time, the most that should be needed is to keep the names of all files in each of the directory levels down to the one where the backup activity is currently working … not the entire tree. So I am looking for something that can do these things rsync does, but without doing this reading of all files in the entire tree.

    There is also a secondary problem. I want to make a backup of the backup. This is resulting in double transfers. Because files in “sync” (on the target, which is now the source for this secondary backup) get moved to “arch” when replaced, the new target gets these files transferred to it as new files in “arch”, even though it has a copy in “sync”. I tried the –fuzzy option on rsync to see if it would find the duplicate “somewhere else”. This has not worked.

    If a new program is made to specifically deal with this, and synchronize primary backups to secondary backups with a minimum of transferred data (replicate the previous moves), that would be great. But it will still need to do smart data-incremental transfers where files being replaced still have most of the old data just like rsync was originally designed for.

    If there is no such existing program, is there any interest in one being developed that focuses on incremental primary and secondary backups?

    1. Totally depends on what OS you are on.. but somewhat coincidentally I have been putting together a utility program running on electronJS to do just that.

      On Windows ViceVersa does precisely what you want.

Leave a Reply

Your email address will not be published. Required fields are marked *