This is an extension of the previous guide: windows to linux box with rsync and ssh This extension includes setting up your working copy of git repository from BitBucket
The following setup is good for remote teams, it allows everyone from directors to developers to view the work in progress. It keeps the dev’s computer light ie no local node, no cpu/memory hogging virtual box required just a good ide or editor and a few light weight linux tools made available vie cygwin
Things you will need to download:
- SourceTree for Git: https://www.sourcetreeapp.com/
- Cygwin for ssh and rsync: https://cygwin.com/ and click the “setup-x86_64.exe” for a 64 bit OS (which is most OS’s these days)
- An IDE, WebStorm is good: https://www.jetbrains.com/webstorm/
Step 1 – Getting some code from your bitucket repository via sourcetree to your local machine.
First up, download source tree: http://www.sourcetreeapp.com/
Once you have it downloaded, first up you will need to register it with Atlassian, you should be prompted once you have it installed and opened for the first time.
With sourcetree installed and registered the next few steps (with BitBucket, as they are atlassian) are simply clicking a few links from the bitbucket website.
- Login to bitbucket and navigate to the repository you need to clone (_the term clone in git basically means the initial checkout of the master branch of the repository. NB all git repo’s have a master branch. Git was designed for linux by the linux man himself Linus Benedict Torvalds, the story goes that Mr Torvalds needed a source control for linux but basically thought all the existing options at the time were not up to scratch. With linux being complex with 1000’s of files and multiple devs he wanted a way for individual developers to work on different sections of the linux files simultaneously and never have one feature impede the release of another. As such, the master branch is deemed the live branch or in other words, the files used by a live website are the files in the master branch. All other branches are there for development purposes only_). So anyway… once in your bitbucket account on the website you should expect to see something like this:
- Click on the first link in the lhs actions menu bar, “Clone”, which should leave you here:
- You have two options now, you can either manually clone the git repo to you local machine with the command is has shown you, or with sourcetree open.. click the “Clone in SourceTree” button. The source tree app should then prompt you for a location to dump the repository on your local machine. Follow the rest of the on screen instructions and you should be left with all the files on your local machine and a bookmark in the lhs bookmarks column in sourcetree.
- Last but not least, you need to assign an email and name to the repo in source tree, you can assign globals but working on multiple projects with different teams i like to use different emails. With the repository open in source tree, go to Repository > Repository Settings from the top menu and the following should pop up. Enter your details and hit ok.
- One more thing. When you work on a file in source tree, the work is first “staged”, then “comitted”, then “pushed”. You stage the work you want to commit, you commit your work to your local working copy, you then push your commit to the source aka bitbucket. The other devs (if they are working in your branch too) would then pull the commit down which would merge the commit in.
- The commit and push steps can be combined (advised else to commit means first press commit, type your msg, then commit, then push then choose branchs blah blah) with sourcetree and the frequency of checks for new work to pull down can configured. Hit Tools > Options:
Step 2 – Sorting out gitflow
As mentioned in step 1, the master branch is the live branch. Dev work is never actually done in the master branch, it is carried out in other development branches and merged into the master branch after testing etc. This way everyone can be sure that the master branch is essentially the latest stable version of the app in use by your customers.
GitFlow… There are lots of different ways to handle branching with git. I have experimented with a few but none have come anywhere close to the neatness of gitflow. Here is a diagram from the atlassian guys but i will explain how to actually use gitflow below: https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow
- You will need to configure gitflow per repository. Just hit the “Git Flow” button and hit OK to accept the defaults (if you choose not to accept the defaults you would have to make sure the rest of your team does the same else this is not going to work):
- That’s basically it. You have now configured gitflow, and the source tree program should have automatically configured your config file which you can see if you go back to Repository settings and hit edit config file (don’t choose notepad as file wont be windows format thus notpad just bundles the whole file onto one line). What should also have happened is you now have two branches under your banches section, master and develop. The new branch that has just appeared will at this point in the game be exactly the same as master (unless you are new team member to an existing code base then it will likely be not).
- Under the “Remotes” sections in sourcetree in an established project using gitflow you would expect to see something akin to this:
There are three folders, feature, hotfix and release.. within each are a bunch of other branches. The remotes are essentially all the branches ever created that live at the source, which in this case is bitbucket. You can checkout any of the branches by double clicking on one or right clicking and selecting checkout.
This is a lot of text to explain the concept of gitflow but I promise is logical and once rolling so efficient and so much more less painful than any other branching i have used (you may here the term cherry picking, this is basically like a kick in balls with a migraine, it basically means picking a particular commit from one branch to another).
1 – Lets say there are no features currently in development and the plan is to create a new page to the app. You would press the gitflow button and choose “Start New Feature”:
You would give your new feature a name and hit ok. This would automatically create a new branch with the name you just entered. Now earlier I mentioned that the develop branch that appeared was exactly the same as master… the new branch you have just created is now an exact copy of develop.
You would now go ahead and create in the new feature branch your new page for the app and commit everything to the said branch. Nothing more complex than that. When you decide you have finished your new feature you simply hit the gitflow button and click “Finish hotfix”. This would tell source tree to merge your new feature and all the hunks of code back into develop (note, not master).
So at this point, develop and master are no longer in sync, develop contains a new feature and master is still just master.
Now lets say another dev needed to create another feature to the app, but starting during the time you were working on your feature. He would be creating a new feature branch based on dev as it was when you started you feature branch. But lets say you finished your branch before this other dev finished his.. when the second feature was finished, his feature branch would now be being merged back into dev but with your new code in too… git will seemlessly combine all the code together and you should be left with the develop branch now with two new features in it waiting to go live (if there were any collisions, ie you both changed code on the same line in the same file, source tree would let you know and the two devs would need to talk to ensure the right work is kept. However, this should not really happen often as long as the dev team is communicating so as not to work on the same files.. ie there is good code separation).
So, you have now got develop with two new features in it. The next step in gitflow is to create a release branch, ie press gitflow and click new release. The release branch is now a new branch based on develop. You might call this release 1.0.2 or something equally as professional looking :). This is now when all the new features are testing together.. you would probably run your automated tests on this new release branch too. If there are any issues, this is were the fixes are applied. Once everyone is happy that the release branch is good to go, guess what.. one of the devs presses gitflow and finishes the current release.
Finishing a release now merges the release branch into both develop and finally master. So develop now contains any fixes that were applied in the release branch and master has a bunch of new features. Bish bash bosh.
The last thing with gitflow is a hotfix. As is always, bugs are missed during testing. So even after all the testing during the build of the new features and the testing in the release branch a stupid little bug managed sneak itself all the way to master aka live aka your angry customers. This is when you would create a hotfix.
Gitflow > create new hotfix. The hotfix is a branch from master (as this is where the bug was spotted. The fix is done in the hotfix branch and comitted. When everyone is happy with the quick fix, the hotfix is finished.. gitflow > finish hotfix. The hotfix branch is now merged into master and also dev. It can get a little complex sometimes here.. merging the fix to master is never an issue however sometimes when there are a few features in develop waiting to go live and a hotfix is merged in there can be conflicts but this cannot be avoided. You would have to manually go in and decide how the hotfix can be applied to develop whilst not interfering with the new features…. And once again, when all the new features are ready a new release is created which would now include all the new features and now a hotfix, the release is tested and when finished merged in master and develop… and so the it goes on and on and on and on… continuous integration aka ci.
Step 3 – Download and install cygwin
https://cygwin.com/ Download the cygwin setup.exe file. Move the .exe file to a new location, eg “C:cygwin_install_directorycygwin.exe”.
The first run through the setup will create install the all the basics and the cywin terminal.
First run through:
1 – Now run the setup-x86_64.exe, choose the first option “Install from internet”.
2 – On the next screen leave the default install location to be “C:cygwin64”.
3 – On the next screen set the “Local Package Directory” to be the folder you created at the start.
4 – Connection type, whatever suits.
5 – Choose a mirror to download from.. I have always used the cygwin.mirror.constant.com
6 – Just click next and whatever else follows to finish the install.
You should now have a cygwin terminal available, this is a linux style terminal. In “C:cygwin64” you will find a std linux directory structure. You don’t have much else yet, ie no ssh or rsync or basically anything.
Second run through, you need to download nano, ssh and rsync:
1 – Run the setup-x86_64.exe again and use the above settings, ie same local package dir and same mirror.
2 – When you land at the “select packages” dialogue box it should look like this:
3 – Don’t just install everything as it takes about a week to get it all, also you only a fraction of what is available. You will have to run this each time you want something new, ie 3 times for nano, ssh and rsync. Start by searching for “nano”. The interface is horrid. After searching for “nano” where it reads “Install” in the image below it should read default on your screen, press the word “Default” and it should switch all the options below to install… the interface really had no thought put into the design:
4 – Just hit next and it will download and install “nano” to c:cygwin64.
5 – Repeat the last four steps for ssh then rsync.
You now have rsync ready to go on you machine, you now just need to add the path to the binaries to your $PATH environment variable.
1 – Go to “Control Panel /All Control Panel /ItemsSystem” then click “Advanced system settings on the left”
2 – On the “advanced tab” click the “Environment variables” at the bottom.
3 – Might as well add the new path to the globals, in the second box scroll down and click to edit the Path variable and add your new option which should be “C:cygwin64bin;” like this:
4 – You now have all the unix tools in your windows cmd, eg to list the contents of a folder both the windows “dir” and the unix “ls” will work. You also have a right click with normal options that cmd bizarely missed out, if think cmd was designed by a chimp on drugs. Here’s a list of some basic unix commands: http://mally.stanford.edu/~sr/computing/basic-unix.html
Step 4 – SSH keys between your local machine and the dev server
You should now have ssh and rsync available to your windows command prompt which means any programs on your local machine have access to rsync, however you don’t want to be entering your password every single time you want to run rsync on your local file set to push changes to your dev server. This is where ssh keys come in.
Create your key set
1 – Open your cygwin terminal.
2 – Run
to make sure you are in you users home directory.
3 – Run
4 – Just press enter and accept all the defaults.
5 – This will have created a new folder “/home/john/.ssh/” with two files inside “id_rsa” and “id_rsa.pub”. List the contents of the folder to see the hidden “.ssh” folder with the options “l” and “a”:
Now you just need to place the public key onto the dev server.
1 – The id_rsa.pub is what goes on the server, run cat on the file to display:
2 – What the cat spat out you need to copy to the server. Open another cygwin terminal and ssh to your dev server:
3 – ensure you are in you home directory,
4 – If your home directory doesn’t have a .ssh folder in it, just run ssh-keygen to create it and accept all the defaults.
5 – Create/edit a file within .ssh/ called “authorized_keys”:
6 – Right click and paste the contents of your public key, it is important that there are no line breaks in the key else shit wont work, each key must be on its own line and must occupy a single line only. Comments start with hash #.
7 – In nano there are a bunch of options listed at the bottom of the page. With the key pasted in, exit the file “ctrl + x”, it will be asking some Q’s at the bottom of the page, just press enter till you exit nano.
8 – If this is a new authorized key file set the permissions to 0600:
chmod 0600 authorized_keys
That’s it, you should now be able to ssh to your dev server without being prompted for a password. Test this by logging out of the server and ssh’ing back in (‘logout’ from cli in the server to logout).
Step 5 – Running the rsync command
So far you should have the cygwin packages for ssh nano and rsync installed and you should be able to ssh to your dev server without being prompted for your password. Now lets try running rsync.
Try running this from a new cygwin terminal, obv with your own details in 🙂
rsync -avz --delete –-chmod=ug=rwx,o=rx -e"ssh -p 22" /cygdrive/d/work/projectX/ firstname.lastname@example.org:/var/node/john.projectx.com/
The command broken down,
-avz options are to do with compression and speed.
–delete this flag will delete any files it finds on the target that do not exist on the source.
–chmod=ug=rwx,o=rx This sets the permission of the files on the target, this is required on windows as permissions are handled differently between unix and windows. Without it you files would not have any permissions set at all
-e”ssh -p 22″ this flag saying use port 22 for the ssh connection
/cygdrive/d/work/projectX/ this is the source, note the trailing /. cygdrive is kind of like the mnt folder in linux, it is where other drives are accessed.
email@example.com:/var/node/john.projectx.com/ this is the target note the trailing /
The / at the end of the paths there says to rsync, copy everything from within the folder.
See these two posts for linking rsync commands to editors: phpstorm/webstorm komodo edit/netbeans/eclipse
Step 6 – Keep your node alive and auto restarts during your rsync
pm2 is your friend here. This can keep your node application running even after you close your terminal. With the –watch flag, your node instance will restart each time a file is changed
pm2 start server.js --watch
After you have started your app you can tail the logs with
Now you are all setup. You pull and commit your code via sourcetree to bitbucket, edit your code in webstorm and sync your working copy files to the dev server for all to see.
Run the rsync command and watch your pm2 logs fly by in a cygwin terminal, save a file and give your graphics designer a shout to get him to look at something without having to get him to come over to you.