Continuous Delivery and Continuous Deployment

 From CI to Continuous Delivery deployment

What exactly is Continuous Delivery, Its attitude to make software safe and to go fast
Lets begin with basics, How much time does it take to deploy one line change or a small bug fix in to the production? Exactly, our goal should be to achieve this as fast as we can.

Why We Need Continuous Delivery?

• Yes, its Speed which is needed in current fast paced software industry.
• Stop waiting around on each other to provide go ahead and open the gate.
• Quality – It raises the overall quality of your application – No surprises.
• Capacity and Clarity – Get every team on one tool – Eliminate manual error

How are we going to do this…..

Key Areas:-

• Speed is essential because there is an opportunity and cost associated with not delivering Software
• Frequent Releases, How frequent is frequent? Why do developers need to wait after writing code?
• Every build can be potential release candidate.

Bottom Line

Remove Manual Work/Steps.
Move towards complete automation

Delivery Flow

Here is the Delivery Flow.

Ingredients:-

Configuration Management
continuous integration/ Automated Build
Automated Deployment
Automated Testing

Obviously no single team in a company alone can achieve CD without the help of other teams. It needs collaboration and team work to implement it, Lets take a look at what do we need from various teams

Dev Team

• Collaboration – Integration is no longer an event
• Check in regularly
• Make small frequent releases because Big Change -> Big risk –> Big effort and Small Change –> Small risk –> Small effort

QA team

• Should have complete automation or whatever max is possible. More manual testing leads to more delay.
• Keep the Build and Test Process Short
• Break the QA automation into small pieces so that they can run simultaneously and do the QA frequently.
• Publish the report

Environment

• Should have complete Environment to test everything.
• Flexible and scalable so that multiple env can be spawn any time and can be extended.
• Managed as prod or at least similar to prod

This increases Confidence in build’s production readiness reducing release risk.

Main blocker to achieve CD

1. manual work
2. Lengthy Process
3. ticket system
4. approval

Now, How do we deal with blockers

Do not add gates, add only checks. Adding gates to prevent issues from creeping into prod is not so smart. There are many more issues hiding, you just don’t know them

Approval!!! Why do we need CAB?

Dev Sign off
QA sign off
Ops and Dependency sign off
Infra sign off

Dev Sign off -> Use proper project mgmt tool to track everything that goes in to software with adequate workflow. Get everyone together at the beginning so that these will not be surprises at the last moment. Keep meeting across the team during entire development. This way you don’t need any approval since you have already taken care of all necessary elements during the entire development process.

QA Sign off -> Publish consolidated QA report with each and every build which should be visible to everyone. You should have proper definition of “Go Ahead” instead of just random checks, Nobody has to ask QA for a thumbs up for the go. • Minimum criteria for Q has to be set.

Ops, Dependency and Infra Sign off -> Include them at the beginning of the project. Have capacity and change planning during requirement gathering, This gives enough time for ops and infra to get ready for the release.

Role of SCM/ Release Mgmt

SCM/ Release Mgmt provide the workflow, tools and necessary filler to enable all teams to interact and automate their part.
The picture below shows How SCM and Release Management help each team.

Frequently used SVN Commands

SVN COMMANDS

SVN Commit – Commit/Chekins changes into Repo

Syntax:
$ svn commit -m "Enter Log messages"

SVN Diff – To See Difference in files

Syntax:
$ svn diff filename

SVN Status – Status of the working copy

Syntax:
$ svn status PATH

SVN add — Add files, directories, or symbolic links

Syntax:
$ svn add path

SVN copy (cp) — Copy a file or directory in a working copy or in the repository.

Syntax:
$ svn copy foo.txt bar.txt

SVN mkdir — Create a new directory under version control.

Syntax:
$ svn mkdir URL

Examining History in Subversion

Your Subversion repository is like a time machine. It keeps a record of every change ever committed and allows you to explore this history by examining previous versions of files and directories as well as the metadata that accompanies them.

Several commands can provide you with historical data from the repository:



svn log

Shows you broad information: log messages with date and author information attached to revisions and which paths changed in each revision

svn diff

Shows line-level details of a particular change

svn cat

Retrieves a file as it existed in a particular revision number and displays it on your screen

svn list

Displays the files in a directory for any given revision



Subversion's Components


Subversion, once installed, has a number of different pieces. The following is a quick overview of what you get. 



svn

The command-line client program

svnversion

A program for reporting the state (in terms of revisions of the items present) of a working copy

svnlook

A tool for directly inspecting a Subversion repository

svnadmin

A tool for creating, tweaking, or repairing a Subversion repository

mod_dav_svn

A plug-in module for the Apache HTTP Server, used to make your repository available to others over a network

svnserve

A custom standalone server program, runnable as a daemon process or invokable by SSH; another way to make your repository available to others over a network.

svndumpfilter

A program for filtering Subversion repository dump streams

svnsync

A program for incrementally mirroring one repository to another over a network

The Roles & Responsibilities of a 'Build and Release Engineer'

The Build and Release Engineer will play a main role in the SDLC life cycle

Roles & Responsibilities :-

• Work with Engineering, Operations, Quality Assurance and Project Management to define and execute projects that implement build, release and deployment of web software.

• Manage the source code control system.

• Develop an automated, continuous, build process that reviews the source code, identifies build errors and notifies appropriate parties to expedite/facilitate synchronization to the latest build.

• Improve productivity by designing & developing full-featured build systems; implementing tools and scripts that enable efficient, flexible builds (Shell, Perl,Python,groovy)

•  Develop an efficient deployment process for web deployments.

• Identify and propagate best practices and processes

• Communicate releases related activities to all the stakeholders and take their feedback to improve the process.
•Implement Continuous integration in organisation


Skillset Required :-

• Extensive experience working with Cruise Control,Jenkins,Hudson and other build tools

• Operational knowledge in Java, Apache, Tomcat, database

• Source control tools: Experience in at least one source control tool like GIT, CVS, Perforce, Subversion  etc. Experience in administrating one of these tools is a plus.

• Build scripting: Fluency in any of the build scripting languages like ANT, nANT, make, Maven,Gradle

• General scripting: Good knowledge of a scripting language like Perl, Python, Unix/Windows scripting is preferred

• Bug tracking Tools: Working experience of one defect tracking system (Bugzilla, Jira,Mantis etc.)

• General understanding of the platform in which you work (J2EE,.Net)..

Locating Files by “find” Command

The find command is used to locate files on a Unix or Linux system.  find will search any set of directories you specify for files that match the supplied search criteria.  You can search for files by name, owner, group, type, permissions, date, and other criteria.  The search is recursive in that it will search all subdirectories too. 

 The syntax looks like this:  find where-to-look criteria what-to-do

You can also find files with various permissions set.  “‑perm /permissions” means to find files with any of the specified permissions on, “‑perm -permissions” means to find files with all of the specified permissions on, and “‑perm permissions” means to find files with exactly permissions.  Permissions can be specified either symbolically (preferred) or with an octal number.  



The following will locate files that are writable by “others” (including symlinks, which should be writable by all):          find . -perm -o=w




A common request is a way to find all the hard links to some file.  Using “ls ‑li file” will tell you how many hard links the file has, and the inode number.  You can locate all pathnames to this file with:
  find mount-point -xdev -inum inode-number

When specifying time with find options such as ‑mmin (minutes) or ‑mtime (24 hour periods, starting from now), you can specify a number “n” to mean exactly n, “‑n” to mean less than n, and “+n” to mean more than n.





Fractional 24-hour periods are truncated!  That means that “find ‑mtime +1” says to match files modified two or more days ago.







For example:
find . -mtime 0   # find files modified between now and 1 day ago
                  # (i.e., within the past 24 hours)
find . -mtime -1  # find files modified less than 1 day ago
                  # (i.e., within the past 24 hours, as before)
find . -mtime 1   # find files modified between 24 and 48 hours ago
find . -mtime +1  # find files modified more than 48 hours ago

find . -mmin +5 -mmin -10 # find files modified between
                          # 6 and 9 minutes ago

Responsibilities of the Linux Admin

Main Responsibilities of the Linux System Administrator

1. Maintaining User Accounts
2. Security
3. Managing the disk space and Backup

The Administrator’s Privileges:

1. Change the contents or attributes of any file like its permission and ownership
2. He can delete any file with rm even if it is write-protected.
3. Initiate or Kill Any Process
4. Change any user’s password without knowing the existing one
5. Set the system clock with date
6. Limit the maximum size of files that users are permitted to create with ulimlit
7. Controls users access to the scheduling services like at and cron
8. Control users access to many networking services like Telnet, FTP, etc

Commands:

# su  :- Acquire  Superuser after entering password but home directory does not change

date – Setting the system Date         # date MMDDhhmm

wall: Communicating with users  wall: Communicating with users

# wall
   The machine will be shutdown today at 14 30
   The Back up will be at 12:30  [Ctrl-d]   

ulimit: Setting Limits on File Size    ulimit: Setting Limits on File Size

passwd – Changing any Password    # passwd amit

Most administrative commands are resident in /sbin and /usr/sbin
Linux uses Bash for normal and system administrative operations. Using Korn or C shell, all command may not work.
# in prompt indicates that the login is from root user.

input/output for C-shell (See man csh)

|
pipe output to input
>
redirect output to a storage file
<
redirect input from a storage file
>>
append redirected output to a storage file
tee
copy input to both file and next program in pipe
script
make file record of all terminal activity



Email and communication

msgs
read system bulletin board messages
mailx
send/read email; can be run by other programs to send exisiting files via email


Editors and formatting utilities

sed
programmable text editor for data streams
vi
full-featured editor for character terminals
emacs
GNU emacs editor for character terminals
xemacs
GNU emacs editor for X Window terminals
pico
very simple text editor


Terminal control characters for C-shell (csh)

^h, backspace
erase previously typed character
^u
erase entire line of input so far typed
^d
end-of-input for programs reading from terminal
^s
suspend writing to terminal (freezes terminal)
^q
continue writing to terminal
^z
suspend currently running job; restart with bg or fg
^c
kill currently running program and allow clean-up before exiting
^\
emergency kill of currently running program with no chance of cleanup


Information

date
show date and time
history
list of previously executed commands
man
show online documentation by program name
info
online documentation for GNU programs
w, who
who is on the system and what they are doing
whoami
who is logged onto this terminal
top
show system stats and top CPU using processes
uptime
show one line summary of system status


File management

cat
combine files
cp
copy files
ls
list files in a directory and their attributes
mv
change file name or directory location
rm
remove files
ln
create another link (name) to a file
chmod
set file permissions
crypt
encode/decode a file with a private key
gzip, gunzip
compress/decompress a file
find
find files that match specific criteria


Display contents of files

cat
copy files to display device
more
show text file on display terminal with paging control
head
show first few lines of a file(s)
tail
show last few lines of a file; or reverse line order
vi
full-featured screen editor for modifying text files
pico
simple screen editor for modifying text files
grep
display lines that match a pattern
lpr
send file to printer
pr
format file with page headers, multiple columns, etc.
diff
compare two files and show differences
cmp
compare two binary files and report if different
comm
compare two files; show common or unique lines
od
display binary files as eqivalent octal/hex codes
strings
show printable text embedded in binary files
file
examine file(s) and guess type: text, data, program, etc.
wc
count characters, words, and lines in a file



Controlling program execution for C-shell (See man csh)

&
run job in background
^c
kill job in foreground
^z
suspend job in foreground
fg
restart suspended job in foreground
bg
run suspended job in background
;
delimit commands on same line
()
group commands on same line
!
re-run earlier commands from history list
jobs
list current jobs
ps
show process information
kill
kill background job or previous process
nice
run program at lower priority
at
run program at a later time
crontab
run program at specified intervals
limit
see or set resource limits for programs
alias
create alias name for program (normally used in .login file)
sh, csh
execute command file


Interpreted languages and data manipulation utilities

sed
programmable text editor for data streams
awk
pattern scanning and processing language
perl
Practical Extraction and Report Language
sort
sort or merge lines in a file(s) by specified fields
tr
translate characters
cut
cut out columns from a file
paste
paste columns into a file
dd
copy data between devices; reblock; convert EBCDIC



Networking/communications

ssh
remote login/command execution; encrypted
scp
remote non-interactive file copy; encrypted
sftp
remote interactive file copy; encrypted
telnet
remote network login; plain text password - not recommended
ftp
network file transfer program; plain text password - not recommended
host
find IP address for given host name, or vice versa
lynx
web browser for character based (text-only) terminals
gzip, gunzip
compress/decompress a file
tar
combine multiple files/dirs into single archive


VI/VIM editor Commands




what will happen ?


                             "man ls |col -b > " will give the text file same as man ls.

what is VI-editor ?


      While in vi you can run AIX commands without exiting the editing session. The! creates a
    shell to execute the command that follows.
    1. :!ls will create a shell
    2. All files in the current directory are listed. Press return to exit the shell and return to the
    vi session or...
    3. While still in command mode, issue the :r snacks command
    4. The contents of snacks, in this case, are read into the vi file. By default, it will appear
    after the current line.
    If you need to run a series of commands without returning to vi after the first command is
    executed, enter :sh. When you have run all the commands, press to exit the shell
    and return to vi.


VI Options:-


                       vi has many modes of operation. Some of these will affect the way text is presented, while
others will make editing easier for novice users.
    :set all display all settings
    :set display settings different than the default
    :set ai sets autoindent on
    :set noai turns autoindent mode off
    :set nu enables line numbers
    :set nonu turns line numbers off
    :set list displays non-printable characters
    :set nolist hides non-printable characters
    :set showmode shows the current mode of operation
    :set noshowmode hides mode of operation
    :set ts=4 sets tabs to 4-character jumps
    :set ic ignores case sensitivity
    :set noic case sensitive

Search


    /word     Search “word” from top to bottom
    ?word     Search “word” from bottom to top
    /jo[ha]n     Search “john” or “joan”
    /\< the     Search “the”, “theatre” or “then”
    /the\>     Search “the” or “breathe”
    /\     Search “the”
    /\     Search all words of 4 letters
    /\/     Search “fred” but not “alfred” or “frederick”
    /fred\|joe     Search “fred” or “joe”
    /\     Search exactly 4 digits
    /^\n\{3}     Find 3 empty lines
    :bufdo /searchstr/     Search in all open files

Replace


    :%s/old/new/g     Replace all occurences of “old” by “new” in file
    :%s/old/new/gw     Replace all occurences with confirmation
    :2,35s/old/new/g     Replace all occurences between lines 2 and 35
    :5,$s/old/new/g     Replace all occurences from line 5 to EOF
    :%s/^/hello/g     Replace the begining of each line by “hello”
    :%s/$/Harry/g     Replace the end of each line by “Harry”
    :%s/onward/forward/gi     Replace “onward” by “forward” , case unsensitive
    :%s/ *$//g     Delete all white spaces
    :g/string/d     Delete all lines containing “string”
    :v/string/d     Delete all lines containing which didn't contain “string”
    :s/Bill/Steve/     Replace the first occurence of “Bill” by “Steve” in current line
    :s/Bill/Steve/g     Replace “Bill” by “Steve” in current line
    :%s/Bill/Steve/g     Replace “Bill” by “Steve” in all the file
    :%s/\r//g     Delete DOS carriage returns (^M)
    :%s/\r/\r/g     Transform DOS carriage returns in returns
    :%s#]\+>##g     Delete HTML tags but keeps text
    :%s/^\(.*\)\n\1$/\1/     Delete lines which appears twice
    Ctrl+a     Increment number under the cursor
    Ctrl+x     Decrement number under cursor
    ggVGg?     Change text to Rot13



Case


    Vu     Lowercase line
    VU     Uppercase line
    g~~     Invert case
    vEU     Switch word to uppercase
    vE~     Modify word case
    ggguG     Set all text to lowercase
    :set ignorecase     Ignore case in searches
    :set smartcase     Ignore case in searches excepted if an uppercase letter is used
    :%s/\<./\u&/g     Sets first letter of each word to uppercase
    :%s/\<./\l&/g     Sets first letter of each word to lowercase
    :%s/.*/\u&     Sets first letter of each line to uppercase
    :%s/.*/\l&     Sets first letter of each line to lowercase

Read/Write files


    :1,10 w outfile     Saves lines 1 to 10 in outfile
    :1,10 w >> outfile     Appends lines 1 to 10 to outfile
    :r infile     Insert the content of infile
    :23r infile     Insert the content of infile under line 23

File explorer


    :e .     Open integrated file explorer
    :Sex     Split window and open integrated file explorer
    :browse e     Graphical file explorer
    :ls     List buffers
    :cd ..     Move to parent directory
    :args     List files
    :args *.php     Open file list
    :grep expression *.php     Returns a list of .php files contening expression
    gf     Open file name under cursor

Interact with Unix


    :!pwd     Execute the “pwd” unix command, then returns to Vi
    !!pwd     Execute the “pwd” unix command and insert output in file
    :sh     Temporary returns to Unix
    $exit     Retourns to Vi

Alignment


    :%!fmt     Align all lines
    !}fmt     Align all lines at the current position
    5!!fmt     Align the next 5 lines

Tabs


    :tabnew     Creates a new tab
    gt     Show next tab
    :tabfirst     Show first tab
    :tablast     Show last tab
    :tabm n(position)     Rearrange tabs
    :tabdo %s/foo/bar/g     Execute a command in all tabs
    :tab ball     Puts all open files in tabs

Window spliting


    :e filename     Edit filename in current window
    :split filename     Split the window and open filename
    ctrl-w up arrow     Puts cursor in top window
    ctrl-w ctrl-w     Puts cursor in next window
    ctrl-w_     Maximise current window
    ctrl-w=     Gives the same size to all windows
    10 ctrl-w+     Add 10 lines to current window
    :vsplit file     Split window vertically
    :sview file     Same as :split in readonly mode
    :hide     Close current window
    :only     Close all windows, excepted current
    :b 2     Open #2 in this window

Auto-completion


    Ctrl+n Ctrl+p (in insert mode)     Complete word
    Ctrl+x Ctrl+l     Complete line
    :set dictionary=dict     Define dict as a dictionnary
    Ctrl+x Ctrl+k     Complete with dictionnary

Marks


    mk     Marks current position as k
    ‘k     Moves cursor to mark k
    d’k     Delete all until mark k
Abbreviations
    :ab pr printf("This is a Demo Ver \n");    Define pr as abbreviation of printf("This is a Demo Ver \n");

Text indent


    :set autoindent     Turn on auto-indent
    :set smartindent     Turn on intelligent auto-indent
    :set shiftwidth=4     Defines 4 spaces as indent size
    ctrl-t, ctrl-d     Indent/un-indent in insert mode
    >>     Indent
    <<     Un-indent

Syntax highlighting


    :syntax on     Turn on syntax highlighting
    :syntax off     Turn off syntax highlighting
    :set syntax=perl     Force syntax highlighting    

How to Exit


    :q[uit]     Quit Vim. This fails when changes have been made.
    :q[uit]!     Quit without writing.
    :cq[uit]     Quit always, without writing.
    :wq     Write the current file and exit.
    :wq!     Write the current file and exit always.
    :wq {file}     Write to {file}. Exit if not editing the last
    :wq! {file}     Write to {file} and exit always.
    :[range]wq[!]     [file] Same as above, but only write the lines in [range].
    ZZ     Write current file, if modified, and exit.
    ZQ     Quit current file and exit (same as ":q!").

Editing a File


    :e[dit]     Edit the current file. This is useful to re-edit the current file, when it has been changed outside                 of Vim.
    :e[dit]!     Edit the current file always. Discard any changes to the current buffer. This is useful if you want                 to start all over again.
    :e[dit] {file}     Edit {file}.
    :e[dit]! {file}     Edit {file} always. Discard any changes to the current buffer.
    gf     Edit the file whose name is under or after the cursor. Mnemonic: "goto file".

Inserting Text


    a     Append text after the cursor [count] times.
    A     Append text at the end of the line [count] times.
    i     Insert text before the cursor [count] times.
    I     Insert text before the first non-blank in the line [count] times.
    gI     Insert text in column 1 [count] times.
    o     Begin a new line below the cursor and insert text, repeat [count] times.
    O     Begin a new line above the cursor and insert text, repeat [count] times.
    

Inserting a file


    :r[ead] [name]     Insert the file [name] below the cursor.
    :r[ead] !{cmd}     Execute {cmd} and insert its standard output below the cursor.