Major Machine Learning Techniques

Major Machine Learning Techniques

* Regression/Estimation
-> Predicting continuous values
Algorithms: Linear Regression, Non-Linear Regression, Multiple Linear Regression

* Classification
-> Predicting the item class/category of a case
Algorithms: K-Nearest Neighbours, Decision Trees, Logistic Regression, Support Vector Machine

* Clustering
-> Finding the structure of data; summarization
Algorithms: k-Means Clustering, Hierarchical Clustering, Density-based Clustering

* Associations
-> Assoicating frequent co-occurring items/events

* Anomaly Detection
-> Discovering abnormal and unusual cases

* Sequence Mining
-> Predicting next events; click-stream (Markov Model, HMM)

* Dimension Reduction
-> Reducing the size of data (PCA)

* Recommendation Systems
-> Recommending items
Algorithms: Content-based Recommendation Engines, Collaborative Filtering

Data Science Methodology

Data Science Methodology

1. Business Understanding
2. Analytic Approach
3. Data Requirements
4. Data Collection
5. Data Understanding
6. Data Preparation
7. Modelling
8. Evaluation
9. Deployment
10. Feedback

Docker: Dockerfile

===Dockerfile===

FROM ubuntu:15.04
COPY . /app
RUN make /app
CMD python /app/app.py

use .dockerignore

===Working with Instructions===

sudo yum install git -y

mkdir docker_images

cd docker_images

mkdir weather-app
cd weather-app

git clone https://github.com/linuxacademy/content-weather-app.git src

vi Dockerfile

FROM node
LABEL org.label-schema.version=v1.1
RUN mkdir -p /var/node
ADD src/ /var/node
WORKDIR /var/node
RUN npm install
EXPOSE 3000
CMD ./bin/www

docker image build -t linuxacademy/weather-app:v1 .

docker image ls

docker container run -d --name weather-app1 -p 8081:3000 linuxacademy/weather-app:v1

docker container ls

curl localhost:8081

===Environment Variables===

cd docker_images
mkdir env
cd env

git clone https://github.com/linuxacademy/content-weather-app.git src

vi Dockerfile

FROM node
LABEL org.label-schema.version=v1.1
ENV NODE_ENV="development"
ENV PORT 3000

RUN mkdir -p /var/node
ADD src/ /var/node/
WORKDIR /var/node
RUN npm install
EXPOSE $PORT
CMD ./bin/www

docker image build -t linuxacademy/weather-app:v2 .

docker image ls

docker image inspect linuxacademy/weather-app:v2

docker container run -d --name weather-dev -p 8082:3001 --env PORT=3001 linuxacademy/weather-app:v2

docker container ls

curl localhost:8082

docker container inspect weather-dev

docker container run -d --name weather-app2 -p 8083:3001 --env PORT=3001 --env NODE_ENV=production linuxacademy/weather-app:v2

docker container inspect weather-app2


curl localhost:8083

docker container logs weather-app2

docker container run -d --name weather-prod -p 8084:3000 --env NODE_ENV=production linuxacademy/weather-app:v2

docker container logs weather-prod

curl localhost:8084

===Build Args===

cd docker_images
mkdir args
cd args

git clone https://github.com/linuxacademy/content-weather-app.git src

vi Dockerfile

FROM node
LABEL org.label-schema.version=v1.1
ARG SRC_DIR=/var/node

RUN mkdir -p $SRC_DIR
ADD src/ $SRC_DIR
WORKDIR $SRC_DIR
RUN npm install
EXPOSE 3000
CMD ./bin/www

docker image build -t linuxacademy/weather-app:v3 --build-arg SRC_DIR=/var/code .

docker image inspect linuxacademy/weather-app:v3 | grep WorkingDir

docker container run -d --name weather-app3 -p 8085:3000 linuxacademy/weather-app:v3

curl localhost:8085


===Working with Non-privileged User===

cd docker_images
mkdir non-privileged-user
cd non-privileged-user

vi Dockerfile

FROM centos:latest
RUN useradd -ms /bin/bash cloud_user
USER cloud_user

docker image build -t centos7/nonroot:v1 .

docker container run -it --name test-build centos7/nonroot:v1 /bin/bash

bash$ sudo su
bash$ su -
bash$ exit

docker container ls

docker container start test-build

docker container exec -u 0 -it test-build /bin/bash

bash$ whoami
bash$ exit

cd ~/docker_images
mkdir node-non-privileged-user
cd node-non-privileged-user

vi Dockerfile

FROM node
LABEL org.label-schema.version=v1.1
RUN useradd -ms /bin/bash node_user
USER node_user
ADD src/ /home/node_user
WORKDIR /home/node_user
RUN npm install
EXPOSE 3000
CMD ./bin/www

git clone https://github.com/linuxacademy/content-weather-app.git src

docker image build -t linuxacademy/weather-app-nonroot:v1 .

docker container run -d --name weather-app-nonroot -p 8086:3000 linuxacademy/weather-app-nonroot:v1

curl localhost:8086

===Order of Execution===

cd docker_images
mkdir centos-conf
cd centos-conf

vi Dockerfile

FROM centos:latest
RUN mkdir -p ~/new-dir1
RUN useradd -ms /bin/bash cloud_user
RUN mkdir -p /etc/myconf
RUN echo "Some config data" >> /etc/myconf/my.conf
USER cloud_user
RUN mkdir -p ~/new-dir2

docker image build -t centos7/myconf:v1 .

===Using the Volume Instruction===

cd docker_images
mkdir volumes
cd volumes

vi Dockerfile

FROM nginx:latest
VOLUME ["/usr/share/nginx/html/"]

docker image build -t linuxacademy/nginx:v1 .

docker container run -d --name nginx-volume linuxacademy/nginx:v1

docker container inspect nginx-volume

docker volume inspect volume-id

sudo ls -la /var/lib/docker/volumes/volume-id/_data

Docker: Container Logging

docker container run --name weather-app -d -p 80:3000 linuxacademycontent/weather-app

docker container ls

docker container logs container_id

docker container logs container_id

docker container run -d --name ghost_blog \
-e database__client=mysql \
-e database__connection_host=mysql \
-e database__connection_user=root \
-e database__connection_password=password \
-e database__connection_database=ghost \
-p 8080:2368 \
ghost:1-alpine

docker container ls

docker container ls -a

docker container logs container_id

===Summary===

Create a container using the weather-app image.

docker container run --name weather-app -d -p 80:3000 linuxacademycontent/weather-app

Show information logged by a running container:

docker container logs [NAME]

Show information logged by all containers participating in a service:

docker service logs [SERVICE]

Logs need to be output to STDOUT and STDERR.

Nginx Example:

RUN ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log

Debug a failed container deploy:

docker container run -d --name ghost_blog \
-e database__client=mysql \
-e database__connection__host=mysql \
-e database__connection__user=root \
-e database__connection__password=P4sSw0rd0! \
-e database__connection__database=ghost \
-p 8080:2368 \
ghost:1-alpine

Docker: Executing Container Commands

docker container run -d nginx

docker container ls

docker container run -it nginx /bin/bash

# nginx -g 'daemon off;'

docker container ls

docker container inspect container_id

curl 172.17.0.3


# exit

docker container ls

docker container ls -a

docker container exec -it container_id ls /usr/share/nginx/html
docker container exec -it container_id /bin/bash

# apt-get update -y
# exit

docker container prune

docker container rm -f container_id

===Summary===

Executing a command:

• Dockerfile
• During a Docker run
• Using the exec command

Commands can be:

• One and done Commands
• Long running Commands

Start a container with a command:

docker container run [IMAGE] [CMD]

Execute a command on a container:

docker container exec -it [NAME] [CMD]

Example:

docker container run -d -p 8080:80 nginx
docker container ps
docker container exec -it [NAME] /bin/bash
docker container exec -it [NAME] ls /usr/share/nginx/html/

Docker: Exposing and Publishing Container Ports

===Exposing Container Ports===

docker container run -d nginx

docker container ls

curl localhost

docker inspect container container_id

curl 172.17.0.2

docker container run -d --expose 3000 nginx

docker container ls

docker rm -f container_id

docker container run -d --expose 3000 -p 80:3000 nginx

docker container ls

curl localhost:3000
connection refused

docker container rm -f container_id


docker container run -d --expose 3000 -p 8080:80 nginx

curl localhost:8080

docker container run -d -p 8081:80/tcp -p 8081:80/udp nginx


curl localhost:8081

docker container run -d -P nginx

docker container ls

curl localhost:32768

docker container port container_id

===Summary===

Exposing:

• Expose a port or a range of ports
• This does not publish the port
• Use --expose [PORT]

docker container run --expose 1234 [IMAGE]

Publishing:

• Maps a container's port to a host`s port
• -p or --publish publishes a container's port(s) to the host
• -P, or --publish-all publishes all exposed ports to random ports

docker container run -p [HOST_PORT]:[CONTAINER_PORT] [IMAGE]
docker container run -p [HOST_PORT]:[CONTAINER_PORT]/tcp -p [HOST_PORT]:[CONTAINER_PORT]/udp [IMAGE]
docker container run -P

Lists all port mappings or a specific mapping for a container:

docker container port [Container_NAME]

Docker - Creating Containers

docker container run --help

docker container run busybox

docker container ls

docker container ls -a

docker container run --rm busybox

docker container ls -a

docker container run nginx

docker container run -d nginx

docker container ls

docker container ls -a

docker container run -it busybox

# ls
# exit

docker container prune -f

docker container run --name my_busybox busybox

docker container ls -a

===Summary===

docker container run:

• --help Print usage
• --rm Automatically remove the container when it exits
• -d, --detach Run container in background and print container ID
• -i, --interactive Keep STDIN open even if not attached
• --name string Assign a name to the container
• -p, --publish list Publish a container's port(s) to the host
• -t, --tty Allocate a pseudo-TTY
• -v, --volume list Mount a volume (the bind type of mount)
• --mount mount Attach a filesystem mount to the container
• --network string Connect a container to a network (default "default")

Create a container and attach to it:

docker container run –it busybox

Create a container and run it in the background:

docker container run –d nginx

Create a container that you name and run it in the background:

docker container run –d –name myContainer busybox

Docker Commands

docker -h | more

docker image -h

docker image ls -h

docker image ls

docker image pull nginx


docker image ls

docker image inspect image_id

docker container -h

docker container ls

docker container run busybox

docker container ls

docker container ls -a

docker container run -P -d nginx

docker container ps

docker container inspect container_id

curl http://172.17.0.2

docker container inspect container_id

docker container top container_id

docker container ls
docker container attach container_id

docker container ls

docker container ls -a

docker container start container_id

docker container ls

docker container stop container_id
docker container start container_id

docker container logs container_id

docker container ls

curl localhost:32774

docker container logs container_id

docker container ls

docker container stats container_id

docker container exec -it container_id /bin/bash

# ls
# ls /usr/share/nginx/html/
# exit

docker container ls

docker container exec -it container_id ls /usr/share/nginx/html/

docker container ls

docker container pause container_id

docker container ls

docker container unpause container_id


docker container ls -a

docker container rm -f container_id

docker container ls -a

docker container prune

docker container ls -a

docker container prune -h

docker container prune -f

===Summary===

Get a list of all of the Docker commands:

docker -h

Management command were introduced in Docker engine v1.13

Management Commands:

• builder Manage builds
• config Manage Docker configs
• container Manage containers
• engine Manage the docker engine
• image Manage images
• network Manage networks
• node Manage Swarm nodes
• plugin Manage plugins
• secret Manage Docker secrets
• service Manage services
• stack Manage Docker stacks
• swarm Manage Swarm
• system Manage Docker
• trust Manage trust on Docker images
• volume Manage volumes

docker image:

• build Build an image from a dockerfile
• history Show the history of an image
• import Import the contents from a tarball to create a filesystem image
• inspect Display detailed information on one or more images
• load Load an image from a tar file or STDIN
• ls List images
• prune Remove unused images
• pull Pull an image or a repository from a registry
• push Push an image or a repository to a registry
• rm Remove one or more images
• save Save one or more images to a tar file (streamed to STDOUT by default)
• tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE

docker container:

• attach Attach local standard input, output, and error streams to a running container
• commit Create a new image from a container's changes
• cp Copy files/folders between a container and the local filesystem
• create Create a new container
• diff Inspect changes to files or directories on a container's filesystem
• exec Run a command in a running container
• export Export a container's filesystem as a tar archive
• inspect Display detailed information on one or more containers
• kill Kill one or more running containers
• logs Fetch the logs of a container
• ls List containers
• pause Pause all processes within one or more containers
• port List port mappings or a specific mapping for the container
• prune Remove all stopped containers
• rename Rename a container
• restart Restart one or more containers
• rm Remove one or more containers
• run Run a command in a new container
• start Start one or more stopped containers
• stats Display a live stream of container(s) resource usage statistics
• stop Stop one or more running containers
• top Display the running processes of a container
• unpause Unpause all processes within one or more containers
• update Update configuration of one or more containers
• wait Block until one or more containers stop, then print their exit codes