Accessing localhost from a Docker Container using native Docker for Mac

Ever had a need to access something from within a Docker container that runs on the host system?

When using native Docker on OSX you have bad luck. When configuring a container and pointing that to localhost will result in the fact the your software will be targeted at the localhost of the docker container.

A solution for this isto define a new local loopback to your localhost

$ sudo ifconfig lo0 alias 172.16.123.1

This will define a loopback network interface that points to your localhost. When you need to access the localhost you can use this ip.

Terraform to the rescue

Getting exposed to Amazon Web Services is fun! Certainly when you see that the infrastructure is growing and supporting the daily need of developers and the business. You slowly start adding services and try to keep everything in a state so that it is repeatable and maintainable. At a certain moment it becomes clear that you need the concept of Infrastructure As Code.

The Amazon way of doing this is by using AWS CloudFormation. This enables you to define the infrastructure in a JSON/YAML format and apply the changes to the infrastructure.

Our team manages a bunch of environments using services like AWS ECS, EC2, ElasticSearch, RDS and more.. Maintaining this infrastructure in CloudFormation seemed the standard way of doing things until we started a proof-of-concept with Terraform.

Why did we start this proof-of-concept?? Mainly because the overwhelming pieces of code that we needed to maintain in CloudFormation became unmaintainable. The use of Terraform was so successful that we decide to rewrite our entire infrastructure codebase using Terraform.

The advantages when using Terraform are:

• less code to maintain because Terraform is less verbose
• when using Terraform an infrastructure change can be planned, this shows what is going to be changed before actually executing the change

$ terraform plan

See what the changes are and then when everything seems ok…

$ terraform apply

Currently we have our entire Infrastructure in Terraform and we could never be more happier. Terraform came to our rescue!

Functional Rest API Testing with Grails/Rest Client Builder

Functional Rest API testing with Grails is easy and fun. We will be creating a simple Rest Controller and test it using Spock and Rest Client Builder.

When running the functional test a real container will be started on a specific port and tests are run against the running container. Just as it should be.

Scenario: Performing a GET request on a url (http://localhost:8080/helloworld) should return a HTTP Status 200 and data with a json payload

{"message":"helloworld"}

So lets get started!

Create a Grails application

$ grails create-app RestHelloWorld

Update your build.gradle to include the Rest Client Builder dependencies which we will need later on

dependencies {
    // add the following line to the 'dependencies' section
    testCompile "org.grails:grails-datastore-rest-client:4.0.7.RELEASE"
}

Create an Integration Test

$ grails create-integration-test HelloWorld

Create a test method inside the integration test

Open up the created HelloWorldControllerSpec inside the /src/integration-test/groovy/resthelloworld/ package

package resthelloworld

import grails.test.mixin.integration.Integration
import grails.transaction.*
import spock.lang.*
import grails.plugins.rest.client.RestBuilder
import grails.plugins.rest.client.RestResponse

@Integration
@Rollback
class HelloWorldSpec extends Specification {

    def setup() {
    }

    def cleanup() {
    }

    def "Ask for a nice HelloWorld"() {
        given:
        RestBuilder rest = new RestBuilder()

        when:
        RestResponse response = rest.get("http://localhost:8080/helloworld/")

        then:
        response.status == 200

        and:
        response.json.message == "helloworld"
    }
}

Run your test

$ grails test-app

Offcourse this will fail as we do not have implement the controller yet.

Create a Rest controller

$ cd RestHelloWorld
$ grails create-controller HelloWorld

Note: The generation of the controller also create a Unit Test for the controller, default this test will fail. We are going to delete the generated Unit Test as we do not need it now. This test is located under the /src/test/ groovy package.

$ rm ./src/test/groovy/resthelloworld/HelloWorldControllerSpec.groovy

Implement the controller function that will return data

package resthelloworld

class HelloWorldController {

    def index() {
        render(status: 200, contentType: "application/json") {
            ["message" : "helloworld"]
        }
    }
}

Modify UrlMapping

In order to get our newly generated controller accessible via Rest we need to modify our UrlMappings.

class UrlMappings {

    static mappings = {
        "/$controller/$action?/$id?(.$format)?"{
            constraints {
                // apply constraints here
            }
        }

        "/"(view:"/index")
        "500"(view:'/error')
        "404"(view:'/notFound')

        // add the line below
        "/helloworld/"  (controller: "helloWorld", action: "index", method: "GET")
    }
}

Test your app

$ grails test-app

You should find that your tests are fine now :)

$ grails test-app
BUILD SUCCESSFUL

Total time: 2.054 secs
| Tests PASSED

Using DavMail Gateway as a mail proxy for Microsoft Exchange (OWA)

If you find yourself into a situation where you have a need for non Microsoft mail client that needs support for Microsoft Exchange then you are often out of luck. In my case I needed Exchange support for the terrific PostBox mail client.

As for now PostBox does not support Microsoft Exhange natively so the hunt starts on how to get Exchange working. As it stands most companies also enable Exchange Web Access (or Outlook Web Access [OWA]) so maybe we can use that to feed our native mail client.

Enter the use of DavMail!

Davmail Gateway

Davmail is a local mail proxy that can work together with Microsoft Exchange [OWA] in a way that DavMail is actually connecting to a Exchange OWA and your mail client connects to DavMail as a proxy.

Configure Davmail

In order to get DavMail working correctly you need to provide the correct settings so it can use the OWA endpoint.

Configure PostBox

In order to get PostBox working with DavMail you need to create an outgoing mail server and an account that will use that outgoing mailserver.

Configure PostBox - Outgoing mailserver

Configure PostBox - Account setup

Configure PostBox - Identity setup

Now you are ready to send mail using your PostBox Client using DavMail and OWA.

Cleaning Grails Domain Objects in a Spock Test

Spock is a nice framework to execute integration tests in your Grails application. It may happen that the Spock test actually creates some domain objects and you want to clean them out on everuy single run of your feature test methods.

Spock provides a setup() and cleanup() method.

When you want to remove your domain objects after each feature test has run you can execute the following:

def setup() { ... }

def cleanup() {
        // make sure to clear out the database on after test
        <YourDomainObject>.withNewSession {
            <YourDomainObject>.findAll().each { it.delete(flush: true) }
        }
}

We need the .withNewSession because there is no Hibernate session provided in the setup() and cleanup() methods.

Adding WebSocket/Stomp support to a Spring Boot application

Adding support for WebSockets / Stomp in a Spring Boot application has never been more easy. You can use WebSockets to receive serverside events or push data to the server using WebSockets.

The following example will enable a server to send messages to a WebSocket/Stomp client.

• Modify build.gradle

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.springframework.boot:spring-boot-starter-websocket")
    compile("org.springframework:spring-messaging")
    testCompile("junit:junit")
}

• Create a WebSocket configuration class that holds the configuration

@Configuration
@EnableWebSocketMessageBroker
public class WebSocketConfig extends AbstractWebSocketMessageBrokerConfigurer {

    @Override
    public void registerStompEndpoints(StompEndpointRegistry registry) {
        // the endpoint for websocket connections
        registry.addEndpoint("/stomp").withSockJS();
    }

    @Override
    public void configureMessageBroker(MessageBrokerRegistry config) {
        // use the /topic prefix for outgoing WebSocket communication
        config.enableSimpleBroker("/topic");

        // use the /app prefix for others
        config.setApplicationDestinationPrefixes("/app");
    }
}

Now a client that connects to /stomp endpoint is able to receive WebSocket messages.

• Create a service that is going to send the data to a WebSocket endpoint

@Service
public class ScheduleTask {

    @Autowired
    private SimpMessagingTemplate template;

    // this will send a message to an endpoint on which a client can subscribe
    @Scheduled(fixedRate = 5000)
    public void trigger() {
        // sends the message to /topic/message
        this.template.convertAndSend("/topic/message", "Date: " + new Date());
    }

}

• Create a client that is able to receive the message

<!DOCTYPE html>
<html>
<head>
    <title>WebSocket Stomp Receiving Example</title>
</head>
<body>
    <div>
        <h3>Messages:</h3>
        <ol id="messages"></ol>
    </div>

    <script type="text/javascript" src="//cdn.jsdelivr.net/jquery/1.11.2/jquery.min.js"></script>
    <script type="text/javascript" src="//cdn.jsdelivr.net/sockjs/0.3.4/sockjs.min.js"></script>
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/stomp.js/2.3.3/stomp.min.js"></script>

    <script type="text/javascript">
        $(document).ready(function() {
            var messageList = $("#messages");

            // defined a connection to a new socket endpoint
            var socket = new SockJS('/stomp');

            var stompClient = Stomp.over(socket);

            stompClient.connect({ }, function(frame) {
                // subscribe to the /topic/message endpoint
                stompClient.subscribe("/topic/message", function(data) {
                    var message = data.body;
                    messageList.append("<li>" + message + "</li>");
                });
            });
        });
    </script>
</body>
</html>

The whole example project can be downloaded https://github.com/mpas/spring-boot-websocket-stomp-server-send-example

Run a Grails 3 generated Fat Jar file in production mode

When creating a Grails WAR/JAR file using:

$ grails war

The resulting artifact can be run in production mode using:

$ java -Dgrails.env=prod -Dserver.port=9000 -jar <name-of-jar-file>.jar

Setting up Docker RabbitMQ with predefined users/vhosts

When creating an Docker image it is nice to have predefined users and vhosts without manually having to create them after the Docker RabbitMQ image has started.

The following is a Dockerfile that extends the default Docker RabbitMQ image including the Management Plugin and it creates a standard set of users / vhosts when the container is created from the image.

It involves a init.sh script that is used to create the users and vhosts.

The Docker File

FROM rabbitmq:3-management

# Add script to create default users / vhosts
ADD init.sh /init.sh

# Set correct executable permissions
RUN chmod +x /init.sh

# Define default command
CMD ["/init.sh"]

The init.sh script

#!/bin/sh

# Create Default RabbitMQ setup
( sleep 10 ; \

# Create users
# rabbitmqctl add_user <username> <password>
rabbitmqctl add_user test_user test_user ; \

# Set user rights
# rabbitmqctl set_user_tags <username> <tag>
rabbitmqctl set_user_tags test_user administrator ; \

# Create vhosts
# rabbitmqctl add_vhost <vhostname>
rabbitmqctl add_vhost dummy ; \

# Set vhost permissions
# rabbitmqctl set_permissions -p <vhostname> <username> ".*" ".*" ".*"
rabbitmqctl set_permissions -p dummy test_user ".*" ".*" ".*" ; \
) &    
rabbitmq-server $@

Place both of these files in a directory and build your image:

$ docker build -t my_rabbitmq_image .

Start a container based on the image using:

$ docker run --rm=true --name my_rabbitmq_container -p 5672:5672 -p 15672:15672  my_rabbitmq_image

Then in your browser navigate to http://localhost:15672 and see if all is ok!

Note: When using Boot2Docker make sure to replace the localhost with the correct IP.

HTTPServletRequestWrapper for ServletInputStream 3.1

A HttpServletRequestWrapper may be handy if you want to be able to read the HTTP Body multi times after you consume it in a filter. The ServletInputStream 3.1 changed a bit and the following methods have to be implemented.

• isFinished
• isReady
• setReadListener

import com.google.common.primitives.Bytes;
import javax.servlet.ReadListener;
import javax.servlet.ServletInputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;

public class AuthenticationRequestWrapper extends HttpServletRequestWrapper {

    // tag::variables[]
    private byte[] requestBody = new byte[0];
    private boolean bufferFilled = false;
    // end::variables[]

    /**
     - Constructs a request object wrapping the given request.
     *
     - @param request The request to wrap
     - @throws IllegalArgumentException if the request is null
     */
    public AuthenticationRequestWrapper(HttpServletRequest request) {
        super(request);
    }


    // tag::getRequestBody[]
    public byte[] getRequestBody() throws IOException {
        if (bufferFilled) {
            return Arrays.copyOf(requestBody, requestBody.length);
        }

        InputStream inputStream = super.getInputStream();

        byte[] buffer = new byte[102400]; // 100kb buffer

        int bytesRead;
        while ((bytesRead = inputStream.read(buffer)) != -1) {
            requestBody = Bytes.concat(this.requestBody, Arrays.copyOfRange(buffer, 0, bytesRead)); // <1>
        }

        bufferFilled = true;

        return requestBody;
    }
    // end::getRequestBody[]

    // tag::getInputStream[]
    @Override
    public ServletInputStream getInputStream() throws IOException {
        return new CustomServletInputStream(getRequestBody()); // <1>
    }
    // end::getInputStream[]

    private static class CustomServletInputStream extends ServletInputStream {

        private ByteArrayInputStream buffer;

        public CustomServletInputStream(byte[] contents) {
            this.buffer = new ByteArrayInputStream(contents);
        }

        @Override
        public int read() throws IOException {
            return buffer.read();
        }

        @Override
        public boolean isFinished() {
            return buffer.available() == 0;
        }

        @Override
        public boolean isReady() {
            return true;
        }

        @Override
        public void setReadListener(ReadListener listener) {
            throw new RuntimeException("Not implemented");
        }
    }
}

Installing Docker Registry 2.0.1 using self signed certificates

The new Docker Registry (2.x) has just been released and is rewritten in Go. The default installation now requires SSL security and I was looking for a way to secure the Registry using a NGINX SSL proxy where users need to provide username/password to access the registry. The setup of the NGINX proxy can be done manually but i decided to see if i can reuse the excellent images from Container Solutions to ease the installation.

So the setup will be that we install the Docker Registry and proxy the SSL user access via self signed certificates using an NGINX proxy image provided by Container Solutions. Check here for more information

Installation of the remote docker registry will be done by using on an Amazon EC2 (Linux AMI). Currently the free tier Amazon Linux AMI 2015.03 (HVM), SSD Volume Type - ami-a10897d6. So spin up the Amazon AMI and let’s install Docker.

Note: when you spin up your Amazon AMI make sure to remember the FQDN/DNS name! We need this name to generate the SSL certificates!

Example:
<domain-name> = ec2-52-16-247-220.eu-west-1.compute.amazonaws.com

So spin up your AMI and install Docker!

Installing docker

• login into your Amazon AMI
• update the system and install Docker

$ sudo yum update -y
$ sudo wget -qO- https://get.docker.com/ | sh

• add the ec2-user to the docker group (optional)

$ sudo usermod -aG docker ec2-user

• start Docker

$ sudo service docker start

• make sure Docker can run the basic “hello-world”

$ sudo docker run hello-world

Create Docker Registry data and configuration directories

We are going to store the registry image data inside /opt/docker/registry/data and configuration files such as the ssl certificates and user login inside /opt/docker/registry/conf.

• create data folders for Docker Registry data and configuration

$ sudo mkdir -p /opt/docker/registry/data
$ sudo mkdir -p /opt/docker/registry/conf

Run the Docker Registry

Now we are able to run the Docker Registry, the data for images that will be pushed are going to be stored in /opt/docker/registry/data and the container will be named docker-registry

• run the registry and name it docker-registry

$ sudo docker run -d -v /opt/docker/registry/data:/tmp/registry-dev \
--name docker-registry registry:2.0.1

• test if the registry is actually running

$ sudo docker ps

So now we have a running Docker Registry, but still no SSL proxy and no user accounts to get access to the registry.

Generate self signed certificates for our SSL proxy

The result of the certificate generation will be placed in /opt/docker/registry/conf and named docker-registry.crt and docker-registry.key.

Note: The docker-registry.crt file is important, we will need this later on to configure our local Docker client on the machine that is going to access the remote registry. So after generating the docker-registry.crt file, grab this and store it on your local machine in a place where you can find it.

• generate the certificates

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /opt/docker/registry/conf/docker-registry.key \
-out /opt/docker/registry/conf/docker-registry.crt

Accept all defaults and make sure you give the correct FQDN /DNS name = <domain-name>.

Create passwords for access to the Docker Registry

In order to let users login into the registry we need to create users (user1/user2). This will be done by using htpasswd. The user data will be stored in docker-registry.htpasswd file and placed in the /opt/docker/registry/conf directory.

• install htpasswd

$ sudo yum install httpd-tools -y

• create the users

$ sudo htpasswd -c /opt/docker/registry/conf/docker-registry.htpasswd user1
$ sudo htpasswd /opt/docker/registry/conf/docker-registry.htpasswd user2

Note: when creating the second user omit the -c otherwise the docker-registry.htpasswd file will be get overwritten!

Run the NGINX Proxy

As mentioned we are going to use the image from Container Solutions to run our NGINX proxy.

• start the NGINX proxy and name it docker-registry-proxy

$ sudo docker run -d -p 443:443  \
-e REGISTRY_HOST="docker-registry" -e REGISTRY_PORT="5000" -e SERVER_NAME="localhost" \
--link docker-registry:docker-registry \
-v /opt/docker/registry/conf/docker-registry.htpasswd:/etc/nginx/.htpasswd:ro \
-v /opt/docker/registry/conf:/etc/nginx/ssl:ro \
--name docker-registry-proxy containersol/docker-registry-proxy

After this step we have a running Docker Registry which is secured using Self Signed certificates and users are able to login using their username/password.

To test this navigate to your registry by using a browser (Chrome) and access: https://<domain-name>:443/v2/. After accepting the security warning provide a username/password and the output should be {}.

Configure the local Docker client

Now that we have a running secured Docker Registry we can configure the Docker client on our machine that is going to access the remote Registry. For this we need a copy of the earlier docker-registry.crt file.

• copy the docker-registry.crt file from our server to your local machine. This file is located in /opt/docker/registry/conf. Put the copy in a place where you can find it.

Ubuntu Docker Client

In order to get the local client working, we need to install Docker and register the docker-registry.crt certificate file!

• install docker

$ sudo wget -qO- https://get.docker.com/ | sh
$ sudo service docker start

• create a directory holding our extra certificates

$ sudo mkdir /usr/share/ca-certificates/extra

• copy the docker-registry.crt file to the directory /usr/share/ca-certificates/extra

• register the certificate

$ sudo dpkg-reconfigure ca-certificates

Now you are almost ready!

• restart the local Docker client

$ sudo service docker restart

• login onto your remote registry using

$ docker login <domain-name>:port

Now we have a remote Docker Registry and the Docker Client is able to connect!

Upgrading from Grails 2.3.8 to 2.4.2

When upgrading to Grails 2.4.2 i ran into an issue where the following error message would pop up.

Error creating bean with name 'grailsApplication' defined in ServletContext resource
[/WEB-INF/applicationContext.xml]: Cannot resolve reference to
bean 'grailsResourceLoader' while setting bean property 'grailsResourceLoader';

To solve this issue you need to delete some lines in the <grails-app>/web-app/WEB-INF/applicationContext.xml file.

Delete the following lines:

<property name="grailsResourceLoader" ref="grailsResourceLoader" />

<bean id="grailsResourceLoader" class="org.codehaus.groovy.grails.commons.GrailsResourceLoaderFactoryBean" />

And you should be up and running quickly.

Skip a contenttype/section to be renderend

When creating a layout it may occur that you want to skip rendering certain types of content, or render only specific content in a part of your layout.

Example: You want to only render contenttype ‘post’ use the following code in your template:

{{ if eq .Type "post" }}

    {{ .Title }}
    {{ .Content }}

{{ end }}