In today’s hybrid cloud work environment, it is very useful to be able to connect private or on- premises cloud with public cloud resources. The Secure Gateway provides a secure, continuous channel in order to securely integrate these serverless functions with the resources that are private or on the premises, with granular resource access to ensure security is maintained.
IBM Cloud
IBM Cloud is a full stack web platform that comes with an AI and data analytics tool suite. It also comes with a host of features to allow easy migration and integration. The services cover intelligent business analysis, deep data, and support for IoT, AI and blockchain. It is easily scalable for global deployment. Overall, it provides more space and security by dint of all the information being in the cloud, which frees up valuable space on the hardware and also allows easy modification for scalability.
Secure Gateway
The IBM Secure Gateway service provides a prompt and secure channel for connecting protected resources to cloud resources. The Secure Gateway Client can be natively installed. On deployment, a secure, continuous connection can be established between the protected environment and the
cloud via an outbound call. The Secure Gateway works to provide granular access to pre-defined resources, as opposed to a traditional VPN that connects environments at the network level and then limits access from the top down.
Use Case
Many organizations today choose to work within a hybrid cloud environment where they depend on both private and public cloud services. There is a likelihood of having a secure database on the premises for sensitive or company confidential data, and using public cloud storage for web apps
and static files. The public cloud option is becoming more accepted by businesses globally, but many still choose to retain sensitive information privately. The challenge is in being able to access the private database from the public cloud.
The Secure Gateway Client allows for an easy set of a secure connection between the private database and public cloud, while limiting access due to its granular access capabilities. This code pattern works by using Docker to locally deploy CouchDB and Minio as the resources on premise, i.e. for the private database.
Development and Process Description
Serverless functions can access private resources securely by using the IBM Secure Gateway. The serverless functions can be triggered by events or REST APIs. This code pattern has a sample application which demonstrates IBM Cloud functions, based on Apache OpenWhisk, which can interact with on-premises resources using the Secure Gateway client. The code pattern uses a local deployment of CouchDB, which can then be accessed by the serverless functions. It also uses Minio open source object storage server. The code pattern is served by a local Node.js web application. The local resources are hence exposed through the Secure Gateway channel.
CouchDB serverless functions use the OpenWhisk Cloudant package, as Cloudant is based on CouchDB and hence has similar APIs. Database changes trigger an action, and here, the action output is the document ID. You can also run actions through REST APIs. All actions interact with a local Node.js web app which uses object storage. The actions create a bucket and an object from any given URL. All these actions use the IBM Secure Gateway channel.
Process Flow
The application demonstrates IBM Cloud Functions (based on Apache OpenWhisk) that interacts with on-premise resources by using Secure Gateway. The use case demonstrates how you can allow IBM Cloud Functions access to your on-premise resources such as databases and web applications. This code pattern uses a local CouchDB deployment that will be accessed by the serverless functions. This code pattern will also use a local Minio, an open source object storage server, and will be served through a local Node.js web application. The local resources will be exposed through a Secure Gateway tunnel.
The serverless functions for CouchDB will utilize the existing OpenWhisk’s Cloudant package. This is possible since Cloudant is based on CouchDB and has almost similar APIs. An action will be triggered by changes in the database. In this case, the action will output the document ID. Other actions such as creating and reading a document can be invoked through the wskcommand. Another scenario is executing actions via REST APIs. The actions will interact with the local Node.js web application that’s using an object storage. The actions will create a bucket and an object from a URL. All the actions will utilize the Secure Gateway tunnel.
When the reader has completed this Code Pattern, they will understand how to:
- Allow Cloud Functions to access on-premise resources
- Create a Secure Gateway tunnel that will be used by serverless functions
Flow
-
The user inserts a document on the on-premise database
-
A trigger is fired once the document was inserted in the database
-
An action connected to the trigger is invoked. The action simply prints the document ID of the inserted document
-
The user can also interact with the on-prem database via the actions available in the cloudant package
-
The user can also interact with serverless functions via REST APIs by using API Gateway that is easily integrated with IBM Cloud Functions.
-
The actions are doing http requests to the available cloud host that is exposed by the Secure gateway service.
-
All the interaction with the on-prem database goes through a tunnel via the Secure Gateway service that is also installed in the on-prem environment.
Steps
- Clone the repo
- Create Secure Gateway service with IBM Cloud
- Run CouchDB and Minio with Docker
- Deploy IBM Cloud Functions
- Test with database events
- Test with REST API to access on-prem webapp
- Clone the repo
Clone this repo locally and cd into the repo’s directory. In a terminal, run:
$ git clone https://github.com/IBM/ibm-cloud-functions-with-on-prem-resources
- Create Secure Gateway service with IBM Cloud
Create a Secure Gateway instance:
Secure Gateway
Then follow the instructions here to properly setup the connection to your on-premise resources:
Setting up Secure Gateway connection
- Run CouchDB and Minio with Docker
Run couchdb and minio in your host machine. Docker is one way of easily deploying it.
$ docker run -p 5984:5984 -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -- name=couchdb-test -d couchdb $ docker run -p 9000:9000 --name minio-test \ -e "MINIO_ACCESS_KEY=1234" \ -e "MINIO_SECRET_KEY=12345678" \ -v /tmp/minio-data:/data \ -v /tmp/minio-config:/root/.minio \ minio/minio server /data
Create a database named testdb in couchdb. Access the dashboard on
http://localhost:5984/_utils/ and enter admin and password
- Deploy IBM Cloud Functions
First, export the Cloud Host in an environment variable that you have from your Secure Gateway destination. It would look something like:
cap-us-east-prd-sg-bm-
03.integration.ibmcloud.com:15006
$ export CLOUD_HOST=”cap-us-east-prd-sg-bm-03.integration.ibmcloud.com:15006″
Then deploy the set of IBM Cloud Function resources with wskdeploy $ wskdeploy -m manifest-couchdb.yaml $ wskdeploy -m manifest-webapp-minio.yaml
- Test with database events
Insert a document in couchdb using the dashboard on http://localhost:5984/_utils/. Insert a simple document
{
"_id": "sample-document-1"
}
Verify that an action is triggered by the event of inserting a document in the dashboard https://cloud.ibm.com/openwhisk/dashboard
You can also invoke available actions from the Cloudant package. Try invoking the create-document action that will create a document in your local couchdb deployment.
$ ibmcloud wsk action invoke couchdb-binding/create-document --param dbname
testdb
--param doc '{"_id":"sample-document-1"}' --result
{
"id": "sample-document-1", "ok": true,
"rev": "1-967a00dff5e02add41819138abb3284d"
}
Read your recently created document with read-document action.
$ ibmcloud wsk action invoke couchdb-binding/read-document --param dbname
testdb -
-param docid sample-document-1 --result
{
"_id": "sample-document-1",
"_rev": "1-967a00dff5e02add41819138abb3284d"
}
- Test with REST API to access on-prem webapp
Cloud Functions can also be executed via REST APIs. The action can do http requests through the Secure Gateway’s cloud host destination that would allow interaction to an on-prem web application.
The web application is a simple API server that is interfaced with a local object storage deployment. Deploy the web app using nodejs.
$ cd containers/webapp-minio $ npm install $ node app.js
Edit the destination for your Secure Gateway to the port 8080 which is the web application
Get the list of the current APIs’ URL created for your actions
$ ibmcloud wsk api list sample-api-gateway-create
# Result #
Action Verb API Name URL /interact-with-on-pre post sample-api-gateway-create https://service.us.apiconnect.ibmcloud.com/gws/apigateway/api/<>/create/object /interact-with-on-pre post sample-api-gateway-create https://service.us.apiconnect.ibmcloud.com/gws/apigateway/api/<>/create/bucket
Create a bucket for your on-prem object storage deployment using the Cloud Function’s exposed
REST APIs. Use the .../create/bucket endpoint.
$ export CF_API='https://service.us.apiconnect.ibmcloud.com/gws/apigateway/
api/
<>/create/bu cket'
$ curl -X POST ''"$CF_API"'?bucketName=sample-bucket'
{"status":"Successfully created bucket"}
Create an object using the …/create/object endpoint. This accepts an image url that will be downloaded to your bucket.
$ export CF_API='https://service.us.apiconnect.ibmcloud.com/gws/apigateway/
api/<>/create/ob ject'
$ curl -X POST -H 'Content-type: application/json' -d
'{"imageUrl":"https://newsroom.ibm.com/image/IBM+logo.jpg","bucketName":
"sample- bucket","objectName":"sample-image.png"}' $CF_API
{"etag":"c9fdfac834f00bc32958384e259c4d50"}
Get another list of APIs.
$ ibmcloud wsk api list sample-api-gateway-get
# Result # Action Verb API Name URL /interact-with-on-pre get sample-api-gateway-create https://service.us.apiconnect.ibmcloud.com/gws/apigateway/api/<>/get/object /interact-with-on-pre get sample-api-gateway-create https://service.us.apiconnect.ibmcloud.com/gws/apigateway/api/<>/get/ojbects
Get the list of objects in your created bucket using the …/get/objects endpoint.
$ export CF_API='https://service.us.apiconnect.ibmcloud.com/gws/apigateway/
api/<>/get/objec ts'
$ curl ''"$CF_API"'?bucketName=sample-bucket'
[{"name":"sample-image.png","lastModified":"2018-12-
06T23:15:24.000Z","etag":"c9fdfac834f00bc32958384e259c4d50","size":19292}]
Download the object using the ../get/object endpoint.
$ export CF_API='https://service.us.apiconnect.ibmcloud.com/gws/apigateway/ api/<>/get/objec t' $ curl ''"$CF_API"'?bucketName=sample-bucket&objectName=sample-image.png' > sample-image.png
## You can open the sample-image.png image in your file explorer.
Clean up
You can now clean up the resources you created.
$ docker stop couchdb-test $ docker stop minio-test $ wskdeploy -m manifest-couchdb.yaml undeploy $ wskdeploy -m manifest-webapp-minio.yaml undeploy
Alternative Deployment Methods
Deploy manually with the ibmcloud wsk command line tool
This approach shows you how to deploy individual the packages, actions, triggers, and rules with CLI commands. It helps you understand and control the underlying deployment artifacts.
Export the Cloud Host in an environment variable that you have from your Secure Gateway destination. It would look something like: cap-us-east-prd-sg-bm-
03.integration.ibmcloud.com:15006
$ export CLOUD_HOST="cap-us-east-prd-sg-bm-03.integration.ibmcloud.com:15006"
Bind the Cloudant package
$ ibmcloud wsk package bind /whisk.system/cloudant couchdb-binding \
-p username admin \
-p password password \
-p host ${CLOUD_HOST}
Create a Trigger to listen to database events
This will listen to changes on the testdb database
$ ibmcloud wsk trigger create couchdb-changes-trigger --feed couchdb- binding /changes \ --param dbname testdb
Create a Rule
The rule connects the trigger to an action.
$ ibmcloud wsk rule create couchdb-trigger-rule couchdb-changes-trigger /whisk.system/utils/echo
This connects the trigger to a simple echo action that just outputs the input which would be the database event.
Create the actions
Create a package to organize the actions that will be created for this repo.
$ ibmcloud wsk package create interact-with-on-prem $ ibmcloud wsk package create interact-with-on-prem-2
Create actions that interact with the web app
$ ibmcloud wsk action create interact-with-on-prem-2/get-object-request actions /get-object.js \ --kind nodejs:8 \ --param CLOUD_HOST $CLOUD_HOST $ ibmcloud wsk action create interact-with-on-prem-2/get-bucket-request actions /get-bucket-objects.js \ --kind nodejs:8 \ --param CLOUD_HOST $CLOUD_HOST $ ibmcloud wsk action create interact-with-on-prem-2/create-bucket-request actions /create-bucket.js \ --kind nodejs:8 \ --param CLOUD_HOST $CLOUD_HOST $ ibmcloud wsk action create interact-with-on-prem-2/create-object-request actions /create-object.js \ --kind nodejs:8 \ --param CLOUD_HOST $CLOUD_HOST
Create REST APIs
$ ibmcloud wsk api create /get /object get interact-with-on-prem-2/ get-object- request -n sample-api-gateway-get --response-type http $ ibmcloud wsk api create /get /objects get interact-with-on-prem-2/ get-bucket- request -n sample-api-gateway-get --response-type http $ ibmcloud wsk api create /create /bucket post interact-with-on-prem-2/ create-bucket-request -n sample-api-gateway-create --response-type http $ ibmcloud wsk api create /create /object post interact-with-on-prem-2/ create-object-request -n sample-api-gateway-create --response-type http
You can now proceed to the next step.
To delete them later:
$ ibmcloud wsk trigger delete couchdb-changes-trigger $ ibmcloud wsk rule delete couchdb-trigger-rule $ ibmcloud wsk package delete couchdb-binding $ ibmcloud wsk package delete interact-with-on-prem $ ibmcloud wsk action delete interact-with-on-prem-2/get-object-request $ ibmcloud wsk action delete interact-with-on-prem-2/get-bucket-request $ ibmcloud wsk action delete interact-with-on-prem-2/create-object-request $ ibmcloud wsk action delete interact-with-on-prem-2/create-bucket-request $ ibmcloud wsk api delete sample-api-gateway-get $ ibmcloud wsk api delete sample-api-gateway-create $ ibmcloud wsk package delete interact-with-on-prem-2
