Quick Glance at Hadoop Ecosystems

As mentioned in Hadoop post, the community around Hadoop has built tremendous tools and technology to support developers. This becomes Hadoop ecosystem. Some of the most popular ones are:

  • Hive
    Hadoop is based on Java language but not everyone can learn Java. Hive is a software built on top of Hadoop, it exposes SQL interface, allowing SQL developers to use powerful Hadoop system in their familiar language. If you know SQL, you don’t have to experience in Java in order to leverage Hadoop. Hive is using HiveQL language, very SQL-like.
  • HBase
    Basically a non-relational database on top of Hadoop. Even though it’s a non-relational, you can integrate with other system just like a traditional database.
  • Pig
    A tool in Hadoop ecosystem used to manipulate data, transforming unstructured data to structured data. It also has interface to query the data, just like Hive.
  • Storm
    Event stream processor that lives in Hadoop, used to process stream of data (as opposed to batch data). Example would be to process stream of IOT data, where data from an IOT device keep flowing through the system.
  • Oozie
    A workflow management system to coordinate between different Hadoop technologies.
  • Flume / Sqoop
    More of integration system that will tranfer data to and from Hadoop system. If you have data that live outside of Hadoop and need to be processed in Hadoop, Flume / Sqoop will do the job.
  • Spark
    A distribute compute engine within Hadoop. It’s used to process large amount of data, prep-ing them for analytics, machine learning, etc. Needless to say, it has a lot of built-in library for machine learning, artificial intelligence, analytics, stream processing and graph processing. Spark also support various different language, Scala, Python, R, etc.

This is definitely an oversimplified explanation of Hadoop ecosystem and there are lots of other technologies not covered here. But, this should give you quick explanation of each of them.

Advertisements

Barebone Angular with D3 v5

Just uploaded Github project showcasing sample Angular project with D3 v5.

The project contain implementation of D3 v5 in Angular project and consist of most popular D3 features such as Basic shape, Animation, Chart, Dragging and Zooming. Each sample is self-contained. D3 dependencies can be seen on `package.json`.
Time permitting, I will add more samples in the future.

For D3 new-comers, check out this resources explaining most important concepts in D3, Selection, Joins and if you work with chart, Scale.

Invoke Google Cloud Functions with HTTP GET/POST

Here is guideline to trigger Google Cloud Functions (GCF) with HTTP GET/POST.

  1. Create new GCF. Set the Trigger to `HTTP`. Please note Function to execute must be valid function name in the code. In this example I call the function `record`. URL is the endpoint of GCF.
  2. cloud functions 3
    cloud functions 4

  3. In this example, I use GCF’s Inline Editor. Following is sample code. Note that parameter `req` and `res` in function `record` is Google convention.
  4. exports.record = (req, res) => {
      // Construct message
      let message = constructMessage(req);
      res.status(200).send(message);
    };
    
    constructMessage = (req) => {
      // Read query string parameter `message`, HTTP GET
      let message = req.query.message;
      // Create response
      let response = '{"value": "' + message + '"}';
      
      // Parse the response as JSON then return
      return JSON.parse(response);
    };
    
  5. Once GCF is created, Google will deploy GCF automatically and soon the URL will be available to invoke.
  6. Sample invocation with HTTP GET, notice I pass in parameter `message` in query string which correspond with my code that look for parameter `message` and return constructed response. I use Postman.
  7. cloud functions 2

  8. To allow HTTP POST invocation, I need to change the code a little bit to look into request body instead of query string. It’s very possible to write code that check for both query string and request body (not discussed here).
  9. exports.record = (req, res) => {
      // Construct message
      let message = constructMessage(req);
      res.status(200).send(message);
    };
    
    constructMessage = (req) => {
      // Read body parameter `message`, HTTP POST
      let message = req.body.message;
      // Create response
      let response = '{"value": "' + message + '"}';
    
      // Parse the response as JSON then return
      return JSON.parse(response);
    };
    
  10. And sample invocation with HTTP POST
  11. cloud functions 1

Invoke Google Apps Script with HTTP GET/POST

To be able to invoke Google Apps Script (GAS) through HTTP, the GAS must be deployed as web app, follow guidelines below.

  1. In script editing screen, add `doGet` and `doPost` functions. This following sample functions take a request parameter and simply check for query string parameter `name` if it’s HTTP GET and request body parameter `name` if it’s HTTP POST. Then return JSON object. The functions’s name `doGet` and `doPost` is mandatory Google conventions in order to accept HTTP GET and HTTP POST. For `ContentService` reference see Google docs.
  2. function doGet(request) {
      // Un-comment to see request output
      // return ContentService.createTextOutput(JSON.stringify(request));
    
      // HTTP GET
      var requestContent = request.parameter;
      var responseContent = '{"response": "' + requestContent.name + '"}';
      var response = ContentService.createTextOutput(responseContent);
      // Set return data type
      response.setMimeType(ContentService.MimeType.JSON);
    
      return response;
    }
    
    function doPost(request) {
      // Un-comment to see request output
      // return ContentService.createTextOutput(JSON.stringify(request));
    
      // HTTP POST
      var requestContent = JSON.parse(request.postData.contents);
      var responseContent = '{"response": "' + requestContent.name + '"}';
      var response = ContentService.createTextOutput(responseContent);
      // Set return data type
      response.setMimeType(ContentService.MimeType.JSON);
    
      return response;
    }
    
  3. In script editing screen, click `Publish` menu and `Deploy as web app…`
  4. apps script 1

  5. Configuration window will pop up. Current web app URL is the URL where the GAS can be invoked. Who has access to the app determine authorization setting on who can invoke the URL, the least restricted setting is `Anyone, even anonymous`.
  6. apps script 2

  7. The first time you publish the GAS, Google will ask for authorization.
  8. apps script 5

  9. Google automatically scan the script to identify authorization scopes. Google requires GAS that need sensitive authorization scopes (any scopes that access user data) to be verified by Google. If your GAS is not verified, Google will display warning message (see screen shot below). Simply click `Advanced` and proceed. However, if you plan to distribute your GAS to public it’s good idea to have Google verify your GAS (not discussed in this article).
  10. apps script 6

  11. Invoke GAS like REST API, here is example of HTTP GET invocation of functions we wrote earlier. Notice I pass in parameter `name` in query string which correspond to my code that look for parameter `name`. I use Postman.
  12. apps script 4

  13. And example of HTTP POST.
  14. apps script 3

How to Change iOS App Name in Ionic 3

To change iOS app name in Ionic 3:

  1. Change `name` property in `config.xml`.
  2. Ionic product name change

  3. Then remove iOS platform with Ionic CLI command.
  4. $ ionic cordova platform rm ios
    
  5. Then add iOS platform back.
  6. $ ionic cordova platform add ios
    
  7. Rebuild the project.

Ionic Plugin Camera Crash

If you add Ionic’s native camera plugin and it crashed when opened in iOS device, add following entries in `Info.plist`, see previous post on how to add entries.

  • NSCameraUsageDescription
  • NSPhotoLibraryUsageDescription
  • NSLocationWhenInUseUsageDescription
  • NSPhotoLibraryAddUsageDescription

It’s important to add description, this is iOS requirement when apps try to access users data.

Source.

Ionic Plugin Camera Info.plist

To add entries to `Info.plist`, use `edit-config` tag in `config.xml`, for example:

<edit-config file="*-Info.plist" mode="merge" target="NSCameraUsageDescription">
    <string>need camera access to take pictures</string>
</edit-config>

cordova plugin camera crash

To verify, build Ionic for iOS platform, then open project properties in Xcode.
cordova plugin camera crash 2

Source.